incubator-hugegraph-toolchain.git
2 years agoSupport customized timezone for date (#107) release-0.10.1 v0.10.1
Linary [Thu, 5 Dec 2019 13:28:53 +0000 (21:28 +0800)] 
Support customized timezone for date (#107)

Change-Id: I9814239e101e1424fc6d247a9984fcaaedbcca5c

2 years agoFix NPE when json file reload (#112)
Linary [Thu, 5 Dec 2019 10:13:14 +0000 (18:13 +0800)] 
Fix NPE when json file reload (#112)

Fix #111

2 years agoSupport read hadoop conf file (#105)
Dominic [Tue, 26 Nov 2019 08:51:58 +0000 (16:51 +0800)] 
Support read hadoop conf file (#105)

implement #104

2 years agoFix line retainAll doesn't modify names and values array (#110)
Linary [Tue, 26 Nov 2019 07:46:26 +0000 (15:46 +0800)] 
Fix line retainAll doesn't modify names and values array (#110)

Change-Id: I383b3b20a162816098aa8ee8468e8255b3553fdc

2 years agoUpgrade version to 0.10.0 (#103)
Linary [Fri, 8 Nov 2019 06:08:58 +0000 (14:08 +0800)] 
Upgrade version to 0.10.0 (#103)

Change-Id: If74cc9fd3e9cf132b3c4fc3f9ff0c135d3077634

2 years agoSupport skip struct to avoid delete it (#101)
Linary [Thu, 7 Nov 2019 08:26:26 +0000 (16:26 +0800)] 
Support skip struct to avoid delete it (#101)

implement #100

Change-Id: Ie7fb4a11e17e957b111dddb972845f444a5e8053

2 years agoFix list or set property update failed (#99)
Linary [Wed, 6 Nov 2019 10:14:19 +0000 (18:14 +0800)] 
Fix list or set property update failed (#99)

Change-Id: Ia4d307776b86c6bcbe2b06957b4cf4cb921d9c3d

2 years agoLog failure record to compensate offset ahead problem (#87)
Linary [Wed, 16 Oct 2019 11:42:14 +0000 (19:42 +0800)] 
Log failure record to compensate offset ahead problem (#87)

Change-Id: Ibba36ae57444ded2f39261512df6704725b486d7

2 years agoSupport batch update element properties with strategies (#97)
imbajin [Wed, 9 Oct 2019 09:09:23 +0000 (17:09 +0800)] 
Support batch update element properties with strategies (#97)

The strategy in JSON is case sensitive now, wait for jackthon update to V2.9+ to support case-insensitive original

2 years agoSupport uuid idstrategy (#98)
Linary [Wed, 9 Oct 2019 02:44:27 +0000 (10:44 +0800)] 
Support uuid idstrategy (#98)

Change-Id: If7bd8a59eccf4cfb6d138a7acba1923f912dbc2c

2 years agoImprove some code (#96)
Linary [Tue, 3 Sep 2019 09:39:10 +0000 (17:39 +0800)] 
Improve some code (#96)

Change-Id: Ic7addbf75b324c931aeeca7f244e993547d136d1

2 years agoSupport unsigned long property (#91)
Linary [Thu, 29 Aug 2019 11:19:34 +0000 (19:19 +0800)] 
Support unsigned long property (#91)

Change-Id: Iae2aa1e6d4f1b765d902fda6613d6143b3590f69

2 years agoFix boolean property parse failed (#92)
Linary [Tue, 27 Aug 2019 11:27:24 +0000 (19:27 +0800)] 
Fix boolean property parse failed (#92)

Change-Id: I539c3058cb9c4b850f5e53fd00872a32b76fe817

2 years agoOptimize processing logic for line with extra or missing columns (#93)
Linary [Mon, 26 Aug 2019 12:03:11 +0000 (20:03 +0800)] 
Optimize processing logic for line with extra or missing columns (#93)

Change-Id: If48cc313cf0073329916e1d33f92b1ae749b1d12

2 years agoFix empty table hasNext return true (#90)
Linary [Mon, 19 Aug 2019 03:18:35 +0000 (11:18 +0800)] 
Fix empty table hasNext return true (#90)

Change-Id: I6449b93a23c01ed804c71c650d9f8712b26a1af7

3 years agoSupport read header in different releation database (#79)
Linary [Fri, 9 Aug 2019 09:07:48 +0000 (17:07 +0800)] 
Support read header in different releation database (#79)

Implement #77
Change-Id: I83bb8ea8652463ceeba520ae3f4a76d169ecae83

3 years agoImprove some perf on reader, parser and builder (#84)
Linary [Mon, 5 Aug 2019 08:39:59 +0000 (16:39 +0800)] 
Improve some perf on reader, parser and builder (#84)

Change-Id: Ida4f3bba6c79c0a53e3ef35826648b7bfd17bb78

3 years agoFix acquired semaphores num error when wait task finished (#86)
Linary [Fri, 2 Aug 2019 09:08:13 +0000 (17:08 +0800)] 
Fix acquired semaphores num error when wait task finished (#86)

Change-Id: I112261828af7267ba7463461153486af3ea97d4f

3 years agoSupport record load progress for resume loading (#70)
Linary [Mon, 1 Jul 2019 13:02:56 +0000 (21:02 +0800)] 
Support record load progress for resume loading (#70)

Implement #69

Change-Id: I39578958f0915f42840bf2789369df647b4867a0

3 years agoSupport filter files by file name suffix (#82)
Linary [Wed, 26 Jun 2019 07:10:06 +0000 (15:10 +0800)] 
Support filter files by file name suffix (#82)

Implement #44

Change-Id: I25484c390015daacd907ea102c31c1f8aaa8f495

3 years agoEnhance summary info (#80)
imbajin [Thu, 20 Jun 2019 03:47:11 +0000 (11:47 +0800)] 
Enhance summary info (#80)

1. add parserSuccess to comfirm data integrity
2. add parserSpeed to get performance results directly
3. record the import speed of each data source in the log for comparison

3 years agoAdd http-connection options (#81)
imbajin [Tue, 18 Jun 2019 13:28:40 +0000 (21:28 +0800)] 
Add http-connection options (#81)

3 years agoFix parser exception cannot be caught (#74)
imbajin [Mon, 17 Jun 2019 05:13:07 +0000 (13:13 +0800)] 
Fix parser exception cannot be caught (#74)

3 years agoAdd an introduction about the supported data sources (#78)
Linary [Fri, 14 Jun 2019 03:56:45 +0000 (11:56 +0800)] 
Add an introduction about the supported data sources (#78)

Change-Id: I63c454f8cdafa922174a952931e171b63c6b2a4e

3 years agoSupport value mapping (#67)
Linary [Tue, 7 May 2019 10:11:15 +0000 (18:11 +0800)] 
Support value mapping (#67)

Change-Id: I39cad735f176cc772dd51c48f4d61537bbcf1c51

3 years agoSupport customized list element delimiter in text file (#66)
Linary [Thu, 25 Apr 2019 07:55:29 +0000 (15:55 +0800)] 
Support customized list element delimiter in text file (#66)

Implement #65
Change-Id: I9c20483c38133a04ce20e9a0843a2dce98d1ec73

3 years agoSupport "selected" in JSON config of data source (#62)
Linary [Mon, 22 Apr 2019 13:12:59 +0000 (21:12 +0800)] 
Support "selected" in JSON config of data source (#62)

Implement #61
Change-Id: I9fe630ebd823d678e1d2d95d497d91f1fdf5760a

3 years agoFix nullable key in mapping field can't work (#64)
Linary [Mon, 22 Apr 2019 08:25:38 +0000 (16:25 +0800)] 
Fix nullable key in mapping field can't work (#64)

Fix #63

Change-Id: I87d47ef14334a8326ce96e4def4571a0ed31e5fd

3 years agoUpgrade version to 0.9.0 (#60)
Linary [Fri, 19 Apr 2019 10:18:46 +0000 (18:18 +0800)] 
Upgrade version to 0.9.0 (#60)

Change-Id: Iee025b4e6700bd61936f5ccea9a02608c1fb8c83

3 years agoreader: improve some error messages (#52)
Jermy Li [Tue, 9 Apr 2019 07:35:15 +0000 (02:35 -0500)] 
reader: improve some error messages (#52)

Change-Id: I974377a676908a9f46c26c45d7a41c7e913298b0

3 years agoremove unused code (#53)
Jermy Li [Tue, 9 Apr 2019 07:26:23 +0000 (02:26 -0500)] 
remove unused code (#53)

Change-Id: I120fe8119b53c484b45a02e84e692424fdece599

3 years agoadd serial id to XxException class (#54)
Jermy Li [Tue, 9 Apr 2019 07:16:35 +0000 (02:16 -0500)] 
add serial id to XxException class (#54)

Change-Id: I00bb4a738333b95312403c5bd5cde954487c0d5b

3 years agoThe import Unmodifiable is never used (#51)
浩宇天罡 [Tue, 9 Apr 2019 06:57:51 +0000 (14:57 +0800)] 
The import Unmodifiable is never used (#51)

The import org.apache.commons.collections.Unmodifiable is never used

3 years agoExample problem fix (#57)
Linary [Tue, 9 Apr 2019 06:50:39 +0000 (14:50 +0800)] 
Example problem fix (#57)

Change-Id: I52b3257f5020cd39689654d5a2d473efcb4cb907

3 years agoUpgrade and temporarily comment vulnerable dependency (#48)
Linary [Wed, 27 Mar 2019 12:28:20 +0000 (20:28 +0800)] 
Upgrade and temporarily comment vulnerable dependency (#48)

Change-Id: Iaebd63883e0878c0c1a822e8e4f9a6bf466c2b5d

3 years agoAdd an option 'token' for authentication (#46)
Linary [Fri, 22 Mar 2019 09:20:40 +0000 (17:20 +0800)] 
Add an option 'token' for authentication (#46)

Change-Id: Id2d748f83ce3c52dc961ba32b044dff6e40b6615

3 years agoimprove url regex regex (#47)
zhoney [Wed, 20 Mar 2019 09:02:41 +0000 (17:02 +0800)] 
improve url regex regex (#47)

fixed: #37

Change-Id: Ieafb22a7805ecb834a7d412a770b115c6bbd98f9

3 years agoSupport config regex to skip specified line (#43)
Linary [Mon, 18 Mar 2019 07:58:52 +0000 (15:58 +0800)] 
Support config regex to skip specified line (#43)

Implement #42

Change-Id: I08fd43a65706f9a402b2ac328a3a5c034b8445df

3 years agoSupport load data from HDFS and relational database (#14)
Linary [Mon, 18 Mar 2019 06:28:06 +0000 (14:28 +0800)] 
Support load data from HDFS and relational database (#14)

Support load data from HDFS and relational database

1. Refactoring code for unit tests
- Add test for ordinary text file on hdfs
- Add test for compressed files
2. Add tests for jdbc(mysql)
3. nstall mysql when profile is jdbc, remove unused test suite
4. Support pass 'fs.defaultFS' to connect remote hadoop

Change-Id: Idbae121701a97d0b7abce903621842f14877bd45

3 years agosupport list/set when load TEXT file (#38)
imbajin [Wed, 6 Mar 2019 03:18:52 +0000 (11:18 +0800)] 
support list/set when load TEXT file (#38)

Although unit-test under normal case can pass, there still have some problems:

1.Collections do not support date formats.
2.In some data formats, the internal order of the collection will change.. (like list)
3.It's better to add a field in struct.json as the list-data's start and end symbol. (optional)
4.In theory, CSV file can support list as well, but default delimiter of CSV file is,
   which is conflict with the separator for collection, wonder if there is any good solution?

3 years agoSupport customized date format
liningrui [Thu, 29 Nov 2018 09:47:44 +0000 (17:47 +0800)] 
Support customized date format

Implement #27

Change-Id: Ib00a1b6d53dee82c7be7cb8420662787707161d5

3 years agoSupport load with a specified directory
liningrui [Sat, 29 Dec 2018 08:23:39 +0000 (16:23 +0800)] 
Support load with a specified directory

Implement #29

Change-Id: Id3d9385b52484a30d6e71652ef1b55c40979dd88

3 years agoSupport to skip the line start with specified symbols
liningrui [Wed, 28 Nov 2018 12:38:32 +0000 (20:38 +0800)] 
Support to skip the line start with specified symbols

Implement #25

Change-Id: Id5a664a3d0c09bac7cb3761862ab69bfd8cca5dd

3 years agoFix edge parser error checking property key when customize id
liningrui [Wed, 28 Nov 2018 11:58:29 +0000 (19:58 +0800)] 
Fix edge parser error checking property key when customize id

Change-Id: If53feef102ecd66b78dc0f2ac2f83c4664b20a29

3 years agoIgnore last redudant column and null value columns if possible
liningrui [Wed, 28 Nov 2018 04:59:25 +0000 (12:59 +0800)] 
Ignore last redudant column and null value columns if possible

Implement #21
Implement #22

Change-Id: I3551d806f9480cfe443451be5592b87b7fb14128

3 years agoFix bug vertices has joint primary keys load error
liningrui [Fri, 16 Nov 2018 08:19:14 +0000 (16:19 +0800)] 
Fix bug vertices has joint primary keys load error

Fix #17

Change-Id: I74ff2445602ff8afb76b58c3e9da5ea68eebeb27

3 years agoFix two bug about single tasks and executor
liningrui [Wed, 17 Oct 2018 09:43:23 +0000 (17:43 +0800)] 
Fix two bug about single tasks and executor

1. When a single insert task times out, it may still cause statistical errors.
2. TaskManager.shutdown() ignored singleExecutor

Fix #11

Change-Id: I9a468241f369f4c7e73a95ba9a5772dac90e0ce1

3 years agoImprove exception message about check file
liningrui [Thu, 25 Oct 2018 15:09:00 +0000 (23:09 +0800)] 
Improve exception message about check file

Fix #15

Change-Id: Id90b08508a2f43508f7099650795f0b844e317e2

3 years agoAdd travis test
liningrui [Thu, 11 Oct 2018 09:22:45 +0000 (17:22 +0800)] 
Add travis test

Change-Id: I4d604d889913d22c22c63249e5e59c78c1c8550c

3 years agohugegraph-1410 add ignore mechanism to data source JSON config file
zhangyi51 [Sat, 29 Sep 2018 09:23:18 +0000 (17:23 +0800)] 
hugegraph-1410 add ignore mechanism to data source JSON config file

implemented: #1

Change-Id: I787451308e205f4dda44a074b826cc0354c94b5b

3 years agoSupport load data with set cardinality property
liningrui [Fri, 12 Oct 2018 10:22:07 +0000 (18:22 +0800)] 
Support load data with set cardinality property

Improve #9

Change-Id: Idafceec6abc3b0feed0027c1b4e3b43dbe7ca166

3 years agoFix bug vertices with CUSTOMIZE_NUMBER id load error
liningrui [Thu, 11 Oct 2018 03:56:08 +0000 (11:56 +0800)] 
Fix bug vertices with CUSTOMIZE_NUMBER id load error

Fix #5

Change-Id: I7cd495b3e27e128cd5c9d75d9ebd54fdbc5bca21

3 years agoFix error in statistical insertion result
liningrui [Wed, 10 Oct 2018 03:00:35 +0000 (11:00 +0800)] 
Fix error in statistical insertion result

Fix #3

Change-Id: Ia3b4956b32de5bf9f43b6c88ee19704f881af1e7

3 years agoHugeGraph-1389: Implement load data from file v0.7.0
liningrui [Mon, 13 Aug 2018 03:49:26 +0000 (11:49 +0800)] 
HugeGraph-1389: Implement load data from file

Co-authored-by: liujie <liujie23@baidu.com>
Co-authored-by: lizhigang <lizhigang03@baidu.com>
Co-authored-by: liningrui <liningrui@baidu.com>
Change-Id: I2ce8c6908e5d349ce9a834c05473431e12e0dd4b

3 years agoFirst commit for Loader
liningrui [Mon, 13 Aug 2018 03:48:24 +0000 (11:48 +0800)] 
First commit for Loader

Change-Id: Icbfaced1a6614ccf95246d7b56e1c6d3b9bacd88