incubator-hugegraph-computer.git
15 months agoImplement StreamGraphInput and EntryInput (#60)
Linary [Fri, 18 Jun 2021 08:25:37 +0000 (16:25 +0800)] 
Implement StreamGraphInput and EntryInput (#60)

15 months agoImplement data receiver of worker (#51)
houzhizhen [Fri, 18 Jun 2021 05:40:23 +0000 (13:40 +0800)] 
Implement data receiver of worker (#51)

* add DataServerManager to manage the data server.
* add MessageRecvBuffers the manages buffers with limit size.
* add MessageRecvPartition to manage the data received for a partition and three subclass
  (VertexMessageRecvPartition,EdgeMessageRecvPartition, ComputeMessageRecvPartition).
* add MessageRecvPartitions to manage the partitions of data received, and three subclass
  (VertexMessageRecvPartitions,EdgeMessageRecvPartitions, ComputeMessageRecvPartitions)
* add MessageRecvManager to implement MessageHandler to handle network messages.
* process start-finish event.
* Partition add iterator and sort logic.
* add class SuperstepFileGenerator.
* create ConnectionManager in WorkerService.
* add mergeBuffers using sort-mamager.

15 months agoUse github action for ci via ci.yml (#62)
Linary [Fri, 18 Jun 2021 03:34:04 +0000 (11:34 +0800)] 
Use github action for ci via ci.yml (#62)

15 months agoLink managers in worker sending end (#52)
Linary [Tue, 15 Jun 2021 07:35:58 +0000 (15:35 +0800)] 
Link managers in worker sending end (#52)

* Link managers in worker sending end
* Optimize WriteBufferPool and SortedBufferQueuePool
* Rafact MessageSendPartition, WriteBuffer and Edge
* Refact writeEdge with label and name
* Let SortManager don't need Sender
* Refact GraphComputeOutput and GraphWritebackOutput
* Refact OptimizedBytesOutput
* Add Input/Output Factory
* Refact writeFixedInt in sort module
* Move pending message into MessageQueue
* Define a MultiQueue support multi put

15 months agoAdd sessionActive for client (#59)
coderzc [Thu, 10 Jun 2021 07:50:08 +0000 (15:50 +0800)] 
Add sessionActive for client (#59)

16 months agoFix EntriesUtil#entryFromInput and some improve (#58)
ShouJing [Fri, 4 Jun 2021 11:45:57 +0000 (19:45 +0800)] 
Fix EntriesUtil#entryFromInput and some improve (#58)

16 months agoadd PointerCombiner (#57)
houzhizhen [Thu, 3 Jun 2021 06:30:36 +0000 (14:30 +0800)] 
add PointerCombiner (#57)

16 months agoAdd store/sort module (#37)
ShouJing [Mon, 31 May 2021 06:51:32 +0000 (14:51 +0800)] 
Add store/sort module (#37)

* add hgkvFile/hgkvDir module
* add EntriesUtil.java
* add SorterTest.java
* move file verification to HgkvDir
* rebuild HgkvFile readFooter and Fix HgkvDir#open
* add sort module
* implement InnerSortFlusher and OuterSortFlusher
* add KvEntry#numSubEntries

16 months agoadd startAsync and finishAsync on client side, add onStarted and onFinished on server...
coderzc [Mon, 31 May 2021 06:33:01 +0000 (14:33 +0800)] 
add startAsync and finishAsync on client side, add onStarted and onFinished on server side (#53)

* add startAsync and finishAsync on client side
* add onStarted and onFinished on server side

16 months agorename DataFileManager to FileManager (#54)
houzhizhen [Fri, 28 May 2021 06:26:06 +0000 (14:26 +0800)] 
rename DataFileManager to FileManager (#54)

* rename DataFileManager to FileManager

* remove #nextFile() from FileGenerator

16 months agoimplement aggregator module (#46)
Jermy Li [Fri, 21 May 2021 07:50:09 +0000 (15:50 +0800)] 
implement aggregator module (#46)

* implement aggregator module
* add DefaultAggregator.newValue()
* integrate aggregator into master-service and worker-service
* ensure worker init() executed after master
* add rpc interface registeredAggregators()
* also add two-tier structure to store current-step and previous-step aggregators
* add repair() for Aggregator
* let master context use config of outer class
* support register aggregator with default value
* add Manager.inited()
* remove worker_id option and use server_port instead
* use rpc-url from master-info

16 months agoadd copy() and assign() methods to Value class (#49)
Jermy Li [Wed, 19 May 2021 11:56:19 +0000 (19:56 +0800)] 
add copy() and assign() methods to Value class (#49)

* add Value.copy()+assign() method
* add IntValueSumCombiner and FloatValueSumCombiner

16 months agoStandardize bsp synchronization between master and worker (#48)
houzhizhen [Thu, 13 May 2021 11:45:31 +0000 (19:45 +0800)] 
Standardize bsp synchronization between master and worker (#48)

16 months agolog job-id in BSP client (#50)
Jermy Li [Thu, 13 May 2021 09:28:43 +0000 (17:28 +0800)] 
log job-id in BSP client (#50)

* log job-id in BSP client

* set BspBase.init() private

16 months agoRefactor Writable and Readable class (#42)
Linary [Wed, 12 May 2021 04:02:47 +0000 (12:02 +0800)] 
Refactor Writable and Readable class (#42)

* Refactor Writable and Readable class

* Also simplify GraphOut and GraphInput

* remove ComputeException

* Add test for duplicate method

* fix write/read properties value type incorrect

16 months agofix some warning, improve start() and finish() (#44)
coderzc [Tue, 11 May 2021 10:06:31 +0000 (18:06 +0800)] 
fix some warning, improve start() and finish() (#44)

* fix some warning
* improve some log
* improve start/finish send to avoid timeout
* blocking -> flowBlocking

16 months agostandard log format of bsp module and improve some error message (#47)
Jermy Li [Mon, 10 May 2021 07:04:14 +0000 (15:04 +0800)] 
standard log format of bsp module and improve some error message (#47)

16 months agoadd DataDirManager (#43)
houzhizhen [Sat, 8 May 2021 11:25:10 +0000 (19:25 +0800)] 
add DataDirManager (#43)

16 months agoadd createId methods in GraphFactory (#45)
houzhizhen [Sat, 8 May 2021 11:24:41 +0000 (19:24 +0800)] 
add createId methods in GraphFactory (#45)

17 months agoadd flowControl, heartbeat (#40)
coderzc [Thu, 6 May 2021 03:37:56 +0000 (11:37 +0800)] 
add flowControl, heartbeat (#40)

* add flow control
* add heartBeat
* add task to check ack
* add task to check ack and remove lastAckTimestamp
* rest BarrierEvent
* remove AckType
* add checkMinPendingRequests
* use accumulate flush
* add timeout parameter method to startSession and finishSession
* use real ip start
* scheduleAtFixedRate -> scheduleWithFixedDelay
* add maxHeartbeatTimes
* add dataMessage release()
* max_heartbeat_timeouts -> max_timeout_heartbeat_count
* set backlog default value 511

17 months agocreate api module (#41)
houzhizhen [Thu, 29 Apr 2021 09:13:53 +0000 (17:13 +0800)] 
create api module (#41)

* move algorithm's used classes to api
* add module computer-api
* signal bsp at end of init in WorkerService and MasterService
* make ValueFactory abstract and add BuiltinValueFactory
* rename USER_DEFINED to CUSTOM_VALUE in ValueType
* Reduce call ComputerContext.instance() directly
* synchronious UnitTestBase.updateWithRequiredOptions and return Config
* pass Aggregator4Master to MasterComputation#init
* add parameter MasterContext in MasterComputation.close

17 months agorename FilterMapComputation to FilterComputation (#38)
houzhizhen [Fri, 16 Apr 2021 10:18:36 +0000 (18:18 +0800)] 
rename FilterMapComputation to FilterComputation (#38)

17 months agoadd transport server, client, encode, decode (#25)
Coderzc [Fri, 16 Apr 2021 07:54:13 +0000 (15:54 +0800)] 
add transport server, client, encode, decode (#25)

add transport server
add transport client
add encoder
add decoder

17 months agoImprove class for Input/Output and fix writeInt(position, v) in BufferedFileOutput...
ShouJing [Thu, 15 Apr 2021 02:58:14 +0000 (10:58 +0800)] 
Improve class for Input/Output and fix writeInt(position, v) in BufferedFileOutput (#32)

* Improve class for Input/Output and fix problem

* update compare method in BufferFileInput/UnsafeByteArrayInput

* add exception declaration on RandomAccessInput#compare

* Add test case for UnsafeByteArrayInput/BufferedFileInput compare method

* Check offset and lenght must be in buffer

* Improve local variable names in BufferedFileTest#testCompare

* Update argument check in UnsafeByteArrayInput#compare

17 months agoadd rpc support (#36)
Jermy Li [Wed, 14 Apr 2021 08:13:23 +0000 (16:13 +0800)] 
add rpc support (#36)

* add rpc server and client: MasterRpcManager,WorkerRpcManager
* add MasterInputManager and WorkerInputManager
* inject rpc service into GraphFetcher
* store managers through map instead of list
* add loadGraph() and mergeGraph() to WorkerInputManager
* add checkInited() for WorkerService and MasterService

17 months agoAdd close method for Input Fetcher (#35)
Linary [Tue, 13 Apr 2021 13:04:05 +0000 (21:04 +0800)] 
Add close method for Input Fetcher (#35)

Co-authored-by: ShouJing <1075185785@qq.com>
17 months agofix some warnings for computer-test (#34)
Jermy Li [Tue, 13 Apr 2021 12:39:11 +0000 (20:39 +0800)] 
fix some warnings for computer-test (#34)

17 months agofix some warnings (#33)
Jermy Li [Mon, 12 Apr 2021 12:16:05 +0000 (20:16 +0800)] 
fix some warnings (#33)

17 months agoadd WorkerService and MasterService (#27)
houzhizhen [Fri, 9 Apr 2021 06:44:13 +0000 (14:44 +0800)] 
add WorkerService and MasterService (#27)

* rename input to inputStep
* MasterService does not implement MasterContext
* add name in Combiner and Computation
* add category in Computation

17 months agocreate Heap/LoserTree (#26)
ShouJing [Wed, 7 Apr 2021 07:32:55 +0000 (15:32 +0800)] 
create Heap/LoserTree (#26)

* add InputsSorting interface and implementation

17 months agoadd Partitioner and HashPartitioner (#29)
houzhizhen [Wed, 7 Apr 2021 02:40:09 +0000 (10:40 +0800)] 
add Partitioner and HashPartitioner (#29)

18 months agoInstall HugeGraphServer from specified commit id (#31)
Linary [Fri, 2 Apr 2021 09:29:39 +0000 (17:29 +0800)] 
Install HugeGraphServer from specified commit id (#31)

18 months agoadd copy, getLast, contains in ListValue (#30)
houzhizhen [Fri, 2 Apr 2021 02:29:59 +0000 (10:29 +0800)] 
add copy, getLast, contains in ListValue (#30)

* make getLast next to get

18 months agoadd createObject(class_option) to config (#28)
houzhizhen [Tue, 30 Mar 2021 14:34:14 +0000 (22:34 +0800)] 
add createObject(class_option) to config (#28)

18 months agoAdd computation interface (#17)
houzhizhen [Fri, 26 Mar 2021 02:41:24 +0000 (10:41 +0800)] 
Add computation interface (#17)

* Add computation interface

* add FilterMapComputation and ReduceComputation

* add MasterComputation

18 months agoadd ComputerDriver interface (#23)
houzhizhen [Tue, 23 Mar 2021 03:02:42 +0000 (11:02 +0800)] 
add ComputerDriver interface (#23)

18 months agoadd ComputerOutput interface (#24)
houzhizhen [Mon, 22 Mar 2021 06:25:58 +0000 (14:25 +0800)] 
add ComputerOutput interface (#24)

18 months agoConfig add getBoolean, getInt, getLong, getDouble, getString (#21)
houzhizhen [Mon, 22 Mar 2021 02:48:32 +0000 (10:48 +0800)] 
Config add  getBoolean, getInt, getLong, getDouble, getString (#21)

18 months agoadd sort interface (#19)
houzhizhen [Thu, 18 Mar 2021 01:33:32 +0000 (09:33 +0800)] 
add sort interface (#19)

* first version of sorter

* add sort interface

* remove first version of sorter

18 months agoadd available in RandomAccessInput (#22)
houzhizhen [Wed, 17 Mar 2021 12:03:47 +0000 (20:03 +0800)] 
add available in RandomAccessInput (#22)

18 months agoadd transport interface (#20)
houzhizhen [Wed, 17 Mar 2021 03:45:20 +0000 (11:45 +0800)] 
add transport interface (#20)

18 months agoImplement vertex/edge fetcher to load data from hugegraph server (#18)
Linary [Mon, 15 Mar 2021 13:00:43 +0000 (21:00 +0800)] 
Implement vertex/edge fetcher to load data from hugegraph server (#18)

* Add install-hugegraph-from-source.sh

* Add HugeConverter

18 months agoadd BufferedFileDataOutput and BufferedFileDataInput (#15)
houzhizhen [Thu, 11 Mar 2021 03:41:07 +0000 (11:41 +0800)] 
add BufferedFileDataOutput and BufferedFileDataInput (#15)

* add BufferedFileDataOutput and BufferedFileDataInput

* add BufferedInputStream and BufferedOutputStream

18 months agoadd DefaultPropertiesCombiner and DefaultVertexValueCombiner (#16)
houzhizhen [Wed, 10 Mar 2021 11:36:21 +0000 (19:36 +0800)] 
add DefaultPropertiesCombiner and DefaultVertexValueCombiner (#16)

* add OverwriteValueCombiner, MergeNewPropertiesCombiner and MergeOldPropertiesCombiner

19 months agoreconstruct StreamGraph{Input/Output} to associate with UnsafeByteArray{Input/Output...
houzhizhen [Mon, 8 Feb 2021 13:04:36 +0000 (21:04 +0800)] 
reconstruct StreamGraph{Input/Output} to associate with UnsafeByteArray{Input/Output} (#14)

19 months agoadd Combiner (#12)
houzhizhen [Fri, 5 Feb 2021 11:58:24 +0000 (19:58 +0800)] 
add Combiner (#12)

* add Combiner, DoubleValueSumCombiner, IdValueMinimumCombiner, LongValueSumCombiner
* Value implements Comparable<Value>

20 months agoadd UnsafeByteArrayGraphInput and UnsafeByteArrayGraphOutput (#13)
houzhizhen [Thu, 4 Feb 2021 09:30:51 +0000 (17:30 +0800)] 
add UnsafeByteArrayGraphInput and UnsafeByteArrayGraphOutput (#13)

20 months agoadd Bsp4Worker, Bsp4Master and Etcd based implementation (#9)
houzhizhen [Thu, 4 Feb 2021 09:14:54 +0000 (17:14 +0800)] 
add Bsp4Worker, Bsp4Master and Etcd based implementation (#9)

20 months agoAdd Vertex, Edge, Properties and JsonStructGraphOutput (#10)
Linary [Thu, 4 Feb 2021 00:59:50 +0000 (08:59 +0800)] 
Add Vertex, Edge, Properties and JsonStructGraphOutput (#10)

* Add Vertex, Edge, Properties and JsonStructGraphOutput

* Add Recycler and Allocator

* Use RecycleReference to hold Handle and Recyclable

* Use Factory to create Vertex and Edge

20 months agoadd etcd dependency (#8)
houzhizhen [Mon, 25 Jan 2021 05:47:04 +0000 (13:47 +0800)] 
add etcd dependency (#8)

* add etcd dependency
* add EtcdClient
* add BspTestSuite
* improve exception messages
* get value from put event
* fix a unit test setting bug
* unuse static import
* define const ENCODING

20 months agoImprove class for Id, Value, GraphOutput and GraphInput (#6)
Linary [Tue, 19 Jan 2021 01:13:59 +0000 (09:13 +0800)] 
Improve class for Id, Value, GraphOutput and GraphInput (#6)

* Improve class for Id, Value, GraphOutput and GraphInput

* Add ListValue and improve code

* Add IdValueList and IdValueListList

* Let StreamGraphOutput as a proxy of DataOutputStream

* Add CsvStructGraphOutput and JsonStructGraphOutput

21 months agoadd Id as base type of vertex id (#4)
houzhizhen [Thu, 31 Dec 2020 06:52:02 +0000 (14:52 +0800)] 
add Id as base type of vertex id (#4)

* add Id as base type of vertex id

* rename method readFields to read in Writable

* add TextId, ByteArrayUtil, CoderUtil, PlainByteArrayComparator

* add DoubleValue

* tiny improve

* add ValueType and NullValue

* rename TextId to Utf8Id, add code in ValueType

* add CommonTestSuite

* update the code type into int from byte, and other tiny improvements

* remove String.format

21 months agoDefine some base exception class (#5)
Linary [Tue, 29 Dec 2020 07:43:38 +0000 (15:43 +0800)] 
Define some base exception class (#5)

* Add test module and codecov plugin

21 months agoAdd checkstyle plugin to check code style (#3)
Linary [Fri, 25 Dec 2020 12:17:46 +0000 (20:17 +0800)] 
Add checkstyle plugin to check code style (#3)

* Add checkstyle plugin to check code style

* Add travis yaml

21 months agoMerge pull request #2 from hugegraph-ee/build-framework
houzhizhen [Sat, 12 Dec 2020 03:41:26 +0000 (11:41 +0800)] 
Merge pull request #2 from hugegraph-ee/build-framework

Build hugegraph-computer project framework

21 months agoBuild hugegraph-computer project framework 2/head
liningrui [Fri, 11 Dec 2020 06:53:59 +0000 (14:53 +0800)] 
Build hugegraph-computer project framework

2 years agoMerge pull request #1 from hugegraph-ee/module
houzhizhen [Mon, 28 Sep 2020 03:30:10 +0000 (11:30 +0800)] 
Merge pull request #1 from hugegraph-ee/module

add module computer-core and computer-algorithm

2 years agoadd module computer-core and computer-algorithm 1/head
houzhizhen@baidu.com [Wed, 23 Sep 2020 07:51:04 +0000 (15:51 +0800)] 
add module computer-core and computer-algorithm

2 years agofirst commit
houzhizhen@baidu.com [Tue, 22 Sep 2020 03:23:15 +0000 (11:23 +0800)] 
first commit