incubator-hugegraph-computer.git
7 months agoasync watch the k8s job (#148) v0.1.1
coderzc [Tue, 23 Nov 2021 12:03:56 +0000 (20:03 +0800)] 
async watch the k8s job (#148)

* async watch k8s job
* let cancelJob return bool
* update version to 0.1.1

7 months agorelease maven package by actions (#119)
coderzc [Tue, 23 Nov 2021 12:02:58 +0000 (20:02 +0800)] 
release maven package by actions (#119)

7 months agochange expression engine (#149)
ShouJing [Tue, 23 Nov 2021 03:55:54 +0000 (11:55 +0800)] 
change expression engine (#149)

7 months agoadd default value() method for HugeGraphOutput (#144)
Jermy Li [Thu, 18 Nov 2021 07:15:35 +0000 (15:15 +0800)] 
add default value() method for HugeGraphOutput (#144)

* add default value() method for HugeGraphOutput
* fix EXPECT_RINGS format

7 months agoimprove: delete useless temp file (#142)
ShouJing [Wed, 17 Nov 2021 07:00:29 +0000 (15:00 +0800)] 
improve: delete useless temp file (#142)

* remove useless temp file in merge file process and inputs file after merge

7 months agofix rings detection unit test bug (#147)
ShouJing [Wed, 17 Nov 2021 05:49:10 +0000 (13:49 +0800)] 
fix rings detection unit test bug (#147)

7 months agoimplement kcore algorithm (#143)
Jermy Li [Tue, 16 Nov 2021 08:46:53 +0000 (16:46 +0800)] 
implement kcore algorithm (#143)

7 months agoclear vertex old files and edge old files on `beforeSuperstep(Constants.INPUT_SUPERST...
coderzc [Wed, 10 Nov 2021 07:46:09 +0000 (15:46 +0800)] 
clear vertex old files and edge old files on `beforeSuperstep(Constants.INPUT_SUPERSTEP + 1)` (#141)

8 months agodelete useless old files (#138)
coderzc [Fri, 5 Nov 2021 09:11:55 +0000 (17:11 +0800)] 
delete useless old files (#138)

* delete useless file

8 months agoimprove ci config (#136)
coderzc [Thu, 4 Nov 2021 09:21:12 +0000 (17:21 +0800)] 
improve ci config (#136)

8 months agoadd subKvEntry size to KvEntryWithFirstSubKv (#139)
ShouJing [Thu, 4 Nov 2021 07:22:56 +0000 (15:22 +0800)] 
add subKvEntry size to KvEntryWithFirstSubKv (#139)

* add subKvEntry size to KvEntryWithFirstSubKv

* bug fix and add unit test

8 months agofix: vertex at compute0() may be inactive (#134)
Jermy Li [Thu, 4 Nov 2021 02:30:04 +0000 (10:30 +0800)] 
fix: vertex at compute0() may be inactive (#134)

8 months agofix: output string value of log-output (#133)
Jermy Li [Wed, 3 Nov 2021 09:56:07 +0000 (17:56 +0800)] 
fix: output string value of log-output (#133)

* fix: output string value of log-output

8 months agounify input/output dir name to lowercase letters (#132)
Jermy Li [Wed, 3 Nov 2021 09:50:00 +0000 (17:50 +0800)] 
unify input/output dir name to lowercase letters (#132)

* unify input/output dir name to lowercase letters

8 months agofix message_stat not reset each superstep (#131)
Jermy Li [Thu, 28 Oct 2021 11:42:13 +0000 (19:42 +0800)] 
fix message_stat not reset each superstep (#131)

8 months agofix: remove no exist path from merge source paths (#130)
coderzc [Wed, 27 Oct 2021 06:10:11 +0000 (14:10 +0800)] 
fix: remove no exist path from merge source paths (#130)

8 months agoadd operator deploy readme (#123)
coderzc [Tue, 26 Oct 2021 09:25:35 +0000 (17:25 +0800)] 
add operator deploy readme (#123)

8 months agorefactor ComputerOutput (#128)
Jermy Li [Tue, 26 Oct 2021 06:29:44 +0000 (14:29 +0800)] 
refactor ComputerOutput (#128)

* refactor ComputerOutput

* fix HdfsOutput

8 months agofix messege-count calculation error (#127)
Jermy Li [Mon, 25 Oct 2021 11:24:27 +0000 (19:24 +0800)] 
fix messege-count calculation error (#127)

8 months agoimplement LPA algorithm (#110)
Jermy Li [Mon, 25 Oct 2021 09:31:02 +0000 (17:31 +0800)] 
implement LPA algorithm (#110)

* implement LPA algorithm

* add LPA algorithm test

8 months agoimprove value interface (#126)
coderzc [Mon, 25 Oct 2021 04:02:45 +0000 (12:02 +0800)] 
improve value interface (#126)

8 months agoUse version 3.1.2 of Hadoop (#124)
coderzc [Thu, 21 Oct 2021 03:23:39 +0000 (11:23 +0800)] 
Use version 3.1.2 of Hadoop (#124)

* improve hadoop pom

* improve hadoop version

8 months agorelease file descriptor after input and compute (#120)
ShouJing [Thu, 21 Oct 2021 03:13:54 +0000 (11:13 +0800)] 
release file descriptor after input and compute (#120)

8 months agoadd fetch input splits timeout (#113)
ShouJing [Mon, 18 Oct 2021 08:59:25 +0000 (16:59 +0800)] 
add fetch input splits timeout (#113)

8 months agoadd hdfs output (#114)
coderzc [Mon, 18 Oct 2021 02:05:31 +0000 (10:05 +0800)] 
add hdfs output (#114)

* add hsfd output
* statistics execute cost
* fix problems with using the same Config object for SenderIntegrateTest

8 months agofix bug: close file stream ahead at input stage (#118)
ShouJing [Fri, 15 Oct 2021 02:09:45 +0000 (10:09 +0800)] 
fix bug: close file stream ahead at input stage (#118)

* delay close file stream

* tiny improve

8 months agoinject more env var when start-computer.sh (#115)
coderzc [Thu, 14 Oct 2021 06:32:13 +0000 (14:32 +0800)] 
inject more env var when start-computer.sh (#115)

8 months agoFix file descriptor leak (#116)
ShouJing [Thu, 14 Oct 2021 03:54:33 +0000 (11:54 +0800)] 
Fix file descriptor leak (#116)

8 months agoupdate `input.edge_freq` default value to MULTIPLE (#106)
ShouJing [Tue, 12 Oct 2021 12:22:30 +0000 (20:22 +0800)] 
update `input.edge_freq` default value to MULTIPLE (#106)

8 months agofix some warnings (#111)
Jermy Li [Sat, 9 Oct 2021 06:45:27 +0000 (14:45 +0800)] 
fix some warnings (#111)

* fix some warnings
* improve algorithm test

9 months agosplit SortManager to SendSortManager and RecvSortManager (#105)
coderzc [Tue, 28 Sep 2021 06:18:40 +0000 (14:18 +0800)] 
split SortManager to SendSortManager and RecvSortManager (#105)

9 months agoimprove number of transport threads and max_timeout_heartbeat_count (#103)
coderzc [Sun, 26 Sep 2021 08:35:20 +0000 (16:35 +0800)] 
improve number of transport threads and max_timeout_heartbeat_count (#103)

9 months agoimplement rings-detection algorithm (#95)
ShouJing [Sun, 26 Sep 2021 07:21:44 +0000 (15:21 +0800)] 
implement rings-detection algorithm (#95)

* add cycle detection algorithm
* improve: add field label in Vertex and Edge
* support properties filter in RingsDetection
* add StringValue
* add method object in Value
* update built-in object name in SpreadFilter and improve code style
* add standard rings-detection algorithm
* add RingsDetectionOutput
* resolve circular dependencies
* add message type IdListList rings-detection

9 months agoImplement triangle_count algorithms and improve some code (#101)
coderzc [Fri, 24 Sep 2021 07:58:27 +0000 (15:58 +0800)] 
Implement triangle_count algorithms and improve some code (#101)

* Implement triangle_count algorithms
* Improve some code

9 months agouse session timeout option for sendControlMessageToWorkers (#92)
coderzc [Mon, 13 Sep 2021 09:58:45 +0000 (17:58 +0800)] 
use session timeout option for sendControlMessageToWorkers (#92)

9 months agoimprove k8s resource allocation (#100)
coderzc [Mon, 13 Sep 2021 08:29:25 +0000 (16:29 +0800)] 
improve k8s resource allocation (#100)

* add DEFAULT_TRANSPORT_PORT
* split master and worker resource quantity
* transport threads don't exceed the count of workers

9 months agobugfix: calculate fileOffset error in method BufferFileInput#readFully when read...
ShouJing [Sun, 12 Sep 2021 13:06:55 +0000 (21:06 +0800)] 
bugfix: calculate fileOffset error in method BufferFileInput#readFully when read length is bigger than buffer capacity (#98)

9 months agoadd clear graph data method in unit test (#99)
ShouJing [Sun, 12 Sep 2021 09:04:07 +0000 (17:04 +0800)] 
add clear graph data method in unit test (#99)

* add clear graph data method

9 months agoFix: params of string format error (#97)
zyxxoo [Tue, 7 Sep 2021 05:48:19 +0000 (13:48 +0800)] 
Fix: params of string format error (#97)

Co-authored-by: vaughn.zhang <vaughn.zhang@zoom.us>
10 months agostandardized algorithm package and name (#96)
Jermy Li [Mon, 30 Aug 2021 13:10:23 +0000 (21:10 +0800)] 
standardized algorithm package and name (#96)

10 months agoimprove wcc test and add property(key) (#94)
coderzc [Fri, 27 Aug 2021 07:18:54 +0000 (15:18 +0800)] 
improve wcc test and add property(key) (#94)

* improve wcc test
* add property(key) for vertex and edge

10 months agoFix the count of input edge less than actual (#93)
Linary [Fri, 27 Aug 2021 03:15:48 +0000 (11:15 +0800)] 
Fix the count of input edge less than actual (#93)

10 months agooutput demo(pagerank) (#72)
zhoney [Thu, 26 Aug 2021 12:28:39 +0000 (20:28 +0800)] 
output demo(pagerank) (#72)

* implement hugegraph pagerank output class
* support multiple threads writeback
* add output configs to driver

10 months agofix FileGraphPartition and some warnings (#90)
Jermy Li [Tue, 17 Aug 2021 03:21:43 +0000 (11:21 +0800)] 
fix FileGraphPartition and some warnings (#90)

* improve FileGraphPartition
* fix warnings
* fix warnings of test
* fix MessageInputTest
* fix tow workers shared one data_dirs

10 months agochore: remove deleted files (#91)
zyxxoo [Mon, 16 Aug 2021 08:24:28 +0000 (16:24 +0800)] 
chore: remove deleted files (#91)

10 months agosort different type of message use different combiner (#85)
ShouJing [Mon, 16 Aug 2021 07:41:10 +0000 (15:41 +0800)] 
sort different type of message use different combiner (#85)

* fix bug: sort use Pointer but combiner used to combine Value or Property
* fix bug: merge to one file until Sorter#iterator is perfected

10 months agofix some k8s bug and improve some code (#83)
coderzc [Mon, 16 Aug 2021 03:37:28 +0000 (11:37 +0800)] 
fix some k8s bug and improve some code (#83)

* fix the pod deleted when job failed
* fix job active state inaccurate
* upgrade fabric8 k8s client version to 5.6.0
* add job cost statistics
* add AUTO_DESTROY_POD option for operator
* improve k8s error log message
* use a separate thread for execution waitJob on unit test
* delete history job before run k8s test

10 months agoremove etcd (#86)
coderzc [Thu, 12 Aug 2021 07:54:29 +0000 (15:54 +0800)] 
remove etcd (#86)

10 months agofeat: add degree centrality (#77)
zyxxoo [Wed, 11 Aug 2021 12:56:34 +0000 (20:56 +0800)] 
feat: add degree centrality (#77)

* feat: add degree centrality

* feat: add degree centrality

* feat: cal weight degreen centrality

* fix: params bug

* chore: improve style

* chore: handle more hugeType for weight property and add output test to test result

* refactor: number value extends Number type

* chore: improve code style

* chore: add TODO

* modify job id

* fix test

Co-authored-by: coderzc <zc1217zc@126.com>
10 months agoUse BytesId unify the Id classes (#78)
Linary [Wed, 11 Aug 2021 09:16:37 +0000 (17:16 +0800)] 
Use BytesId unify the Id classes (#78)

10 months agoadd wcc algorithm (#81)
houzhizhen [Wed, 11 Aug 2021 07:39:52 +0000 (15:39 +0800)] 
add wcc algorithm (#81)

10 months agoAdd mvn release plugin (#82)
Linary [Mon, 9 Aug 2021 08:18:10 +0000 (16:18 +0800)] 
Add mvn release plugin (#82)

10 months agoFix bug: last edge attched in currVertex missed to send (#80)
Linary [Mon, 9 Aug 2021 04:53:11 +0000 (12:53 +0800)] 
Fix bug: last edge attched in currVertex missed to send (#80)

10 months agofeat: add startup and make docker image code (#71)
zyxxoo [Fri, 6 Aug 2021 08:42:05 +0000 (16:42 +0800)] 
feat: add startup and make docker image code (#71)

10 months agoadd LICENSE for hugegraph-computer (#79)
coderzc [Fri, 6 Aug 2021 05:26:22 +0000 (13:26 +0800)] 
add LICENSE for hugegraph-computer (#79)

11 months agoadd page-rank algorithm (#70)
houzhizhen [Thu, 5 Aug 2021 10:54:42 +0000 (18:54 +0800)] 
add page-rank algorithm (#70)

* add page-rank algorithm
* Set algorithm's parameters in ComputerContextUtil#initContext
* register ComputerOptions to overwrite the config in driver.config.ComputerOptions

11 months agoimprove k8s operator test (#76)
coderzc [Tue, 3 Aug 2021 09:02:40 +0000 (17:02 +0800)] 
improve k8s operator test (#76)

* improve k8s operator test

* * improve k8s operator test
* modify ci badge

* add k8s operator test

11 months agoadd computer-k8s and computer-k8s-operator (#56)
coderzc [Tue, 3 Aug 2021 03:28:57 +0000 (11:28 +0800)] 
add computer-k8s and computer-k8s-operator (#56)

11 months agoFix bug: some input splits missed in iteration (#75)
Linary [Tue, 3 Aug 2021 03:28:01 +0000 (11:28 +0800)] 
Fix bug: some input splits missed in iteration (#75)

11 months agoFix bug: Number id like int convert LongId failed (#74)
Linary [Fri, 30 Jul 2021 06:24:15 +0000 (14:24 +0800)] 
Fix bug: Number id like int convert LongId failed (#74)

11 months agofix: use ip instead of hostname due to can't resolved on k8s (#73)
coderzc [Wed, 28 Jul 2021 03:24:40 +0000 (11:24 +0800)] 
fix: use ip instead of hostname due to can't resolved on k8s (#73)

11 months agoFix bug: judge whether a notice is needed is wrong for MessageQueue (#69)
Linary [Tue, 20 Jul 2021 08:15:06 +0000 (16:15 +0800)] 
Fix bug: judge whether a notice is needed is wrong for MessageQueue (#69)

11 months agoadd ComputeManager (#66)
houzhizhen [Wed, 7 Jul 2021 07:08:46 +0000 (15:08 +0800)] 
add ComputeManager (#66)

add class ComputeManager, FileGraphPartition, EdgesInput, MessageInput, ReusablePointer, VertexInput, LogOutput, MessageRecvStat
add class ComputeTestSuite, ComputeManagerTest, MockComputation, MockMessageSender, EdgesInputTest, EmptyEdgesTest, MessageInputTest
add method copy in IdValueList.
remove mergeGraph from MessageRecvManager;
add method recvStat in MessageRecvPartition
add method recvStats in MessageRecvPartitions
add methods merge(recvStat)recvStat in Partition
WorkerService links compute and output to corresponding methods in ComputeManager
move method writeMessage from ComputeMessageRecvPartitionTest to ReceiverUtil

12 months agoadd seqfile (#65)
ShouJing [Mon, 5 Jul 2021 08:31:58 +0000 (16:31 +0800)] 
add seqfile (#65)

* add ValueFileOutput/ValueFileInput
* add BitsFileReader/BitFileWriter
* add AbstractBufferedFileInput/AbstractBufferedFileOutput

12 months agochore: add issue template & auto stale issues and pr (#64)
imbajin [Mon, 5 Jul 2021 08:22:58 +0000 (16:22 +0800)] 
chore: add issue template & auto stale issues and pr (#64)

12 months agofix copyToByteArray(): remove duplicate() (#67)
coderzc [Mon, 5 Jul 2021 07:04:30 +0000 (15:04 +0800)] 
fix copyToByteArray(): remove duplicate() (#67)

12 months agoAdd test for MultiQueue take in wait status (#68)
Linary [Mon, 5 Jul 2021 06:27:28 +0000 (14:27 +0800)] 
Add test for MultiQueue take in wait status (#68)

12 months agoLet Sender and Receiver process compute message with type MESSAGE_CLASS (#63)
houzhizhen [Tue, 22 Jun 2021 08:53:10 +0000 (16:53 +0800)] 
Let Sender and Receiver process compute message with type MESSAGE_CLASS (#63)

* delete ValueFactory
* rename VALUE_CLASS to ALGORITHM_RESULT_CLASS, remove VALUE_TYPE
* add option OUTPUT_VALUE_NAME
* check channel not null in QueuedMessageSender#init

12 months agoFix send thread may blocked all the time (#61)
Linary [Mon, 21 Jun 2021 06:22:46 +0000 (14:22 +0800)] 
Fix send thread may blocked all the time (#61)

* Fix send thread may blocked all the time
* Add back MultiQueue class

12 months agoImplement StreamGraphInput and EntryInput (#60)
Linary [Fri, 18 Jun 2021 08:25:37 +0000 (16:25 +0800)] 
Implement StreamGraphInput and EntryInput (#60)

12 months agoImplement data receiver of worker (#51)
houzhizhen [Fri, 18 Jun 2021 05:40:23 +0000 (13:40 +0800)] 
Implement data receiver of worker (#51)

* add DataServerManager to manage the data server.
* add MessageRecvBuffers the manages buffers with limit size.
* add MessageRecvPartition to manage the data received for a partition and three subclass
  (VertexMessageRecvPartition,EdgeMessageRecvPartition, ComputeMessageRecvPartition).
* add MessageRecvPartitions to manage the partitions of data received, and three subclass
  (VertexMessageRecvPartitions,EdgeMessageRecvPartitions, ComputeMessageRecvPartitions)
* add MessageRecvManager to implement MessageHandler to handle network messages.
* process start-finish event.
* Partition add iterator and sort logic.
* add class SuperstepFileGenerator.
* create ConnectionManager in WorkerService.
* add mergeBuffers using sort-mamager.

12 months agoUse github action for ci via ci.yml (#62)
Linary [Fri, 18 Jun 2021 03:34:04 +0000 (11:34 +0800)] 
Use github action for ci via ci.yml (#62)

12 months agoLink managers in worker sending end (#52)
Linary [Tue, 15 Jun 2021 07:35:58 +0000 (15:35 +0800)] 
Link managers in worker sending end (#52)

* Link managers in worker sending end
* Optimize WriteBufferPool and SortedBufferQueuePool
* Rafact MessageSendPartition, WriteBuffer and Edge
* Refact writeEdge with label and name
* Let SortManager don't need Sender
* Refact GraphComputeOutput and GraphWritebackOutput
* Refact OptimizedBytesOutput
* Add Input/Output Factory
* Refact writeFixedInt in sort module
* Move pending message into MessageQueue
* Define a MultiQueue support multi put

12 months agoAdd sessionActive for client (#59)
coderzc [Thu, 10 Jun 2021 07:50:08 +0000 (15:50 +0800)] 
Add sessionActive for client (#59)

13 months agoFix EntriesUtil#entryFromInput and some improve (#58)
ShouJing [Fri, 4 Jun 2021 11:45:57 +0000 (19:45 +0800)] 
Fix EntriesUtil#entryFromInput and some improve (#58)

13 months agoadd PointerCombiner (#57)
houzhizhen [Thu, 3 Jun 2021 06:30:36 +0000 (14:30 +0800)] 
add PointerCombiner (#57)

13 months agoAdd store/sort module (#37)
ShouJing [Mon, 31 May 2021 06:51:32 +0000 (14:51 +0800)] 
Add store/sort module (#37)

* add hgkvFile/hgkvDir module
* add EntriesUtil.java
* add SorterTest.java
* move file verification to HgkvDir
* rebuild HgkvFile readFooter and Fix HgkvDir#open
* add sort module
* implement InnerSortFlusher and OuterSortFlusher
* add KvEntry#numSubEntries

13 months agoadd startAsync and finishAsync on client side, add onStarted and onFinished on server...
coderzc [Mon, 31 May 2021 06:33:01 +0000 (14:33 +0800)] 
add startAsync and finishAsync on client side, add onStarted and onFinished on server side (#53)

* add startAsync and finishAsync on client side
* add onStarted and onFinished on server side

13 months agorename DataFileManager to FileManager (#54)
houzhizhen [Fri, 28 May 2021 06:26:06 +0000 (14:26 +0800)] 
rename DataFileManager to FileManager (#54)

* rename DataFileManager to FileManager

* remove #nextFile() from FileGenerator

13 months agoimplement aggregator module (#46)
Jermy Li [Fri, 21 May 2021 07:50:09 +0000 (15:50 +0800)] 
implement aggregator module (#46)

* implement aggregator module
* add DefaultAggregator.newValue()
* integrate aggregator into master-service and worker-service
* ensure worker init() executed after master
* add rpc interface registeredAggregators()
* also add two-tier structure to store current-step and previous-step aggregators
* add repair() for Aggregator
* let master context use config of outer class
* support register aggregator with default value
* add Manager.inited()
* remove worker_id option and use server_port instead
* use rpc-url from master-info

13 months agoadd copy() and assign() methods to Value class (#49)
Jermy Li [Wed, 19 May 2021 11:56:19 +0000 (19:56 +0800)] 
add copy() and assign() methods to Value class (#49)

* add Value.copy()+assign() method
* add IntValueSumCombiner and FloatValueSumCombiner

13 months agoStandardize bsp synchronization between master and worker (#48)
houzhizhen [Thu, 13 May 2021 11:45:31 +0000 (19:45 +0800)] 
Standardize bsp synchronization between master and worker (#48)

13 months agolog job-id in BSP client (#50)
Jermy Li [Thu, 13 May 2021 09:28:43 +0000 (17:28 +0800)] 
log job-id in BSP client (#50)

* log job-id in BSP client

* set BspBase.init() private

13 months agoRefactor Writable and Readable class (#42)
Linary [Wed, 12 May 2021 04:02:47 +0000 (12:02 +0800)] 
Refactor Writable and Readable class (#42)

* Refactor Writable and Readable class

* Also simplify GraphOut and GraphInput

* remove ComputeException

* Add test for duplicate method

* fix write/read properties value type incorrect

13 months agofix some warning, improve start() and finish() (#44)
coderzc [Tue, 11 May 2021 10:06:31 +0000 (18:06 +0800)] 
fix some warning, improve start() and finish() (#44)

* fix some warning
* improve some log
* improve start/finish send to avoid timeout
* blocking -> flowBlocking

13 months agostandard log format of bsp module and improve some error message (#47)
Jermy Li [Mon, 10 May 2021 07:04:14 +0000 (15:04 +0800)] 
standard log format of bsp module and improve some error message (#47)

13 months agoadd DataDirManager (#43)
houzhizhen [Sat, 8 May 2021 11:25:10 +0000 (19:25 +0800)] 
add DataDirManager (#43)

13 months agoadd createId methods in GraphFactory (#45)
houzhizhen [Sat, 8 May 2021 11:24:41 +0000 (19:24 +0800)] 
add createId methods in GraphFactory (#45)

14 months agoadd flowControl, heartbeat (#40)
coderzc [Thu, 6 May 2021 03:37:56 +0000 (11:37 +0800)] 
add flowControl, heartbeat (#40)

* add flow control
* add heartBeat
* add task to check ack
* add task to check ack and remove lastAckTimestamp
* rest BarrierEvent
* remove AckType
* add checkMinPendingRequests
* use accumulate flush
* add timeout parameter method to startSession and finishSession
* use real ip start
* scheduleAtFixedRate -> scheduleWithFixedDelay
* add maxHeartbeatTimes
* add dataMessage release()
* max_heartbeat_timeouts -> max_timeout_heartbeat_count
* set backlog default value 511

14 months agocreate api module (#41)
houzhizhen [Thu, 29 Apr 2021 09:13:53 +0000 (17:13 +0800)] 
create api module (#41)

* move algorithm's used classes to api
* add module computer-api
* signal bsp at end of init in WorkerService and MasterService
* make ValueFactory abstract and add BuiltinValueFactory
* rename USER_DEFINED to CUSTOM_VALUE in ValueType
* Reduce call ComputerContext.instance() directly
* synchronious UnitTestBase.updateWithRequiredOptions and return Config
* pass Aggregator4Master to MasterComputation#init
* add parameter MasterContext in MasterComputation.close

14 months agorename FilterMapComputation to FilterComputation (#38)
houzhizhen [Fri, 16 Apr 2021 10:18:36 +0000 (18:18 +0800)] 
rename FilterMapComputation to FilterComputation (#38)

14 months agoadd transport server, client, encode, decode (#25)
Coderzc [Fri, 16 Apr 2021 07:54:13 +0000 (15:54 +0800)] 
add transport server, client, encode, decode (#25)

add transport server
add transport client
add encoder
add decoder

14 months agoImprove class for Input/Output and fix writeInt(position, v) in BufferedFileOutput...
ShouJing [Thu, 15 Apr 2021 02:58:14 +0000 (10:58 +0800)] 
Improve class for Input/Output and fix writeInt(position, v) in BufferedFileOutput (#32)

* Improve class for Input/Output and fix problem

* update compare method in BufferFileInput/UnsafeByteArrayInput

* add exception declaration on RandomAccessInput#compare

* Add test case for UnsafeByteArrayInput/BufferedFileInput compare method

* Check offset and lenght must be in buffer

* Improve local variable names in BufferedFileTest#testCompare

* Update argument check in UnsafeByteArrayInput#compare

14 months agoadd rpc support (#36)
Jermy Li [Wed, 14 Apr 2021 08:13:23 +0000 (16:13 +0800)] 
add rpc support (#36)

* add rpc server and client: MasterRpcManager,WorkerRpcManager
* add MasterInputManager and WorkerInputManager
* inject rpc service into GraphFetcher
* store managers through map instead of list
* add loadGraph() and mergeGraph() to WorkerInputManager
* add checkInited() for WorkerService and MasterService

14 months agoAdd close method for Input Fetcher (#35)
Linary [Tue, 13 Apr 2021 13:04:05 +0000 (21:04 +0800)] 
Add close method for Input Fetcher (#35)

Co-authored-by: ShouJing <1075185785@qq.com>
14 months agofix some warnings for computer-test (#34)
Jermy Li [Tue, 13 Apr 2021 12:39:11 +0000 (20:39 +0800)] 
fix some warnings for computer-test (#34)

14 months agofix some warnings (#33)
Jermy Li [Mon, 12 Apr 2021 12:16:05 +0000 (20:16 +0800)] 
fix some warnings (#33)

14 months agoadd WorkerService and MasterService (#27)
houzhizhen [Fri, 9 Apr 2021 06:44:13 +0000 (14:44 +0800)] 
add WorkerService and MasterService (#27)

* rename input to inputStep
* MasterService does not implement MasterContext
* add name in Combiner and Computation
* add category in Computation

14 months agocreate Heap/LoserTree (#26)
ShouJing [Wed, 7 Apr 2021 07:32:55 +0000 (15:32 +0800)] 
create Heap/LoserTree (#26)

* add InputsSorting interface and implementation

14 months agoadd Partitioner and HashPartitioner (#29)
houzhizhen [Wed, 7 Apr 2021 02:40:09 +0000 (10:40 +0800)] 
add Partitioner and HashPartitioner (#29)