ShouJing [Wed, 11 May 2022 01:59:52 +0000 (09:59 +0800)]
fix ci install hdfs
ShouJing [Wed, 11 May 2022 01:56:49 +0000 (09:56 +0800)]
update ci install hdsf
imbajin [Mon, 14 Mar 2022 06:40:28 +0000 (14:40 +0800)]
chore: support codeQL check (#183)
imbajin [Thu, 3 Mar 2022 11:27:24 +0000 (19:27 +0800)]
chore: use cla assistant to support robot pr (#182)
Co-authored-by: imbajin <imbajin@users.noreply.github.com>
coderzc [Wed, 9 Feb 2022 10:28:43 +0000 (18:28 +0800)]
expand some operator function (#179)
coderzc [Mon, 24 Jan 2022 11:50:47 +0000 (19:50 +0800)]
update hugegraph-client & hugegraph-rpc & hugegraph-common version (#180)
coderzc [Tue, 18 Jan 2022 08:58:50 +0000 (16:58 +0800)]
update java version to 11 (#170)
coderzc [Mon, 17 Jan 2022 10:14:47 +0000 (18:14 +0800)]
feature: support input from hdfs and support hdfs kerberos (#172)
* support input from local/hdfs file (based on loader)
* support hdfs kerberos for input/output
* support filter vertex for hdfs output
coderzc [Fri, 14 Jan 2022 04:03:50 +0000 (12:03 +0800)]
refactor: add common hugegraph-output and writeType config (#175)
ShouJing [Fri, 24 Dec 2021 07:14:40 +0000 (15:14 +0800)]
refactor combiner: add v3 as result param (#169)
Linary [Mon, 20 Dec 2021 07:56:58 +0000 (15:56 +0800)]
Implement betweenness centrality algorithm (#164)
coderzc [Tue, 14 Dec 2021 06:07:30 +0000 (14:07 +0800)]
add configmap and secret mount to container (#158)
Linary [Mon, 13 Dec 2021 15:07:21 +0000 (23:07 +0800)]
Implement closeness centrality algorithm (#104)
Jermy Li [Mon, 13 Dec 2021 06:15:39 +0000 (14:15 +0800)]
fix race condition of MessageRecvBuffers (#162)
* fix race condition of MessageRecvBuffers
* Separate sort-start event and sort-end event
coderzc [Mon, 13 Dec 2021 05:23:02 +0000 (13:23 +0800)]
improve: edge triggered sendAvailable (#140)
Jermy Li [Tue, 7 Dec 2021 14:18:32 +0000 (22:18 +0800)]
fix some warnings: k8s-test client not closed (#152)
coderzc [Sat, 4 Dec 2021 07:51:51 +0000 (15:51 +0800)]
fix build images (#160)
coderzc [Fri, 3 Dec 2021 06:14:42 +0000 (14:14 +0800)]
add docker pulls badge (#154)
* add docker pulls badge
* rename hugegraph-computer-based-algorithm to hugegraph-builtin-algorithms
* fix publish images ci
coderzc [Thu, 2 Dec 2021 03:15:46 +0000 (11:15 +0800)]
Publish docker images by actions (#153)
* publish docker images by actions
* rename computer-algorithm-based.jar to hugegraph-builtin-algorithms.jar
* add build_images.sh
ShouJing [Wed, 1 Dec 2021 13:12:04 +0000 (21:12 +0800)]
improve: close hgkvfile input immediately (#155)
ShouJing [Thu, 25 Nov 2021 11:52:12 +0000 (19:52 +0800)]
add license-maven-plugin (#151)
Jermy Li [Wed, 24 Nov 2021 09:07:37 +0000 (17:07 +0800)]
remove template param from Value class (#146)
coderzc [Tue, 23 Nov 2021 12:03:56 +0000 (20:03 +0800)]
async watch the k8s job (#148)
* async watch k8s job
* let cancelJob return bool
* update version to 0.1.1
coderzc [Tue, 23 Nov 2021 12:02:58 +0000 (20:02 +0800)]
release maven package by actions (#119)
ShouJing [Tue, 23 Nov 2021 03:55:54 +0000 (11:55 +0800)]
change expression engine (#149)
Jermy Li [Thu, 18 Nov 2021 07:15:35 +0000 (15:15 +0800)]
add default value() method for HugeGraphOutput (#144)
* add default value() method for HugeGraphOutput
* fix EXPECT_RINGS format
ShouJing [Wed, 17 Nov 2021 07:00:29 +0000 (15:00 +0800)]
improve: delete useless temp file (#142)
* remove useless temp file in merge file process and inputs file after merge
ShouJing [Wed, 17 Nov 2021 05:49:10 +0000 (13:49 +0800)]
fix rings detection unit test bug (#147)
Jermy Li [Tue, 16 Nov 2021 08:46:53 +0000 (16:46 +0800)]
implement kcore algorithm (#143)
coderzc [Wed, 10 Nov 2021 07:46:09 +0000 (15:46 +0800)]
clear vertex old files and edge old files on `beforeSuperstep(Constants.INPUT_SUPERSTEP + 1)` (#141)
coderzc [Fri, 5 Nov 2021 09:11:55 +0000 (17:11 +0800)]
delete useless old files (#138)
* delete useless file
coderzc [Thu, 4 Nov 2021 09:21:12 +0000 (17:21 +0800)]
improve ci config (#136)
ShouJing [Thu, 4 Nov 2021 07:22:56 +0000 (15:22 +0800)]
add subKvEntry size to KvEntryWithFirstSubKv (#139)
* add subKvEntry size to KvEntryWithFirstSubKv
* bug fix and add unit test
Jermy Li [Thu, 4 Nov 2021 02:30:04 +0000 (10:30 +0800)]
fix: vertex at compute0() may be inactive (#134)
Jermy Li [Wed, 3 Nov 2021 09:56:07 +0000 (17:56 +0800)]
fix: output string value of log-output (#133)
* fix: output string value of log-output
Jermy Li [Wed, 3 Nov 2021 09:50:00 +0000 (17:50 +0800)]
unify input/output dir name to lowercase letters (#132)
* unify input/output dir name to lowercase letters
Jermy Li [Thu, 28 Oct 2021 11:42:13 +0000 (19:42 +0800)]
fix message_stat not reset each superstep (#131)
coderzc [Wed, 27 Oct 2021 06:10:11 +0000 (14:10 +0800)]
fix: remove no exist path from merge source paths (#130)
coderzc [Tue, 26 Oct 2021 09:25:35 +0000 (17:25 +0800)]
add operator deploy readme (#123)
Jermy Li [Tue, 26 Oct 2021 06:29:44 +0000 (14:29 +0800)]
refactor ComputerOutput (#128)
* refactor ComputerOutput
* fix HdfsOutput
Jermy Li [Mon, 25 Oct 2021 11:24:27 +0000 (19:24 +0800)]
fix messege-count calculation error (#127)
Jermy Li [Mon, 25 Oct 2021 09:31:02 +0000 (17:31 +0800)]
implement LPA algorithm (#110)
* implement LPA algorithm
* add LPA algorithm test
coderzc [Mon, 25 Oct 2021 04:02:45 +0000 (12:02 +0800)]
improve value interface (#126)
coderzc [Thu, 21 Oct 2021 03:23:39 +0000 (11:23 +0800)]
Use version 3.1.2 of Hadoop (#124)
* improve hadoop pom
* improve hadoop version
ShouJing [Thu, 21 Oct 2021 03:13:54 +0000 (11:13 +0800)]
release file descriptor after input and compute (#120)
ShouJing [Mon, 18 Oct 2021 08:59:25 +0000 (16:59 +0800)]
add fetch input splits timeout (#113)
coderzc [Mon, 18 Oct 2021 02:05:31 +0000 (10:05 +0800)]
add hdfs output (#114)
* add hsfd output
* statistics execute cost
* fix problems with using the same Config object for SenderIntegrateTest
ShouJing [Fri, 15 Oct 2021 02:09:45 +0000 (10:09 +0800)]
fix bug: close file stream ahead at input stage (#118)
* delay close file stream
* tiny improve
coderzc [Thu, 14 Oct 2021 06:32:13 +0000 (14:32 +0800)]
inject more env var when start-computer.sh (#115)
ShouJing [Thu, 14 Oct 2021 03:54:33 +0000 (11:54 +0800)]
Fix file descriptor leak (#116)
ShouJing [Tue, 12 Oct 2021 12:22:30 +0000 (20:22 +0800)]
update `input.edge_freq` default value to MULTIPLE (#106)
Jermy Li [Sat, 9 Oct 2021 06:45:27 +0000 (14:45 +0800)]
fix some warnings (#111)
* fix some warnings
* improve algorithm test
coderzc [Tue, 28 Sep 2021 06:18:40 +0000 (14:18 +0800)]
split SortManager to SendSortManager and RecvSortManager (#105)
coderzc [Sun, 26 Sep 2021 08:35:20 +0000 (16:35 +0800)]
improve number of transport threads and max_timeout_heartbeat_count (#103)
ShouJing [Sun, 26 Sep 2021 07:21:44 +0000 (15:21 +0800)]
implement rings-detection algorithm (#95)
* add cycle detection algorithm
* improve: add field label in Vertex and Edge
* support properties filter in RingsDetection
* add StringValue
* add method object in Value
* update built-in object name in SpreadFilter and improve code style
* add standard rings-detection algorithm
* add RingsDetectionOutput
* resolve circular dependencies
* add message type IdListList rings-detection
coderzc [Fri, 24 Sep 2021 07:58:27 +0000 (15:58 +0800)]
Implement triangle_count algorithms and improve some code (#101)
* Implement triangle_count algorithms
* Improve some code
coderzc [Mon, 13 Sep 2021 09:58:45 +0000 (17:58 +0800)]
use session timeout option for sendControlMessageToWorkers (#92)
coderzc [Mon, 13 Sep 2021 08:29:25 +0000 (16:29 +0800)]
improve k8s resource allocation (#100)
* add DEFAULT_TRANSPORT_PORT
* split master and worker resource quantity
* transport threads don't exceed the count of workers
ShouJing [Sun, 12 Sep 2021 13:06:55 +0000 (21:06 +0800)]
bugfix: calculate fileOffset error in method BufferFileInput#readFully when read length is bigger than buffer capacity (#98)
ShouJing [Sun, 12 Sep 2021 09:04:07 +0000 (17:04 +0800)]
add clear graph data method in unit test (#99)
* add clear graph data method
zyxxoo [Tue, 7 Sep 2021 05:48:19 +0000 (13:48 +0800)]
Fix: params of string format error (#97)
Co-authored-by: vaughn.zhang <vaughn.zhang@zoom.us>
Jermy Li [Mon, 30 Aug 2021 13:10:23 +0000 (21:10 +0800)]
standardized algorithm package and name (#96)
coderzc [Fri, 27 Aug 2021 07:18:54 +0000 (15:18 +0800)]
improve wcc test and add property(key) (#94)
* improve wcc test
* add property(key) for vertex and edge
Linary [Fri, 27 Aug 2021 03:15:48 +0000 (11:15 +0800)]
Fix the count of input edge less than actual (#93)
zhoney [Thu, 26 Aug 2021 12:28:39 +0000 (20:28 +0800)]
output demo(pagerank) (#72)
* implement hugegraph pagerank output class
* support multiple threads writeback
* add output configs to driver
Jermy Li [Tue, 17 Aug 2021 03:21:43 +0000 (11:21 +0800)]
fix FileGraphPartition and some warnings (#90)
* improve FileGraphPartition
* fix warnings
* fix warnings of test
* fix MessageInputTest
* fix tow workers shared one data_dirs
zyxxoo [Mon, 16 Aug 2021 08:24:28 +0000 (16:24 +0800)]
chore: remove deleted files (#91)
ShouJing [Mon, 16 Aug 2021 07:41:10 +0000 (15:41 +0800)]
sort different type of message use different combiner (#85)
* fix bug: sort use Pointer but combiner used to combine Value or Property
* fix bug: merge to one file until Sorter#iterator is perfected
coderzc [Mon, 16 Aug 2021 03:37:28 +0000 (11:37 +0800)]
fix some k8s bug and improve some code (#83)
* fix the pod deleted when job failed
* fix job active state inaccurate
* upgrade fabric8 k8s client version to 5.6.0
* add job cost statistics
* add AUTO_DESTROY_POD option for operator
* improve k8s error log message
* use a separate thread for execution waitJob on unit test
* delete history job before run k8s test
coderzc [Thu, 12 Aug 2021 07:54:29 +0000 (15:54 +0800)]
remove etcd (#86)
zyxxoo [Wed, 11 Aug 2021 12:56:34 +0000 (20:56 +0800)]
feat: add degree centrality (#77)
* feat: add degree centrality
* feat: add degree centrality
* feat: cal weight degreen centrality
* fix: params bug
* chore: improve style
* chore: handle more hugeType for weight property and add output test to test result
* refactor: number value extends Number type
* chore: improve code style
* chore: add TODO
* modify job id
* fix test
Co-authored-by: coderzc <zc1217zc@126.com>
Linary [Wed, 11 Aug 2021 09:16:37 +0000 (17:16 +0800)]
Use BytesId unify the Id classes (#78)
houzhizhen [Wed, 11 Aug 2021 07:39:52 +0000 (15:39 +0800)]
add wcc algorithm (#81)
Linary [Mon, 9 Aug 2021 08:18:10 +0000 (16:18 +0800)]
Add mvn release plugin (#82)
Linary [Mon, 9 Aug 2021 04:53:11 +0000 (12:53 +0800)]
Fix bug: last edge attched in currVertex missed to send (#80)
zyxxoo [Fri, 6 Aug 2021 08:42:05 +0000 (16:42 +0800)]
feat: add startup and make docker image code (#71)
coderzc [Fri, 6 Aug 2021 05:26:22 +0000 (13:26 +0800)]
add LICENSE for hugegraph-computer (#79)
houzhizhen [Thu, 5 Aug 2021 10:54:42 +0000 (18:54 +0800)]
add page-rank algorithm (#70)
* add page-rank algorithm
* Set algorithm's parameters in ComputerContextUtil#initContext
* register ComputerOptions to overwrite the config in driver.config.ComputerOptions
coderzc [Tue, 3 Aug 2021 09:02:40 +0000 (17:02 +0800)]
improve k8s operator test (#76)
* improve k8s operator test
* * improve k8s operator test
* modify ci badge
* add k8s operator test
coderzc [Tue, 3 Aug 2021 03:28:57 +0000 (11:28 +0800)]
add computer-k8s and computer-k8s-operator (#56)
Linary [Tue, 3 Aug 2021 03:28:01 +0000 (11:28 +0800)]
Fix bug: some input splits missed in iteration (#75)
Linary [Fri, 30 Jul 2021 06:24:15 +0000 (14:24 +0800)]
Fix bug: Number id like int convert LongId failed (#74)
coderzc [Wed, 28 Jul 2021 03:24:40 +0000 (11:24 +0800)]
fix: use ip instead of hostname due to can't resolved on k8s (#73)
Linary [Tue, 20 Jul 2021 08:15:06 +0000 (16:15 +0800)]
Fix bug: judge whether a notice is needed is wrong for MessageQueue (#69)
houzhizhen [Wed, 7 Jul 2021 07:08:46 +0000 (15:08 +0800)]
add ComputeManager (#66)
add class ComputeManager, FileGraphPartition, EdgesInput, MessageInput, ReusablePointer, VertexInput, LogOutput, MessageRecvStat
add class ComputeTestSuite, ComputeManagerTest, MockComputation, MockMessageSender, EdgesInputTest, EmptyEdgesTest, MessageInputTest
add method copy in IdValueList.
remove mergeGraph from MessageRecvManager;
add method recvStat in MessageRecvPartition
add method recvStats in MessageRecvPartitions
add methods merge(recvStat)recvStat in Partition
WorkerService links compute and output to corresponding methods in ComputeManager
move method writeMessage from ComputeMessageRecvPartitionTest to ReceiverUtil
ShouJing [Mon, 5 Jul 2021 08:31:58 +0000 (16:31 +0800)]
add seqfile (#65)
* add ValueFileOutput/ValueFileInput
* add BitsFileReader/BitFileWriter
* add AbstractBufferedFileInput/AbstractBufferedFileOutput
imbajin [Mon, 5 Jul 2021 08:22:58 +0000 (16:22 +0800)]
chore: add issue template & auto stale issues and pr (#64)
coderzc [Mon, 5 Jul 2021 07:04:30 +0000 (15:04 +0800)]
fix copyToByteArray(): remove duplicate() (#67)
Linary [Mon, 5 Jul 2021 06:27:28 +0000 (14:27 +0800)]
Add test for MultiQueue take in wait status (#68)
houzhizhen [Tue, 22 Jun 2021 08:53:10 +0000 (16:53 +0800)]
Let Sender and Receiver process compute message with type MESSAGE_CLASS (#63)
* delete ValueFactory
* rename VALUE_CLASS to ALGORITHM_RESULT_CLASS, remove VALUE_TYPE
* add option OUTPUT_VALUE_NAME
* check channel not null in QueuedMessageSender#init
Linary [Mon, 21 Jun 2021 06:22:46 +0000 (14:22 +0800)]
Fix send thread may blocked all the time (#61)
* Fix send thread may blocked all the time
* Add back MultiQueue class
Linary [Fri, 18 Jun 2021 08:25:37 +0000 (16:25 +0800)]
Implement StreamGraphInput and EntryInput (#60)
houzhizhen [Fri, 18 Jun 2021 05:40:23 +0000 (13:40 +0800)]
Implement data receiver of worker (#51)
* add DataServerManager to manage the data server.
* add MessageRecvBuffers the manages buffers with limit size.
* add MessageRecvPartition to manage the data received for a partition and three subclass
(VertexMessageRecvPartition,EdgeMessageRecvPartition, ComputeMessageRecvPartition).
* add MessageRecvPartitions to manage the partitions of data received, and three subclass
(VertexMessageRecvPartitions,EdgeMessageRecvPartitions, ComputeMessageRecvPartitions)
* add MessageRecvManager to implement MessageHandler to handle network messages.
* process start-finish event.
* Partition add iterator and sort logic.
* add class SuperstepFileGenerator.
* create ConnectionManager in WorkerService.
* add mergeBuffers using sort-mamager.
Linary [Fri, 18 Jun 2021 03:34:04 +0000 (11:34 +0800)]
Use github action for ci via ci.yml (#62)
Linary [Tue, 15 Jun 2021 07:35:58 +0000 (15:35 +0800)]
Link managers in worker sending end (#52)
* Link managers in worker sending end
* Optimize WriteBufferPool and SortedBufferQueuePool
* Rafact MessageSendPartition, WriteBuffer and Edge
* Refact writeEdge with label and name
* Let SortManager don't need Sender
* Refact GraphComputeOutput and GraphWritebackOutput
* Refact OptimizedBytesOutput
* Add Input/Output Factory
* Refact writeFixedInt in sort module
* Move pending message into MessageQueue
* Define a MultiQueue support multi put
coderzc [Thu, 10 Jun 2021 07:50:08 +0000 (15:50 +0800)]
Add sessionActive for client (#59)
ShouJing [Fri, 4 Jun 2021 11:45:57 +0000 (19:45 +0800)]
Fix EntriesUtil#entryFromInput and some improve (#58)
houzhizhen [Thu, 3 Jun 2021 06:30:36 +0000 (14:30 +0800)]
add PointerCombiner (#57)
ShouJing [Mon, 31 May 2021 06:51:32 +0000 (14:51 +0800)]
Add store/sort module (#37)
* add hgkvFile/hgkvDir module
* add EntriesUtil.java
* add SorterTest.java
* move file verification to HgkvDir
* rebuild HgkvFile readFooter and Fix HgkvDir#open
* add sort module
* implement InnerSortFlusher and OuterSortFlusher
* add KvEntry#numSubEntries
coderzc [Mon, 31 May 2021 06:33:01 +0000 (14:33 +0800)]
add startAsync and finishAsync on client side, add onStarted and onFinished on server side (#53)
* add startAsync and finishAsync on client side
* add onStarted and onFinished on server side