ASF Git Repos - giraph.git/log

GIRAPH-1124 - Create documentation on how to make Giraph release

Test Plan: mvn clean site

Reviewers: rvs, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D65313

commit | commitdiff | tree

Sergey Edunov [Mon, 14 Nov 2016 19:58:59 +0000 (11:58 -0800)]

Fix Checkstyle

Test Plan:
mvn clean site -DskipTests -Phadoop_2 -Ddependency.locations.enabled=false
mvn clean install -Phadoop_2 -Prelease
mvn clean install -Phadoop_1 -Prelease

Reviewers: dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D65499

commit | commitdiff | tree

Sergey Edunov [Tue, 25 Oct 2016 00:20:02 +0000 (17:20 -0700)]

Bump Apache Giraph version to 1.3.0-SNAPSHOT

Test Plan: none

Reviewers: dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D65391

commit | commitdiff | tree

Sergey Edunov [Fri, 14 Oct 2016 20:43:50 +0000 (13:43 -0700)]

GIRAPH-1122 Javadoc generation fails for Giraph 1.2.0

Test Plan: mvn clean site -DskipTests -Phadoop_2 -Ddependency.locations.enabled=false

Reviewers: majakabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D64995

commit | commitdiff | tree

Sergey Edunov [Tue, 11 Oct 2016 23:49:16 +0000 (16:49 -0700)]

Fixing RAT checks for Apache Giraph release

Test Plan:
mvn apache-rat:check -Phadoop_2
mvn apache-rat:check -Phadoop_1
mvn clean verify -Phadoop_facebook

Reviewers: maja.kabiljo, majakabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D64917

commit | commitdiff | tree

Sergey Edunov [Thu, 6 Oct 2016 18:13:49 +0000 (11:13 -0700)]

GIRAPH-1118 - Giraph-gora and Giraph-rexster test cases fail in release-1.2

Test Plan:
mvn clean verify -Phadoop_facebook
rm -rf ~/.m2/repository/org/apache/giraph
mvn clean install -Phadoop_1
rm -rf ~/.m2/repository/org/apache/giraph
mvn clean install -Phadoop_2

Reviewers: maja.kabiljo, majakabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D64719

commit | commitdiff | tree

Sergey Edunov [Wed, 5 Oct 2016 22:05:58 +0000 (15:05 -0700)]

GIRAPH-1118 - Giraph-gora and Giraph-rexster test cases fail in release-1.2

Test Plan:
mvn clean verify -Phadoop_facebook
mvn clean install -Phadoop_1
mvn clean install -Phadoop_2

Reviewers: majakabiljo, dionysis.logothetis, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D64683

commit | commitdiff | tree

Sergey Edunov [Thu, 29 Sep 2016 23:54:18 +0000 (16:54 -0700)]

[GIRAPH-1117] Provide a flexible way to decide whether to create vertex when it is not present in the input

Test Plan: run hello pagerank with this feature on and off

Reviewers: majakabiljo, maja.kabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D64485

commit | commitdiff | tree

Sergey Edunov [Wed, 21 Sep 2016 21:47:10 +0000 (14:47 -0700)]

GIRAPH-1094 remove hbase1 from distribution for hadoop_1

Summary: Missed that part in the last diff.

Test Plan:
mvn clean package -Phadoop_2 -fae
then checked that giraph-hbase.jar is in the distribution

mvn clean package -Phadoop_1 -fae
then checked that giraph-hbase.jar is not in the distribution

Reviewers: maja.kabiljo, majakabiljo, dionysis.logothetis

Reviewed By: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D64203

commit | commitdiff | tree

Sergey Edunov [Wed, 21 Sep 2016 18:04:34 +0000 (11:04 -0700)]

GIRAPH-1094 Remove hbase from hadoop_1

Summary: Hadoop_1 and current versions of hbase are incompatible. Removing support for HBASE from Hadoop_1 profile

Test Plan: mvn clean package -Phadoop_1 -fae

Reviewers: majakabiljo, maja.kabiljo, dionysis.logothetis

Reviewed By: maja.kabiljo, dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D64197

commit | commitdiff | tree

Maja Kabiljo [Wed, 14 Sep 2016 23:35:22 +0000 (16:35 -0700)]

GIRAPH-1114: Expose StatusReporter from workers in blocks framework

Summary: Sometimes we need to call progress or update status from workers, expose this functionality

Test Plan: verify

Differential Revision: https://reviews.facebook.net/D63999

commit | commitdiff | tree

Maja Kabiljo [Mon, 19 Sep 2016 19:26:49 +0000 (12:26 -0700)]

GIRAPH-1115: Move UncaughtExceptionHandler setup to GraphTaskManager

Test Plan: Ran a job which isn't using GraphMapper and verified exception handler was set properly

Differential Revision: https://reviews.facebook.net/D64113

commit | commitdiff | tree

Sergey Edunov [Wed, 14 Sep 2016 17:20:25 +0000 (10:20 -0700)]

GIRAPH-1111 - FileOutputFormat#setOutputPath is not always available

Test Plan:
mvn clean install
+ run a few jobs

Reviewers: majakabiljo, dionysis.logothetis, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D63837

commit | commitdiff | tree

spupyrev [Wed, 31 Aug 2016 00:32:13 +0000 (17:32 -0700)]

faster maps

Summary:
The idea is to replace HashMap<LongWritable, V> to Long2ObjectOpenHashMap<V> (and Map<Int...> to Int2Object...)
This will save space and speed up some applications.

I changed the type of such a map in TestGraph.java, which gives up to 2x speed up on an
example of page rank computation (see comment below)

JIRA: https://issues.apache.org/jira/browse/GIRAPH-1049

Test Plan: TestBasicCollections.java contain some tests

Reviewers: sergey.edunov, maja.kabiljo, dionysis.logothetis, heslami, ikabiljo

Reviewed By: heslami

Differential Revision: https://reviews.facebook.net/D55587

commit | commitdiff | tree

Maja Kabiljo [Fri, 26 Aug 2016 20:51:37 +0000 (13:51 -0700)]

GIRAPH-1108: Allow measuring time spent doing GC in some interval

Summary: Sometimes when things are slow, we want to know whether it's because of GC or not. Keep track of last k GC pauses and a way to check how much time since some timestamp was spent doing GC.

Test Plan: Ran a job which periodically prints stats from this and manually verified based on GC logs that it's measuring it correctly

Differential Revision: https://reviews.facebook.net/D62727

commit | commitdiff | tree

Tyler Serdar Bulut [Tue, 30 Aug 2016 18:31:13 +0000 (13:31 -0500)]

Out-of-core is logging too aggressively

Summary:
Example aggressive logging at INFO level:

INFO    <datestamp> [ooc-io-0] org.apache.giraph.ooc.policy.ThresholdBasedOracle  - getNextIOActions: usedMemoryFraction = 0.79
INFO    <datestamp> [ooc-io-0] org.apache.giraph.ooc.OutOfCoreIOCallable  - call: thread 0's next IO command is: LoadPartitionIOCommand: (partitionId = 4676, superstep = 0)
INFO    <datestamp> [ooc-io-0] org.apache.giraph.ooc.OutOfCoreIOCallable  - call: thread 0's command LoadPartitionIOCommand: (partitionId = 4676, superstep = 0) completed: bytes= 0, duration=0, bandwidth=NaN, bandwidth (excluding GC time)=NaN

Test Plan: mvn clean verify -P hadoop_facebook

Reviewers: majakabiljo, maja.kabiljo, sergey.edunov, heslami

Reviewed By: heslami

Subscribers: dionysis.logothetis

Differential Revision: https://reviews.facebook.net/D62853

commit | commitdiff | tree

Maja Kabiljo [Mon, 8 Aug 2016 18:13:35 +0000 (11:13 -0700)]

GIRAPH-1103: Another try to fix jobs getting stuck after channel failure

Summary:
With GIRAPH-1087 we see jobs stuck after channel failure less often, but it still happens. There are several additional issues I found: requests failing to send at the first place so they never get retried, callbacks for channel failures not being triggered always.
Added a thread which will periodically check on open requests even when we are not waiting on all open requests (since in many places we don't), remove the check that request wass ent when retrying it, added some thread utils while at it.

Test Plan: Before the change, failure rate of a particular job was about 1 in 50. Had over 200 successful runs with this change.

Differential Revision: https://reviews.facebook.net/D61719

commit | commitdiff | tree

Maja Kabiljo [Tue, 23 Aug 2016 18:52:26 +0000 (11:52 -0700)]

GIRAPH-1107: Allow observers to access job counters

Summary: From mapper/master/worker observer we might want to update some job counters for stats. For that we should allow observers to access job context.

Test Plan: Ran a job which accesses counters from WorkerObserver

Reviewers: sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D62391

commit | commitdiff | tree

Maja Kabiljo [Fri, 12 Aug 2016 21:57:53 +0000 (14:57 -0700)]

GIRAPH-1105: Fix number of open requests in FacebookConfiguration

Test Plan: This was significantly better in some experiments, but we can investigate more in the future

Differential Revision: https://reviews.facebook.net/D62019

commit | commitdiff | tree

Maja Kabiljo [Wed, 10 Aug 2016 19:56:19 +0000 (12:56 -0700)]

GIRAPH-1104: NegativeArraySize exception in BigDataOutput

Summary:
BigDataIO is not properly handling large byte[] being written to it. Chunk them up when needed to respect the max single data output size.
With D61791 job was still failing with the same exception.

Test Plan: The job which was failing because of large edges now works, added a test

Differential Revision: https://reviews.facebook.net/D61839

commit | commitdiff | tree

Hassan Eslami [Tue, 26 Jul 2016 18:27:24 +0000 (11:27 -0700)]

Fixing Giraph pom.xml to reflect new project committers

Summary:
Fixed the list of project committers. Please review your information and let me know if I should change anything.

This will be the first diff that I'll be committing all by myself, more like a test to see my username is gone through Apache's internal :-)

Test Plan: N/A

Reviewers: ikabiljo, pavanka, avery.ching, sergey.edunov

Reviewed By: sergey.edunov

Differential Revision: https://reviews.facebook.net/D61197

commit | commitdiff | tree

Sergey Edunov [Wed, 20 Jul 2016 17:20:36 +0000 (10:20 -0700)]

GIRAPH-1098 Job may get stuck if zookeeper port fixed and is in use

Test Plan: mvn clean verify -Phadoop_facebook

Reviewers: majakabiljo, dionysis.logothetis, maja.kabiljo

Reviewed By: maja.kabiljo

Differential Revision: https://reviews.facebook.net/D60945

commit | commitdiff | tree

Maja Kabiljo [Tue, 12 Jul 2016 17:27:47 +0000 (10:27 -0700)]

GIRAPH-1087: Retry requests after channel failure

Summary: We currently don't have a callback to retry requests after channel failure, and would either wait for request timeout or not retrying request at all at places where we don't wait for open requests.

Test Plan: Hard to reproduce the issue (ran many jobs but was unable to), we'll see if the problem happens again in prod with this change.

Differential Revision: https://reviews.facebook.net/D60675

commit | commitdiff | tree

Sergey Edunov [Tue, 19 Jul 2016 00:30:04 +0000 (17:30 -0700)]

GIRAPH-1097 Fix TestOutOfCore.testOutOfCoreLocalDiskAccessor

Summary:
On my laptop it failed because of an NPE in WorkerSuperstepMetrics.
I tracked it down and found that it is triggered from the branch of code that prints out metrics. We don't normally print out metrics in unit tests, so I'd expect this feature doesn't exist or not functional in hadoop_1. I'll try to disable it, to see how jenkins reacts.

Test Plan: mvn test -pl giraph-examples -am -Dtest=TestOutOfCore -DfailIfNoTests=false -Phadoop_1

Reviewers: maja.kabiljo, dionysis.logothetis, heslami

Reviewed By: heslami

Differential Revision: https://reviews.facebook.net/D60873

Apache Giraph

RSS Atom