Jun He [Thu, 9 Aug 2018 05:49:09 +0000 (05:49 +0000)] 
CRUNCH-671: Failed to generate reports using "mvn site"
Crunch build failed due to "ClassNotFound" in doxia.
This is caused by maven-project-info-reports-plugin updated to 3.0.0, depends on
doxia-site-renderer 1.8 (which has org.apache.maven.doxia.siterenderer.DocumentContent
this class), while maven-site-plugin:3.3 depends on doxia-site-renderer:1.4 (which
doesn't have org.apache.maven.doxia.siterenderer.DocumentContent)
Specify maven-site-plugin to 3.7 can resolve this.
Signed-off-by: Jun He <jun.he@linaro.org>
Signed-off-by: Josh Wills <jwills@apache.org>
Josh Wills [Mon, 23 Jul 2018 20:31:00 +0000 (13:31 -0700)] 
CRUNCH-619: Update to HBase 2.0.1. Contributed by Attila Sasvari.
Josh Wills [Mon, 30 Apr 2018 18:47:15 +0000 (11:47 -0700)]
CRUNCH-669: Add an option to disable temp dir deletion in the finalize() method of a DistributedPipeline
Clément MATHIEU [Tue, 27 Mar 2018 15:55:15 +0000 (17:55 +0200)] 
CRUNCH-668: Support globbing patterns in From#avroFile
Signed-off-by: Josh Wills <jwills@apache.org>
Clément MATHIEU [Tue, 6 Mar 2018 16:47:48 +0000 (17:47 +0100)]
Fix HCatSourceITSpec.testBasic
Signed-off-by: Josh Wills <jwills@apache.org>
Clément MATHIEU [Wed, 7 Mar 2018 09:13:51 +0000 (10:13 +0100)]
CRUNCH-665: Add crunch.max.poll.interval property
Signed-off-by: Josh Wills <jwills@apache.org>
Nathan Schile [Mon, 5 Feb 2018 15:08:46 +0000 (09:08 -0600)] 
CRUNCH-664 Fixes HBase configuration properties being overwritten
Signed-off-by: Josh Wills <jwills@apache.org>
Ben Roling [Wed, 24 Jan 2018 16:40:18 +0000 (10:40 -0600)] 
Expose combine file split file path via Hadoop config
Signed-off-by: Josh Wills <jwills@apache.org>
Bryan Baugher [Wed, 24 Jan 2018 20:14:31 +0000 (14:14 -0600)] 
CRUNCH-662: Updated KafkaRecordReader to better handle errors, empty reads and appropriately retry
Signed-off-by: Josh Wills <jwills@apache.org>
Josh Wills [Thu, 18 Jan 2018 21:11:26 +0000 (13:11 -0800)]
CRUNCH-661: Make DataBaseSource.Builder methods public
Josh Wills [Mon, 11 Dec 2017 17:56:38 +0000 (09:56 -0800)]
CRUNCH-654: KafkaSource should use the new Kafka Consumer API instead of the SimpleConsumer. Contributed by Bryan Baugher.
Stephen Durfey [Mon, 4 Dec 2017 16:49:59 +0000 (10:49 -0600)] 
CRUNCH-340: added HCatSource & HCatTarget
Signed-off-by: Josh Wills <jwills@apache.org>
Stephen Durfey [Thu, 7 Dec 2017 15:55:56 +0000 (09:55 -0600)]
CRUNCH-659: updated hive dependency to 2.1
Signed-off-by: Micah Whitacre <mkwhit@gmail.com>
Josh Wills [Fri, 27 Oct 2017 04:09:27 +0000 (21:09 -0700)]
CRUNCH-652: Fix to make the SourceTargetHelperTest less flakey on hadoop 3.0.0. Contributed by Gergo Repas.
Bryan Baugher [Wed, 16 Aug 2017 21:19:42 +0000 (16:19 -0500)]
CRUNCH-653: Created KafkaSource that provides ConsumerRecord messages
Signed-off-by: Josh Wills <jwills@apache.org>
Josh Wills [Fri, 12 May 2017 16:52:49 +0000 (09:52 -0700)]
CRUNCH-647: Remove obsolete jackson dependencies
Gabriel Reid [Thu, 27 Apr 2017 12:52:16 +0000 (14:52 +0200)] 
CRUNCH-644 Supply preferred node for HFile writes
Designate the preferred HDFS data node when creating HFiles for
bulk load to improve data locality of the created HFiles.
Tom White [Thu, 13 Apr 2017 15:10:23 +0000 (16:10 +0100)] 
CRUNCH-618: Run on Spark 2. Contributed by Gergő Pásztor.
Xavier Talpe [Thu, 13 Apr 2017 05:52:43 +0000 (07:52 +0200)] 
CRUNCH-642 Enable GroupingOptions for Distinct operations.
This fixes the existing call for numReducers as it was not working as
intended for non-memory PCollections due to using an invalid amount
of numReducers. To increase flexibility when using the API,
another call was added that allow to directly pass the GroupingOptions.
Signed-off-by: Josh Wills <jwills@apache.org>
Tom White [Wed, 12 Apr 2017 14:03:41 +0000 (15:03 +0100)]
CRUNCH-641: Wrong decimal format in dot files. Contributed by Gergő Pásztor.
Xavier Talpe [Mon, 10 Apr 2017 13:51:32 +0000 (15:51 +0200)]
CRUNCH-642 Enable numReducers option for Distinct operations.
Signed-off-by: Josh Wills <jwills@apache.org>
Attila Sasvari [Thu, 23 Mar 2017 20:35:36 +0000 (21:35 +0100)] 
CRUNCH-636: amend Make replication factor for temporary files configurable
Signed-off-by: Josh Wills <jwills@apache.org>
Attila Sasvari [Mon, 20 Mar 2017 10:17:55 +0000 (11:17 +0100)]
CRUNCH-636: Make replication factor for temporary files configurable
Signed-off-by: Josh Wills <jwills@apache.org>
Tom White [Tue, 7 Mar 2017 14:38:52 +0000 (14:38 +0000)]
CRUNCH-638: Improve dot file generation for better supportability. Contributed by Gergő Pásztor.
Tom White [Mon, 20 Feb 2017 10:28:05 +0000 (10:28 +0000)]
CRUNCH-633: Remove the commons-httpclient:commons-httpclient dependency. Contributed by Gergő Pásztor.
Gabriel Reid [Mon, 13 Feb 2017 18:57:32 +0000 (19:57 +0100)]
CRUNCH-634 Fix typo in log message
Contributed by Attila Sasvari
Micah Whitacre [Wed, 7 Dec 2016 02:50:02 +0000 (21:50 -0500)] 
CRUNCH-628: Upgraded to Kafka 0.10.0.x
Josh Wills [Sun, 5 Feb 2017 19:22:06 +0000 (11:22 -0800)]
[maven-release-plugin] prepare for next development iteration
Josh Wills [Sun, 5 Feb 2017 19:22:05 +0000 (11:22 -0800)]
[maven-release-plugin] prepare branch apache-crunch-0.15
Micah Whitacre [Thu, 12 Jan 2017 02:51:26 +0000 (20:51 -0600)]
CRUNCH-632: Added support for compressed CSVSource files.
CRUNCH-632: Wrote simple test showing it now working on compressed CSV file.
Signed-off-by: Josh Wills <jwills@apache.org>
Micah Whitacre [Thu, 12 Jan 2017 01:53:05 +0000 (19:53 -0600)]
Merge branch 'CRUNCH-630'
Brian Tieman [Tue, 13 Dec 2016 15:01:08 +0000 (09:01 -0600)] 
CRUNCH-629: Kafka source pulling is aggressive
Added some parenthesis to force proper order of operations in KafkaRecordReader.
Signed-off-by: Micah Whitacre <mkwhit@gmail.com>
Micah Whitacre [Tue, 3 Jan 2017 17:39:31 +0000 (11:39 -0600)]
CRUNCH-630: set a better default for the situation where offsets are out of range.
Dimitry Goldin [Fri, 14 Oct 2016 16:39:41 +0000 (18:39 +0200)] 
Quick and Dirty Workaround for Crunch DistCache
Signed-off-by: Micah Whitacre <mkwhit@gmail.com>
Signed-off-by: Josh Wills <jwills@apache.org>
Josh Wills [Sat, 3 Dec 2016 19:56:59 +0000 (11:56 -0800)]
CRUNCH-622: From.avroFile fails if path not on default filesystem. Contributed by Micah Whitacre.
Stefan Mendoza [Tue, 13 Sep 2016 03:38:41 +0000 (22:38 -0500)] 
CRUNCH-620: Reduce "isn't a known config" warnings by slimming down ConsumerConfig properties
Resolved by tagging the Kafka connection properties so that the Kafka Consumers can be built with slimmer ConsumerConfig properties.
Signed-off-by: Micah Whitacre <mkwhit@gmail.com>
David Whiting [Thu, 20 Oct 2016 14:17:00 +0000 (16:17 +0200)] 
CRUNCH-625: Add missing .union implementations for LTables with LTables and PTables
Micah Whitacre [Tue, 13 Sep 2016 15:35:35 +0000 (10:35 -0500)]
CRUNCH-621: Added check into hasPendingData to check if there is a large number of requests with no data to make sure there is still data there.
Nathan Schile [Thu, 29 Sep 2016 21:24:14 +0000 (16:24 -0500)]
CRUNCH-623: Improves Javadoc of PTable#cogroup
Signed-off-by: Josh Wills <jwills@apache.org>
Micah Whitacre [Tue, 6 Sep 2016 20:55:56 +0000 (15:55 -0500)]
CRUNCH-617: Support defensively handling null when partition leader cannot be found.
Signed-off-by: Micah Whitacre <mkwhit@gmail.com>
Tom White [Thu, 8 Sep 2016 13:12:30 +0000 (14:12 +0100)]
CRUNCH-616: Replace (possibly copyrighted) Maugham text with Dickens. Contributed by Sean Owen.
Remove non-applicable Project Gutenberg license. Adjust lots of tests to match new text.
Josh Wills [Wed, 24 Aug 2016 17:59:14 +0000 (10:59 -0700)]
CRUNCH-601: Handle empty PCollections correctly in Crunch-on-Spark. Created by Micah Whitacre,
Mikael Goldmann, and Josh Wills.
Josh Wills [Wed, 24 Aug 2016 03:07:23 +0000 (20:07 -0700)]
CRUNCH-519: Add more detail to plan dot file. Contributed by Ron Hashimshony.
Micah Whitacre [Tue, 2 Aug 2016 21:29:55 +0000 (16:29 -0500)]
CRUNCH-604: Avoid expensive Writables.reloadWritableComparableCodes
Micah Whitacre [Tue, 2 Aug 2016 20:58:29 +0000 (15:58 -0500)]
CRUNCH-611: Corrected files that were missing the APL headers.
Micah Whitacre [Wed, 13 Jul 2016 15:18:17 +0000 (10:18 -0500)]
CRUNCH-611: Added API for Offset reading/writing along with a simple implementation that supports doing it from hdfs.
Signed-off-by: Micah Whitacre <mkwhit@apache.org>
Josh Wills [Sat, 30 Jul 2016 00:47:09 +0000 (17:47 -0700)]
CRUNCH-614: Fix HFileUtils.writeToHFilesForIncrementalLoad slowed dramatically by copying KeyValue byte array. Contributed by Ben Roling.
Clément MATHIEU [Tue, 19 Jul 2016 19:30:35 +0000 (21:30 +0200)]
CRUNCH-613: Fix FileTargetImplTest.testHandleOutputsMovesFilesToDestination instability
Signed-off-by: Micah Whitacre <mkwhit@gmail.com>
CRUNCH-613: Fixed up the test to consolidate constants used.
Clément MATHIEU [Tue, 19 Jul 2016 19:01:20 +0000 (21:01 +0200)]
CRUNCH-612: Add support of private ctors to AvroDeepCopier
Signed-off-by: Micah Whitacre <mkwhit@gmail.com>
Micah Whitacre [Tue, 28 Jun 2016 20:44:15 +0000 (15:44 -0500)]
CRUNCH-609: Improved KafkaRecordReader to keep retrying when the range of offsets has not been fully consumed.
Micah Whitacre [Mon, 23 May 2016 20:13:02 +0000 (15:13 -0500)]
CRUNCH-606: Handle setting version correctly and removed stray System.out in test.
Micah Whitacre [Mon, 11 Apr 2016 14:47:33 +0000 (09:47 -0500)]
CRUNCH-606: Kafka Source for Crunch which supports reading data as BytesWritable
* Some of the code contributed by Bryan Baugher and Andrew Olson
Gabriel Reid [Tue, 10 May 2016 09:02:11 +0000 (11:02 +0200)]
CRUNCH-608 Write Bloom filters in HFiles
Use a correctly-configured StoreFile.Writer (instead of HFile.Writer)
for writing HFiles so that Bloom filter data is also included in
the written HFiles.
Gabriel Reid [Mon, 2 May 2016 15:31:20 +0000 (17:31 +0200)]
CRUNCH-607 Allow collection reuse in MemPipeline
Prevent SingleUseIterable from throwing an IllegalArgumentException
when legal reuse of PGroupedCollections are done with the
MemPipeline.
This simply prevents materializing the transformed contents of
a MemCollection until it is iterated over.
Josh Wills [Sun, 24 Apr 2016 02:23:02 +0000 (19:23 -0700)]
[maven-release-plugin] prepare for next development iteration
Josh Wills [Sun, 24 Apr 2016 02:23:01 +0000 (19:23 -0700)]
[maven-release-plugin] prepare branch apache-crunch-0.14
mkwhitacre [Mon, 23 Nov 2015 00:07:30 +0000 (18:07 -0600)] 
CRUNCH-579: Supported access to counters from original TaskContext
Signed-off-by: Micah Whitacre <mkwhit@gmail.com>
Igor Bernstein [Sun, 10 Apr 2016 19:42:10 +0000 (15:42 -0400)] 
CRUNCH-600: pass job credentials when building multiple outputs
Signed-off-by: Micah Whitacre <mkwhit@gmail.com>
David Whiting [Thu, 31 Mar 2016 10:06:45 +0000 (12:06 +0200)]
CRUNCH-599: Fix increment and incrementIf methods in crunch-lambda so they also emit the incoming element
Josh Wills [Thu, 24 Mar 2016 16:55:16 +0000 (09:55 -0700)]
CRUNCH-597: Upgrade to Parquet 1.8.1
tworec [Fri, 4 Mar 2016 17:58:01 +0000 (18:58 +0100)] 
CRUNCH-596 Support right-outer bloom join
Signed-off-by: Gabriel Reid <greid@apache.org>
Joel [Thu, 24 Dec 2015 16:55:03 +0000 (17:55 +0100)] 
Added the ability to specify the amount of reducers when doing a sharded join.
Signed-off-by: Josh Wills <jwills@apache.org>
Josh Wills [Tue, 19 Jan 2016 06:33:23 +0000 (22:33 -0800)]
CRUNCH-586: Make SparkPipeline support reads from HBaseSourceTargets
Stephen Durfey [Fri, 12 Feb 2016 23:14:01 +0000 (17:14 -0600)] 
changes to support only affected regions during hfile write
Signed-off-by: Micah Whitacre <mkwhit@gmail.com>
Micah Whitacre [Fri, 12 Feb 2016 03:54:59 +0000 (21:54 -0600)]
CRUNCH-594: Moved the crunch-lambda to a java8 activated profile.
Micah Whitacre [Fri, 12 Feb 2016 03:38:51 +0000 (21:38 -0600)]
CRUNCH-593: Added exclusions for iml and target directory.
Tom White [Wed, 10 Feb 2016 16:02:31 +0000 (16:02 +0000)]
CRUNCH-592: Job fails for null ByteBuffer value in Avro tables.
Josh Wills [Tue, 2 Feb 2016 19:05:04 +0000 (11:05 -0800)]
CRUNCH-590: Fix CSVInputFormat to work with S3
Josh Wills [Mon, 7 Dec 2015 02:13:00 +0000 (18:13 -0800)]
CRUNCH-582: Upgrade Crunch Guava to 14.0.1
Steffen Grohsschmiedt [Thu, 21 Jan 2016 09:42:07 +0000 (10:42 +0100)] 
Make replication factor for DistCache configurable
Add configuration option to set the replication factor for files
distributed with the DistCache helper class.
Signed-off-by: mkwhitacre <mkwhitacre@gmail.com>
David Whiting [Mon, 18 Jan 2016 14:07:13 +0000 (15:07 +0100)]
CRUNCH-587: Add missing filter(), filterByKey() and filterByValue() functions from Lambda LTable implementation
David Whiting [Sat, 2 Jan 2016 10:28:34 +0000 (11:28 +0100)]
Java 8 lambda support for Apache Crunch.
Remove lambda support from crunch-core, and instead implement a new module called crunch-lambda.
This will allow full use of Java 8 features in implementing support for lambda expressions and
method references, without requiring a dependency on Java 8 for crunch-core. Pthings are wrapped
into analagous Lthings which can be operated on with an API inspired both by the existing Crunch
API and the Java 8 streams API.
David Whiting [Thu, 10 Dec 2015 16:20:35 +0000 (17:20 +0100)]
CRUNCH-584: Add missing mapValues in PGroupedTable for Java 8 lambdas
Tom White [Thu, 10 Dec 2015 13:19:24 +0000 (13:19 +0000)]
CRUNCH-583: Scrunch classloader failure in distcache
Jeff Quinn [Wed, 25 Nov 2015 18:11:04 +0000 (10:11 -0800)] 
CRUNCH-580: Use thread pools in org.apache.crunch.io.impl.FileTargetImpl#handleOutputs for file renaming.
Signed-off-by: Josh Wills <jwills@apache.org>
Josh Wills [Tue, 17 Nov 2015 01:35:48 +0000 (17:35 -0800)]
CRUNCH-578: Add support for mutable collection type serialization to Scrunch
Gabriel Reid [Fri, 6 Nov 2015 13:25:21 +0000 (14:25 +0100)]
CRUNCH-576 Add multi-scan methods to FromHBase
Add factory methods which allow reading a PCollection from HBase
that uses multiple Scans.
Tomáš Čechal [Tue, 10 Nov 2015 11:11:13 +0000 (12:11 +0100)] 
CRUNCH-577 Use getLongBytes() to correctly parse dfs block size.
Signed-off-by: Josh Wills <jwills@apache.org>
Tom White [Tue, 27 Oct 2015 09:22:07 +0000 (09:22 +0000)]
CRUNCH-561: Scrunch case classes fail in the REPL.
Josh Wills [Mon, 19 Oct 2015 19:15:10 +0000 (12:15 -0700)] 
CRUNCH-515: Decrease collision probability on temp dir cleanup. Contributed by Sean Owen.
Micah Whitacre [Mon, 19 Oct 2015 13:57:47 +0000 (08:57 -0500)]
CRUNCH-574: Upgraded the commons-lang dependency to 2.6 courtesy of Sean Owen (srowen)
Tom White [Mon, 19 Oct 2015 09:21:36 +0000 (10:21 +0100)]
CRUNCH-571: Scrunch functions fail serialization check in the REPL
Tom White [Mon, 19 Oct 2015 09:05:27 +0000 (10:05 +0100)]
CRUNCH-562: Support one output file per key for Parquet.
Josh Wills [Tue, 6 Oct 2015 21:30:24 +0000 (14:30 -0700)]
CRUNCH-568: Don't use a null key in the Aggregators.aggregate implementation
‘Mac [Fri, 2 Oct 2015 00:30:24 +0000 (19:30 -0500)] 
CRUNCH-565_CSVInputFormat-Configuration-Defensiveness
Signed-off-by: Micah Whitacre <mkwhit@gmail.com>
Gabriel Reid [Sat, 3 Oct 2015 15:00:15 +0000 (17:00 +0200)]
CRUNCH-567 Fix potential NPE on close
Prevent a potential NPE on calling close on an incompletely
initialized AvroRecordReader or HFileRecordReader.
Contributed by Sean Owen.
Josh Wills [Sun, 27 Sep 2015 18:45:03 +0000 (11:45 -0700)]
CRUNCH-563: Add support for BigDecimal aggregators. Contributed by Vasu Doppalapudi.
Josh Wills [Wed, 9 Sep 2015 05:09:21 +0000 (22:09 -0700)]
CRUNCH-559: Enabling Spark integration tests
Signed-off-by: Micah Whitacre <mkwhit@gmail.com>
Micah Whitacre [Thu, 3 Sep 2015 12:47:37 +0000 (07:47 -0500)]
CRUNCH-558: Set the Pipeline name as accumulator name
Josh Wills [Wed, 1 Jul 2015 22:20:50 +0000 (15:20 -0700)]
CRUNCH-538: Java lambdas for Crunch business logic.
Josh Wills [Thu, 13 Aug 2015 05:15:58 +0000 (22:15 -0700)]
CRUNCH-556: Fix partitioner configuration in Crunch-on-Spark
Gabriel Reid [Wed, 29 Jul 2015 08:26:52 +0000 (10:26 +0200)]
CRUNCH-554 Add MAX_COMPARABLES and MIN_COMPARABLES
Josh Wills [Thu, 30 Jul 2015 19:06:24 +0000 (14:06 -0500)]
[maven-release-plugin] prepare for next development iteration
Josh Wills [Thu, 30 Jul 2015 19:06:24 +0000 (14:06 -0500)]
[maven-release-plugin] prepare branch apache-crunch-0.13
Micah Whitacre [Tue, 28 Jul 2015 01:57:07 +0000 (20:57 -0500)] 
CRUNCH-550: Removed deprecations in crunch-hbase also added
support for TableName.
Josh Wills [Tue, 28 Jul 2015 01:45:23 +0000 (18:45 -0700)]
CRUNCH-553: Fix record drop issue that can occur w/From.formattedFile TableSources
Josh Wills [Mon, 27 Jul 2015 23:00:49 +0000 (16:00 -0700)]
CRUNCH-551: Make the use of Configuration objects consistent in CrunchInputSplit and CrunchRecordReader
Josh Wills [Tue, 28 Jul 2015 00:13:30 +0000 (17:13 -0700)]
CRUNCH-552: Add support/tests for Parquet files w/Crunch on Spark
Josh Wills [Wed, 22 Jul 2015 17:38:56 +0000 (10:38 -0700)]
CRUNCH-547: Properly handle nullability for Avro union types
Josh Wills [Wed, 22 Jul 2015 17:45:19 +0000 (10:45 -0700)]
CRUNCH-548: Have the AvroReflectDeepCopier use the class of the source object when constructing new instances instead of the target class (which might be an interface/abstract class)
projects / crunch.git / log
