6 years ago[maven-release-plugin] prepare release drill-1.6.0 1.6.0 434/head drill-1.6.0
Parth Chandra [Fri, 11 Mar 2016 00:34:37 +0000 (16:34 -0800)] 
[maven-release-plugin] prepare release drill-1.6.0

6 years agoDRILL-4482: Fix Avro nested field selection regression
Jason Altekruse [Tue, 8 Mar 2016 22:16:03 +0000 (14:16 -0800)] 
DRILL-4482: Fix Avro nested field selection regression

Update some of the Avro tests to properly verify their results,
others still need to be fixed. These will be addressed in DRILL-4110.

Closes #419

6 years agoDRILL-4441: Fix varchar data read out of Avro filtering incorrectly due to metadata bug
Jason Altekruse [Fri, 26 Feb 2016 17:55:05 +0000 (09:55 -0800)] 
DRILL-4441: Fix varchar data read out of Avro filtering incorrectly due to metadata bug

The precision of the Varchar datatype was not being set causing inconsistent
truncation of values to the default length of 1. Fixed the same issue with varbinary.

The test framework was previously taking a string as the baseline for a binary value,
which cannot express all possible values. Fixed the test to intstead use a byte array.
Thie required updating the hive tests that were using the old method of specifying
baselines with a String.

Fix cast to varbinary when reading from a data source with schema needed for writing
a test.

Updated patch to remove varchar lengths from table creation.
This issue was fixed more generally by DRILL-4465, which provides a default
type length for varchar and varbinary during the setup of calcite. This update now
just provides tests to verify the fix in this case.

Closes #393

6 years agoDRILL-4493 - Fixed issues in various POMs with MapR profile
Aditya Kishore [Wed, 9 Mar 2016 08:38:33 +0000 (00:38 -0800)] 
DRILL-4493 - Fixed issues in various POMs with MapR profile
This closes #421

6 years agoDRILL-4487: add unit test for DRILL-4449
adeneche [Tue, 8 Mar 2016 15:54:31 +0000 (16:54 +0100)] 
DRILL-4487: add unit test for DRILL-4449

6 years agoDRILL-4474: Ensure that ConvertCountToDirectScan does not push through project when...
Jinfeng Ni [Tue, 8 Mar 2016 22:15:27 +0000 (14:15 -0800)] 
DRILL-4474: Ensure that ConvertCountToDirectScan does not push through project when nullable input of count is not RexInputRef
This closes #416

6 years agoDRILL-4474: Ensure that ConvertCountToDirectScan only pushes through project when...
Jacques Nadeau [Fri, 4 Mar 2016 21:27:26 +0000 (13:27 -0800)] 
DRILL-4474: Ensure that ConvertCountToDirectScan only pushes through project when project is trivial.
This closes #406

6 years agoDRILL-4485 - MapR profile - switch to MapR 5.1.0, and improve compatibility with...
Patrick Wong [Tue, 8 Mar 2016 02:22:08 +0000 (18:22 -0800)] 
DRILL-4485 - MapR profile - switch to MapR 5.1.0, and improve compatibility with maprfs storage format and MapR DB storage plugin.
 This closes #417

6 years agoDRILL-4443: MIN/MAX on VARCHAR throw a NullPointerException
adeneche [Fri, 26 Feb 2016 21:41:36 +0000 (13:41 -0800)] 
DRILL-4443: MIN/MAX on VARCHAR throw a NullPointerException

6 years agoDRILL-4332: Makes vector comparison order stable in test framework
Laurent Goujon [Tue, 23 Feb 2016 22:14:37 +0000 (14:14 -0800)] 
DRILL-4332: Makes vector comparison order stable in test framework

In the test framework, a vector is a map of <String, Object>. When comparing
actual values with baseline, the comparison is made column by column, but
a HashMap key ordering is not guaranteed, and the ordering actually changed
between Java7 and Java8 in Oracle/OpenJDK.

Replacing HashMap with TreeMap which has a guaranteed ordering by design.

Small update by jason during merge, fixed test failure on JDK 7 due to map key ordering,
just replaced two more uses of HashMap with TreeMap.

Closes #389

6 years agoDRILL-4486: Fix expression serialization escaping
Steven Phillips [Sat, 5 Mar 2016 04:29:26 +0000 (20:29 -0800)] 
DRILL-4486: Fix expression serialization escaping

Closes #412

6 years agoAdding Jason's GPG key
Jason Altekruse [Mon, 1 Feb 2016 02:35:42 +0000 (18:35 -0800)] 
Adding Jason's GPG key

6 years agoDRILL-4375: Fix the maven release profile
Jason Altekruse [Tue, 2 Feb 2016 07:17:37 +0000 (23:17 -0800)] 
DRILL-4375: Fix the maven release profile

This generated pom file was being discovered and maven was trying to
run the target directory in jdbc-all as a submodule.

This change reverts to the default output location (the module root)
and adds corresponding .gitignore and RAT exclude entries. NOTE:
this is considered bad practice as generated files should appear in
the target directory and be removed upon a maven clean. This default
location is considered to be a known shortcoming of the shade plugin.

Also updated integration test for the jdbc-all jar with small
path change, as changing the location of the dependency-reduced-pom.xml
actually changed the directory the test was being executed from.

Closes #402

6 years agoDRILL-4483: Fix text plan regression in query profiles 411/head
vkorukanti [Mon, 7 Mar 2016 23:08:58 +0000 (15:08 -0800)] 
DRILL-4483: Fix text plan regression in query profiles

6 years agoDRILL-4313: C++ Client - Thread safe Logging. Improved Drill bit selection.
Parth Chandra [Fri, 12 Feb 2016 23:42:53 +0000 (15:42 -0800)] 
DRILL-4313: C++ Client - Thread safe Logging.  Improved Drill bit selection.
 - Update random drill bit selection. Shuffle the list initially, then round robin. Add Utility methods to get random numbers and to shuffle and add vectors. Whitespace cleanup
 - Add Git properties to build and print to log.
 - Add interface to get error based on query handle.
 - Add support for Pooled connections. Allows switching between pooled and unpooled connections based on environment variables

6 years agoDRILL-4457: Difference in results returned by window function over BIGINT data
adeneche [Mon, 7 Mar 2016 14:08:08 +0000 (15:08 +0100)] 
DRILL-4457: Difference in results returned by window function over BIGINT data

this closes #410

6 years agoDRILL-4281: Support authorized proxy users to impersonate other users 400/head
Sudheesh Katkam [Sat, 5 Mar 2016 23:18:28 +0000 (15:18 -0800)] 
DRILL-4281: Support authorized proxy users to impersonate other users

closes #400

6 years agoDRILL-4449: Wrong results when metadata cache is used with specific set of queries
adeneche [Thu, 25 Feb 2016 03:19:09 +0000 (19:19 -0800)] 
DRILL-4449: Wrong results when metadata cache is used with specific set of queries

close apache/drill#392

6 years agoDRILL-4467: Fix field ordering issue in PrelUtil
Jacques Nadeau [Fri, 4 Mar 2016 11:04:40 +0000 (03:04 -0800)] 
DRILL-4467: Fix field ordering issue in PrelUtil

6 years agoDRILL-4465: Simplify Calcite parsing & planning integration 401/head
Jacques Nadeau [Thu, 25 Feb 2016 18:45:40 +0000 (10:45 -0800)] 
DRILL-4465: Simplify Calcite parsing & planning integration

- Canonicalize Planning phases with PlannerPhase enumeration
- Canonicalize PlannerType transforms
- Remove dependency on Calcite's Frameworks.Planner since Drill need stop heavily customize interactions
- Update AbstractStoragePlugin to implement a phase-aware planning rule injection behavior.
- Avoid (or at least reduce) duplicated registerSchemas() invocations

This closes #401.

6 years agoDRILL-4354: Remove sessions in anonymous (auth disabled) WebUI access
vkorukanti [Tue, 2 Feb 2016 14:48:18 +0000 (06:48 -0800)] 
DRILL-4354: Remove sessions in anonymous (auth disabled) WebUI access

6 years agoDRILL-4410: ListVector should initialize bits in allocateNew
Minji Kim [Thu, 18 Feb 2016 06:29:11 +0000 (22:29 -0800)] 
DRILL-4410: ListVector should initialize bits in allocateNew

Remove large arrays.json files and generate it in /tmp, Check test results
Add another unit test for ListVector in TestValueVector, and use tempDir in TestComplexTypeReader

Closes #380

6 years agoDRILL-4384: Fix Profile regressions
Jacques Nadeau [Sun, 21 Feb 2016 18:29:09 +0000 (10:29 -0800)] 
DRILL-4384: Fix Profile regressions

6 years agoDRILL-4383: Allow custom configurations to be specified for a FileSystem plugin
Jason Altekruse [Fri, 12 Feb 2016 00:52:38 +0000 (16:52 -0800)] 
DRILL-4383: Allow custom configurations to be specified for a FileSystem plugin

add an example s3 plugin, disabled by default

Closes #375

6 years agoDRILL-4327: Fix rawtypes warnings in drill codebase
Laurent Goujon [Thu, 28 Jan 2016 03:01:13 +0000 (19:01 -0800)] 
DRILL-4327: Fix rawtypes warnings in drill codebase

Fixing most rawtypes warning issues in drill modules.

Closes #347

6 years agoDRILL-3488: Allow Java 1.8
adeneche [Tue, 23 Feb 2016 17:02:38 +0000 (09:02 -0800)] 
DRILL-3488: Allow Java 1.8

6 years agoDRILL-3688: Drill should honor "skip.header.line.count" and "skip.footer.line.count...
Arina Ielchiieva [Thu, 11 Feb 2016 17:16:30 +0000 (17:16 +0000)] 
DRILL-3688: Drill should honor "skip.header.line.count" and "skip.footer.line.count" attribute of Hive table

1. Functionality to skip header and footer lines while reading Hive data.
2. Unit tests.

6 years agoDRILL-4346: Handle NumberFormatException when casting empty string to int in hbase...
vataga [Tue, 2 Feb 2016 11:18:49 +0000 (13:18 +0200)] 
DRILL-4346: Handle NumberFormatException when casting empty string to int in hbase/maprdb

- Made replacing of functions casting for nullable input varbinary and var16char types are working (in CastFunctions.java setting 'drill.exec.functions.cast_empty_string_to_null = true' didn't work);
- Added new generated classes for casting varbinary and var16char to int (CastEmptyStringVarTypesToNullableNumeric template and Casts.tdd data);
- Created a test (in TestHBaseQueries.java) for checking an empty string to integer casting in maprdb/hbase table;
- Small design changes according to comments in review (import statements, prefixes ...).

This closes #384

6 years agoDRILL-3944: Drill MAXDIR Unknown variable or type FILE_SEPARATOR. This
Arina Ielchiieva [Tue, 2 Feb 2016 16:48:29 +0000 (18:48 +0200)] 
DRILL-3944: Drill MAXDIR Unknown variable or type FILE_SEPARATOR. This
closes #391

6 years agoDRILL-4434: Deprecate GroupScan.enforceWidth API 390/head
vkorukanti [Wed, 24 Feb 2016 22:13:21 +0000 (14:13 -0800)] 
DRILL-4434: Deprecate GroupScan.enforceWidth API

6 years agoDRILL-4423: TestCsvHeader#testEmptyFinalColumn is causing the unit test framework...
adeneche [Mon, 22 Feb 2016 20:48:11 +0000 (12:48 -0800)] 
DRILL-4423: TestCsvHeader#testEmptyFinalColumn is causing the unit test framework to print 10K rows in Stdout

this closes #387

6 years agoDRILL-4387: GroupScan or ScanBatchCreator should not use star column in case of skipA...
Jinfeng Ni [Fri, 12 Feb 2016 22:18:59 +0000 (14:18 -0800)] 
DRILL-4387: GroupScan or ScanBatchCreator should not use star column in case of skipAll query.

The skipAll query should be handled in RecordReader.

6 years agoDRILL-4260: Adding support for some custom window frames
adeneche [Tue, 19 Jan 2016 21:33:22 +0000 (13:33 -0800)] 
DRILL-4260: Adding support for some custom window frames

this includes the following JIRAs:

this closes #340

6 years agoDRILL-4287: During initial DrillTable creation don't read the metadata cache file...
Aman Sinha [Mon, 18 Jan 2016 18:26:59 +0000 (10:26 -0800)] 
DRILL-4287: During initial DrillTable creation don't read the metadata cache file; instead do it during ParquetGroupScan.

Maintain state in FileSelection to keep track of whether certain operations have been done on that selection.

Remove ParquetFileSelection since its only purpose was to carry the metadata cache information which is not needed anymore.


Resolve issues after rebasing:

1) JsonIgnore fileSelection in ParquetGroupScan
2) FileSysemPartitionDescriptor change.


DRILL-4287: Address code review comments and follow-up changes after rebasing:

- In FileSelection: updated call to the Stopwatch, set all flags appropriately in minusDirectories(), modify supportDirPruning()
- In ParquetGroupScan: Simplify directory checking in constructor,  set the parquetTableMetadata field after reading metadata cache.
- Fix unit tests to use an alias for the reserved dir<N> columns as partition-by columns.

More follow-up changes:

 - Get rid of fileSelection attribute in ParquetGroupScan
 - Initialize entries after expanding the selection when metadata cache is used
 - For non-metadata cache, don't do any expansion in the constructor; let init() handle it
 - In FileSystemPartitionDescriptor, the createPartitionSublists is modified to check for parquet scan

When reading from metadata cache , ensure selection root does not contain the scheme and authority prefix.  Minor refactoring.

Address code review comments and fix a bug. Simplify FileSelection state management based on review comment.

close apache/drill#376

6 years agoUpdate version to 1.6.0-SNAPSHOT
Jason Altekruse [Fri, 19 Feb 2016 21:56:09 +0000 (13:56 -0800)] 
Update version to 1.6.0-SNAPSHOT

6 years agoDRILL-4392: Fix CTAS partition to remove one unnecessary internal field in generated... 383/head
Jinfeng Ni [Thu, 18 Feb 2016 23:38:42 +0000 (15:38 -0800)] 
DRILL-4392: Fix CTAS partition to remove one unnecessary internal field in generated parquet files.

6 years agoDRILL-4275: create TransientStore for short-lived objects; refactor PersistentStore... 374/head
Hanifi Gunes [Mon, 15 Feb 2016 23:16:47 +0000 (15:16 -0800)] 
DRILL-4275: create TransientStore for short-lived objects; refactor PersistentStore to introduce pagination mechanism

6 years agoDRILL-4382: Remove dependency on drill-logical from vector package 373/head
Steven Phillips [Fri, 5 Feb 2016 01:43:17 +0000 (17:43 -0800)] 
DRILL-4382: Remove dependency on drill-logical from vector package

6 years agoDRILL-4363: Row count based pruning for parquet table used in Limit n query. 371/head
Jinfeng Ni [Tue, 2 Feb 2016 23:31:47 +0000 (15:31 -0800)] 
DRILL-4363: Row count based pruning for parquet table used in Limit n query.

Modify two existint unit testcase:
1) TestPartitionFilter.testMainQueryFalseCondition(): rowCount pruning applied after false condition is transformed into LIMIT 0
2) TestLimitWithExchanges.testPushLimitPastUnionExchange(): modify the testcase to use Json source, so that it does not mix with PushLimitIntoScanRule.

6 years agoDRILL-4380: Fix performance regression: in creation of FileSelection in ParquetFormat...
Parth Chandra [Fri, 18 Dec 2015 00:30:42 +0000 (16:30 -0800)] 
DRILL-4380: Fix performance regression: in creation of FileSelection in ParquetFormatPlugin to not set files if metadata cache is available. This closes #369

6 years agoDRILL-4230: Fixing query failures when mongo collection is empty
Kamesh [Fri, 1 Jan 2016 09:09:29 +0000 (14:39 +0530)] 
DRILL-4230: Fixing query failures when mongo collection is empty

6 years agoDRILL-4331: Fix TestFlattenPlanning.testFlattenPlanningAvoidUnnecessaryProject test...
Laurent Goujon [Mon, 1 Feb 2016 19:06:01 +0000 (11:06 -0800)] 
DRILL-4331: Fix TestFlattenPlanning.testFlattenPlanningAvoidUnnecessaryProject test for Java8

Test TestFlattenPlanning.testFlattenPlanningAvoidUnnecessaryProject fails with Java8
because of Map order differences between Java7 and Java8. Test now checks for both

Closes #351

6 years agoDRILL-4295: Removes obsolete protobuf-generated files
Laurent Goujon [Thu, 21 Jan 2016 18:16:29 +0000 (10:16 -0800)] 
DRILL-4295: Removes obsolete protobuf-generated files

These files are obsolete protobuf-generated files, whose definition has been

Cleaning src/main/java files before calling protobuf compiler so that obsolete
files are removed from the directory.

Closes #333

6 years agoDRILL-4353: Add HttpSessionListener to release resources of expired/invalidated sessions
vkorukanti [Thu, 4 Feb 2016 21:06:54 +0000 (13:06 -0800)] 
DRILL-4353: Add HttpSessionListener to release resources of expired/invalidated sessions

Closes #359

6 years agoDRILL-4314: Unit Test Framework can support schema check
Hsuan-Yi Chu [Wed, 27 Jan 2016 00:33:39 +0000 (16:33 -0800)] 
DRILL-4314: Unit Test Framework can support schema check

Closes #339

6 years agoDRILL-4225 TestDateFunctions#testToChar fails when the locale is non-English - Change...
Akihiko Kusanagi [Sat, 26 Dec 2015 17:02:03 +0000 (02:02 +0900)] 
DRILL-4225 TestDateFunctions#testToChar fails when the locale is non-English - Changed to use LocalDate, LocalTime and LocalDateTime to build expectedResults instead of string literals

Closes #311

6 years agoDRILL-4361: Let FileSystemPlugin FormatCreator class be overridable by subclasses
Laurent Goujon [Fri, 5 Feb 2016 22:38:10 +0000 (14:38 -0800)] 
DRILL-4361: Let FileSystemPlugin FormatCreator class be overridable by subclasses

Allow for FileSystemPlugin subclasses to customize FormatPlugin by injecting
their own version of FormatCreator.

This closes #365.

6 years agoDRILL-4359: Adds equals/hashCode methods to EndpointAffinity
Laurent Goujon [Fri, 5 Feb 2016 22:06:00 +0000 (14:06 -0800)] 
DRILL-4359: Adds equals/hashCode methods to EndpointAffinity

Adds equals/hashCode methods to EndpointAffinity to allow for comparison in

This closes #363.

6 years agoDRILL-4358: Fix NPE in UserServer.close()
Jacques Nadeau [Fri, 5 Feb 2016 18:45:32 +0000 (10:45 -0800)] 
DRILL-4358: Fix NPE in UserServer.close()

- Also remove untested CustomSerDe's from CustomTunnel.
- Fix GuavaPatcher copy-paste comment mistake.

closes #362

6 years agoDRILL-4362: Exclude log4j for hbase dependency under mapr profile
Sudheesh Katkam [Fri, 5 Feb 2016 23:36:52 +0000 (15:36 -0800)] 
DRILL-4362: Exclude log4j for hbase dependency under mapr profile

This closes #366

6 years agoDRILL-4297: Enable custom serializers and deserializers when using CustomTunnel
Jacques Nadeau [Fri, 5 Feb 2016 02:09:54 +0000 (18:09 -0800)] 
DRILL-4297: Enable custom serializers and deserializers when using CustomTunnel

- Adds support for customer serializers and deserializers
- Adds pre-built serializers and deserializers for Protobuf, Jackson, Protostuff (protobuf) and Protostuff (json)

6 years agoDRILL-3581: Upgrade to Guava 18.0
Jacques Nadeau [Mon, 14 Sep 2015 20:01:57 +0000 (13:01 -0700)] 
DRILL-3581: Upgrade to Guava 18.0

- Replace Stopwatch constructors with .createStarted() or .createUnstarted()
- Stop using InputSupplier and Closeables.closeQuietly
- Clean up quiet closes to log or (preferably) propagate.
- Add log4j to enforcer exclusions.
- Update HBaseTestSuite to add patching of Closeables.closeQuietly() and Stopwatch legacy methods. Only needed when running HBaseMiniCluster.
- Remove log4j from HBase's pom to provide exception logging.
- Remove log4j from Hive's shaded pom.
- Update Catastrophic failures to use the same pattern to ensure reporting.
- Update test framework to avoid trying IPv6 resolution. (This removes 90s pause from HBase startup in my tests)

This closes #361.
This closes #157.

6 years agoDRILL-3581: Upgrade Jackson to version 2.7.1
Jacques Nadeau [Wed, 3 Feb 2016 03:30:47 +0000 (19:30 -0800)] 
DRILL-3581: Upgrade Jackson to version 2.7.1

6 years agoDRILL-3581: Upgrade HPPC to 0.7.1
Jacques Nadeau [Wed, 3 Feb 2016 03:30:31 +0000 (19:30 -0800)] 
DRILL-3581: Upgrade HPPC to 0.7.1

6 years agoDRILL-4323: Handle skipAll query when use HiveDrillNativeParquetScan
Hsuan-Yi Chu [Fri, 29 Jan 2016 21:20:12 +0000 (13:20 -0800)] 
DRILL-4323: Handle skipAll query when use HiveDrillNativeParquetScan

Do not add Project when no column is needed to be read out from Scan (e.g., select count(*) from hive.table)

6 years agoDRILL-4349: parquet reader returns wrong results when reading a nullable column that...
adeneche [Wed, 3 Feb 2016 23:42:22 +0000 (15:42 -0800)] 
DRILL-4349: parquet reader returns wrong results when reading a nullable column that starts with a large number of nulls (>30k)

6 years agoDRILL-4235: record new state STARTING when state transitions to STARTING from ENQUEUED 358/head
Hanifi Gunes [Thu, 4 Feb 2016 20:07:36 +0000 (12:07 -0800)] 
DRILL-4235: record new state STARTING when state transitions to STARTING from ENQUEUED

6 years agoDRILL-4339: Reverse the function signature change made to AbstractRecordReader.setCo...
Jinfeng Ni [Tue, 2 Feb 2016 21:13:39 +0000 (13:13 -0800)] 
DRILL-4339:  Reverse the function signature change made to AbstractRecordReader.setColumns() in DRILL-4279.

6 years agoDRILL-4128: Fix NPE when calling getString on a JDBC ResultSet when the type is not... 353/head
Jason Altekruse [Mon, 1 Feb 2016 21:03:23 +0000 (13:03 -0800)] 
DRILL-4128: Fix NPE when calling getString on a JDBC ResultSet when the type is not varchar

6 years agoDRILL-4279: Improve performance for skipAll query against Text/JSON/Parquet table.
Jinfeng Ni [Wed, 20 Jan 2016 00:47:08 +0000 (16:47 -0800)] 
DRILL-4279: Improve performance for skipAll query against Text/JSON/Parquet table.

6 years agoDRILL-2517: Move directory-based partition pruning to Calcite logical planning phase.
Jinfeng Ni [Fri, 8 Jan 2016 18:28:53 +0000 (10:28 -0800)] 
DRILL-2517: Move directory-based partition pruning to Calcite logical planning phase.

1) Make directory-based pruning rule both work in calcite logical and drill logical planning phase.

2) Only apply directory-based pruning in logical phase when there is no metadata cache.

3) Make FileSelection constructor public, since FileSelection.create() would modify selectionRoot.

6 years agoDRILL-2517: (Prototype from Mehant) Move directory based partition pruning to logical...
Mehant Baid [Wed, 11 Nov 2015 06:26:26 +0000 (22:26 -0800)] 
DRILL-2517: (Prototype from Mehant) Move directory based partition pruning to logical phase.

6 years agoDRILL-2517: (Prototype from Adam) Apply directory-based pruning before reading file.
AdamPD [Mon, 18 Jan 2016 18:40:54 +0000 (10:40 -0800)] 
DRILL-2517: (Prototype from Adam) Apply directory-based pruning before reading file.

6 years agoDRILL-4328: Fix backward compatibility regression caused by DRILL-4198 348/head
vkorukanti [Thu, 28 Jan 2016 22:54:05 +0000 (14:54 -0800)] 
DRILL-4328: Fix backward compatibility regression caused by DRILL-4198

6 years agoDRILL-4313: CPP client - Improve method to pick random drillbit from a cluster. Updat...
Parth Chandra [Tue, 26 Jan 2016 19:55:59 +0000 (11:55 -0800)] 
DRILL-4313: CPP client - Improve method to pick random drillbit from a cluster. Update build for protobuf changes. This closes #346

6 years agoDRILL-4322: Add underlying exception message when IOException causes DROP TABLE failure
Jason Altekruse [Thu, 28 Jan 2016 19:18:43 +0000 (11:18 -0800)] 
DRILL-4322: Add underlying exception message when IOException causes DROP TABLE failure

This closes #344

6 years agoDRILL-2653: Improve web UI experience when there is an error in a storage plugin...
Jason Altekruse [Thu, 28 Jan 2016 18:01:30 +0000 (10:01 -0800)] 
DRILL-2653: Improve web UI experience when there is an error in a storage plugin configuration

Fixed success message, made the error messages red

This closes #343

6 years agoDRILL-4291: Fix Missing classes when trying to query varchar[] using JDBC
Jacques Nadeau [Sun, 24 Jan 2016 17:59:12 +0000 (09:59 -0800)] 
DRILL-4291: Fix Missing classes when trying to query varchar[] using JDBC

- Create a new simplified version of the Hadoop Text class that doesn't include massive dependencies.
- Update Vectors to use new Text class.
- Update the jdbc-all module to have a test which also includes complex types.
- Clean up exclusions in Jdbc jar file to reduce file size.
- Add an enforcer rule that ensures the adbc-all jar exclusions are maintained in the future.

This closes #336.

6 years agoDRILL-4196 Fix to stop returning no more data when output batch is full during merge.
Amit Hadke [Wed, 27 Jan 2016 00:52:25 +0000 (16:52 -0800)] 
DRILL-4196 Fix to stop returning no more data when output batch is full during merge.

6 years agoDRILL-4270: Create a separate WindowFramer that supports the FRAME clause
adeneche [Mon, 11 Jan 2016 22:47:32 +0000 (14:47 -0800)] 
DRILL-4270: Create a separate WindowFramer that supports the FRAME clause

separate DefaultFrameTemplate into 2 implementations: one that supports custom frames (aggregations, first_value, last_value) and one that doesn't

this closes #322

6 years agoDRILL-4190 Don't hold on to batches from left side of merge join.
Amit Hadke [Mon, 4 Jan 2016 06:08:21 +0000 (22:08 -0800)] 
DRILL-4190 Don't hold on to batches from left side of merge join.

6 years agoDRILL-4182 Set type to LATE when values are nulls.
Amit Hadke [Thu, 17 Dec 2015 22:58:30 +0000 (14:58 -0800)] 
DRILL-4182 Set type to LATE when values are nulls.

6 years agoDRILL-4182 Take care of null comparisions.
Amit Hadke [Thu, 17 Dec 2015 01:19:13 +0000 (17:19 -0800)] 
DRILL-4182 Take care of null comparisions.

6 years agoDRILL-4256: Create HiveConf per HiveStoragePlugin and reuse it wherever needed. 329/head
vkorukanti [Mon, 11 Jan 2016 23:01:02 +0000 (15:01 -0800)] 
DRILL-4256: Create HiveConf per HiveStoragePlugin and reuse it wherever needed.

Creating new instances of HiveConf() are very costly, we should avoid creating new ones as much as possible.
Also get rid of hiveConfigOverride and use HiveConf in HiveStoregPlugin wherever we need the HiveConf.

6 years agoDRILL-4288: Removes obsolete protobuf-generated files
Laurent Goujon [Wed, 20 Jan 2016 01:03:27 +0000 (17:03 -0800)] 
DRILL-4288: Removes obsolete protobuf-generated files

This closes #332.

6 years agoDRILL-4285: Bumps FMPP version to 0.9.15/FreeMarker to 2.3.21
Laurent Goujon [Tue, 19 Jan 2016 01:28:39 +0000 (17:28 -0800)] 
DRILL-4285: Bumps FMPP version to 0.9.15/FreeMarker to 2.3.21

FMPP 0.9.14 pom.xml specifies a version range for FreeMarker dependency,
which should be okay except that latest SNAPSHOT version doesn't resolve
FMPP 0.9.15 now uses a fixed stable version (2.3.21).

This closes #330.

6 years agoDRILL-4246: Fix Allocator concurrency bug and improve error detection
Jacques Nadeau [Mon, 11 Jan 2016 04:08:03 +0000 (20:08 -0800)] 
DRILL-4246: Fix Allocator concurrency bug and improve error detection

- Rename the internal DrillBuf field to udle to better express its purpose.
- Rename AllocatorManager to AllocationManager to better express its purpose.
- Address situation where dangling ledger could be transferred into while it was being released released by protecting association and release inside the AllocationManager.
- Add allocator assertions to ensure allocator operations are done while the allocator is open.
- Simplify AllocationManager locking model.
- Exclude HDFS reference to netty-all
- Improve debugging messages for allocators (and fix debug message bugs)

This closes #323.

6 years agoDRILL-4277: Fix for JdbcPrel serialization issue.
Jacques Nadeau [Fri, 15 Jan 2016 19:45:29 +0000 (11:45 -0800)] 
DRILL-4277: Fix for JdbcPrel serialization issue.

This closes #326.

6 years agoDRILL-4278: Heap memory leak issues
Jacques Nadeau [Tue, 19 Jan 2016 01:42:33 +0000 (17:42 -0800)] 
DRILL-4278: Heap memory leak issues

- Fix issue where WorkspaceConfig was not returning consistent hashCode()s for equal objects.
- Fix issue where we were misusing recycler causing object reference leaks

This closes #331.

6 years agoDRILL-4131: Move RPC allocators under Drill's root allocator & accounting
Jacques Nadeau [Fri, 15 Jan 2016 23:53:13 +0000 (15:53 -0800)] 
DRILL-4131: Move RPC allocators under Drill's root allocator & accounting

- Allow settings to be set to ensure RPC reservation and maximums (currently unset by default). Defaults set in drill-module.conf
- Add new metrics to report RPC layer memory consumption.
- Check for memory leaks from RPC layer at shutdown.
- Add a multi-Drillbit single JVM safe DrillMetrics.register()
- Remove invalid verifyAllocator checks while RPC connection (and PING/PONG) are maintained

This closes #327.

6 years agoDRILL-3845: UnorderedReceiver shouldn't terminate until it receives a final batch
adeneche [Wed, 6 Jan 2016 22:50:38 +0000 (14:50 -0800)] 
DRILL-3845: UnorderedReceiver shouldn't terminate until it receives a final batch

MergingRecordBatch doesn't wait for last batch when it's an early termination

this closes #319

6 years agoDRILL-3201: Support security for access through Web UI
vkorukanti [Sun, 13 Sep 2015 07:44:56 +0000 (00:44 -0700)] 
DRILL-3201: Support security for access through Web UI

6 years agoDRILL-4241: Add Single Tablet Writer
Jacques Nadeau [Fri, 20 Nov 2015 02:20:03 +0000 (18:20 -0800)] 
DRILL-4241: Add Single Tablet Writer

- Also move to a test bootstrap
- Update to the latest kudu and Drill
- Add plugin to Drill distribution
- Checkstyle and directory cleanup

This closes #314.

6 years agoDRILL-4241: Timestamps should divide by 1000 in reader
Todd Lipcon [Fri, 20 Nov 2015 00:19:45 +0000 (16:19 -0800)] 
DRILL-4241: Timestamps should divide by 1000 in reader

6 years agoDRILL-4241: Add wait stats and estimated row count in RecordReader
Steven Phillips [Fri, 20 Nov 2015 00:54:41 +0000 (16:54 -0800)] 
DRILL-4241: Add wait stats and estimated row count in RecordReader

6 years agoDRILL-4241: Rewrite RecordReader to support NULLs and be less Java-like
Todd Lipcon [Fri, 20 Nov 2015 00:15:36 +0000 (16:15 -0800)] 
DRILL-4241: Rewrite RecordReader to support NULLs and be less Java-like

6 years agoDRILL-4241: Create table with multiple tablets in kudu test.
Amit Hadke [Fri, 20 Nov 2015 00:16:17 +0000 (16:16 -0800)] 
DRILL-4241: Create table with multiple tablets in kudu test.

6 years agoDRILL-4241: Add parallelization and assignment
Steven Phillips [Fri, 20 Nov 2015 00:00:40 +0000 (16:00 -0800)] 
DRILL-4241: Add parallelization and assignment

6 years agoDRILL-4241: Improve record reader and type mappings
Todd Lipcon [Thu, 19 Nov 2015 23:00:00 +0000 (15:00 -0800)] 
DRILL-4241: Improve record reader and type mappings

6 years agoDRILL-4241: Add table metadata and DROP table support
Jacques Nadeau [Thu, 19 Nov 2015 22:21:14 +0000 (14:21 -0800)] 
DRILL-4241: Add table metadata and DROP table support

6 years agoDRILL-4241: Add pushdown of column projections
Todd Lipcon [Thu, 19 Nov 2015 22:15:58 +0000 (14:15 -0800)] 
DRILL-4241: Add pushdown of column projections

6 years agoDRILL-4241: Various Type Fixes
Todd Lipcon [Mon, 11 Jan 2016 06:50:30 +0000 (22:50 -0800)] 
DRILL-4241: Various Type Fixes

6 years agoDRILL-4241: Fixing the build, make RAT and checkstyle happy.
Jason Altekruse [Mon, 11 Jan 2016 06:50:05 +0000 (22:50 -0800)] 
DRILL-4241: Fixing the build, make RAT and checkstyle happy.

6 years agoDRILL-4241: initial commit
Jacques Nadeau [Thu, 19 Nov 2015 02:45:56 +0000 (18:45 -0800)] 
DRILL-4241: initial commit

6 years agoDRILL-4257: Fix StoragePluginRegistry clean-up behavior and misc clean up.
Jacques Nadeau [Sun, 10 Jan 2016 01:48:26 +0000 (17:48 -0800)] 
DRILL-4257: Fix StoragePluginRegistry clean-up behavior and misc clean up.

- Create the storage plugin registry using a Drill configuration parameter to be able to replace the registry implementation.
- Change StoragePluginRegistry into an interface and move the implementation to an impl class.
- Write documentation for StoragePluginRegistry.
- Make StoragePluginRegistry and StoragePluginMap AutoCloseable and ensure that Drillbit closes registry.
- Misc Drillbit code reorganization so that fields are at top of class (and static methods at bottom).
- Update DrillConfig to support reflection-based creation of storage plugin objects
- Remove final from DrillConfig so that application developers can extend config object.

This closes #321.

6 years agoDRILL-4238: Add a custom RPC interface on the Control channel for extensible communic...
Jacques Nadeau [Sat, 2 Jan 2016 22:55:21 +0000 (14:55 -0800)] 
DRILL-4238: Add a custom RPC interface on the Control channel for extensible communication between bits.

This closes #313.

6 years agoFix issue where FutureBitComment didn't return buffer as part of future.
Jacques Nadeau [Sun, 3 Jan 2016 00:32:36 +0000 (16:32 -0800)] 
Fix issue where FutureBitComment didn't return buffer as part of future.

6 years agoDRILL-4174: fix for DRILL-4081 mistakenly regresses the fix for DRILL-3786
adeneche [Fri, 8 Jan 2016 19:58:48 +0000 (11:58 -0800)] 
DRILL-4174: fix for DRILL-4081 mistakenly regresses the fix for DRILL-3786

6 years agoDRILL-4250: Fix directory-based partition pruning when directory has both subdirector... 318/head
Jinfeng Ni [Thu, 7 Jan 2016 00:40:50 +0000 (16:40 -0800)] 
DRILL-4250: Fix directory-based partition pruning when directory has both subdirectory and file.

6 years agoDRILL-4236: ExternalSort should use the new allocator functionality to better manage...
adeneche [Wed, 30 Dec 2015 22:41:20 +0000 (14:41 -0800)] 
DRILL-4236: ExternalSort should use the new allocator functionality to better manage it's memory usage

this closes #317