drill.git
4 years agoDRILL-6553: Fix TopN for unnest operator
Volodymyr Vysotskyi [Thu, 28 Jun 2018 17:20:38 +0000 (20:20 +0300)] 
DRILL-6553: Fix TopN for unnest operator

closes #1353

4 years agoDRILL-6557: Use size in bytes during Hive statistics calculation if present
Arina Ielchiieva [Fri, 29 Jun 2018 15:25:59 +0000 (18:25 +0300)] 
DRILL-6557: Use size in bytes during Hive statistics calculation if present

1. Check size in bytes presence in stats before fetching input splits and use it if present.
2. Add log trace suggesting to use ANALYZE command before running queries if statistics is unavailable and Drill had to fetch all input splits.
3. Minor refactoring /  cleanup in HiveMetadataProvider class.

closes #1357

4 years agoDRILL-6546: Allow unnest function with nested columns and complex expressions
Volodymyr Vysotskyi [Thu, 14 Jun 2018 16:32:43 +0000 (19:32 +0300)] 
DRILL-6546: Allow unnest function with nested columns and complex expressions

Fix loss of projected names in right side of correlate when single field is projected

4 years agoDRILL-6576: Unnest reports incoming record counts incorrectly
Parth Chandra [Thu, 7 Jun 2018 20:30:01 +0000 (13:30 -0700)] 
DRILL-6576: Unnest reports incoming record counts incorrectly

This closes #1362

4 years agoDRILL-6494: Drill Plugins Handler
Vitalii Diravka [Tue, 3 Jul 2018 17:23:03 +0000 (20:23 +0300)] 
DRILL-6494: Drill Plugins Handler

- Storage Plugins Handler service is used op the Drill start-up stage and it updates storage plugins configs from
  storage-plugins-override.conf file. If plugins configs are present in the persistence store - they are updated,
  otherwise bootstrap plugins are updated and the result configs are loaded to persistence store. If the enabled
  status is absent in the storage-plugins-override.conf file, the last plugin config enabled status persists.
- 'drill.exec.storage.action_on_plugins_override_file' Boot option is added. This is the action, which should be
  performed on the storage-plugins-override.conf file after successful updating storage plugins configs.
  Possible values are: "none" (default), "rename" and "remove".
- The "NULL" issue with updating Hive plugin config by REST is solved. But clients are still being instantiated for disabled
  plugins - DRILL-6412.
- "org.honton.chas.hocon:jackson-dataformat-hocon" library is added for the proper deserializing HOCON conf file
- additional refactoring: "com.typesafe:config" and "org.apache.commons:commons-lang3" are placed into DependencyManagement
  block with proper versions; correct properties for metrics in "drill-override-example.conf" are specified

closes #1345

4 years agoDRILL-6561: Lateral excluding the columns from output container provided by projectio...
Sorabh Hamirwasia [Fri, 29 Jun 2018 17:27:55 +0000 (10:27 -0700)] 
DRILL-6561: Lateral excluding the columns from output container provided by projection push into rules

This closes #1356

4 years agoDRILL-6530: JVM crash with a query involving multiple json files with one file having...
Sorabh Hamirwasia [Tue, 26 Jun 2018 17:53:53 +0000 (10:53 -0700)] 
DRILL-6530: JVM crash with a query involving multiple json files with one file having a schema change of one column from string to list

This closes #1343

4 years agoDRILL-6535: ClassCastException in Lateral Unnest queries when dealing with schema...
Sorabh Hamirwasia [Fri, 22 Jun 2018 05:57:00 +0000 (22:57 -0700)] 
DRILL-6535: ClassCastException in Lateral Unnest queries when dealing with schema changed json data Note: The issue was happening because for a left incoming all right batches were filtered and hence outputIndex was still 0 when new left incoming came with OK_NEW_SCHEMA. The OK_NEW_SCHEMA change was consumed without updating output container schema.

This closes #1339

4 years agoDRILL-6310: limit batch size for hash aggregate
Padma Penumarthy [Tue, 26 Jun 2018 21:00:07 +0000 (14:00 -0700)] 
DRILL-6310: limit batch size for hash aggregate

closes #1324

4 years agoDRILL-6537: Limit the batch size for buffering operators based on how much memory...
Padma Penumarthy [Sun, 1 Jul 2018 16:43:40 +0000 (09:43 -0700)] 
DRILL-6537: Limit the batch size for buffering operators based on how much memory they get

closes #1342

4 years agoDRILL-6554: Minor code improvements in parquet statistics handling
Vlad Rozov [Wed, 27 Jun 2018 17:45:49 +0000 (10:45 -0700)] 
DRILL-6554: Minor code improvements in parquet statistics handling

closes #1349

4 years agoDRILL-6548: IllegalStateException: Unexpected EMIT outcome received in buildSchema...
Sorabh Hamirwasia [Fri, 29 Jun 2018 05:20:32 +0000 (22:20 -0700)] 
DRILL-6548: IllegalStateException: Unexpected EMIT outcome received in buildSchema phase

closes #1352

4 years agoDRILL-6545: Projection Push down into Lateral Join operator.
HanumathRao [Fri, 22 Jun 2018 01:42:24 +0000 (18:42 -0700)] 
DRILL-6545: Projection Push down into Lateral Join operator.

closes #1347

4 years agoDRILL-4580: Support for exporting storage plugin configurations
agirish [Mon, 10 Apr 2017 00:42:57 +0000 (17:42 -0700)] 
DRILL-4580: Support for exporting storage plugin configurations

closes #1350

4 years agoDRILL-4020: The not-equal operator returns incorrect results when used on the HBase...
Akihiko Kusanagi [Tue, 5 Jun 2018 14:12:13 +0000 (23:12 +0900)] 
DRILL-4020: The not-equal operator returns incorrect results when used on the HBase row key

- Added a condition that checks if the filter to the scan specification doesn't have NOT_EQUAL operator
- Added testFilterPushDownRowKeyNotEqual() to TestHBaseFilterPushDown

closes #309

4 years agoDRILL-6147: Adding Columnar Parquet Batch Sizing functionality
Salim Achouche [Wed, 20 Jun 2018 02:23:41 +0000 (19:23 -0700)] 
DRILL-6147: Adding Columnar Parquet Batch Sizing functionality

closes #1330

4 years agoDRILL-6539: Record count not set for this vector container error
Padma Penumarthy [Tue, 26 Jun 2018 04:27:48 +0000 (21:27 -0700)] 
DRILL-6539: Record count not set for this vector container error

closes #1340

4 years agoDRILL-6512: Remove unnecessary processing overhead from RecordBatchSizer
Padma Penumarthy [Tue, 26 Jun 2018 02:06:51 +0000 (19:06 -0700)] 
DRILL-6512: Remove unnecessary processing overhead from RecordBatchSizer

closes #1341

4 years agoDRILL-6498: Support for EMIT outcome in ExternalSortBatch
Sorabh Hamirwasia [Thu, 28 Jun 2018 17:54:31 +0000 (10:54 -0700)] 
DRILL-6498: Support for EMIT outcome in ExternalSortBatch

* DRILL-6498: Support for EMIT outcome in ExternalSortBatch
* Updated TestTopNEmitOutcome to use RowSetComparison for comparing expected and actual output batches produced

closes #1323

4 years agoDRILL-6526: Refactor FileSystemConfig to disallow direct access from the code to...
Arina Ielchiieva [Fri, 22 Jun 2018 16:36:41 +0000 (19:36 +0300)] 
DRILL-6526: Refactor FileSystemConfig to disallow direct access from the code to its variables

4 years agoDRILL-6503: Performance improvements in lateral
Sorabh Hamirwasia [Wed, 27 Jun 2018 06:19:25 +0000 (23:19 -0700)] 
DRILL-6503: Performance improvements in lateral

closes #1328

4 years agoDRILL-6515: Render a link between the Unnest operator and it's source
Kunal Khatua [Tue, 26 Jun 2018 20:21:13 +0000 (13:21 -0700)] 
DRILL-6515: Render a link between the Unnest operator and it's source

* DRILL-6515: Render a link between the Unnest operator and it's source

1. Unnest operator is expected to specify srcOp=##-## to help identify the source of its input.
2. Implicit Source Map is leveraged to capture implicit data flow pipelines between Unnest and Lateral. (Can be expanded to other operators too).
3. In addition, code refactored for more readability.

* Minor comment updates

* Minor Update

Converted implicitSrc to a local variable

closes #1327

4 years agoDRILL-6479: Support EMIT for the Hash Aggr 1311/head
Ben-Zvi [Mon, 4 Jun 2018 21:07:47 +0000 (14:07 -0700)] 
DRILL-6479: Support EMIT for the Hash Aggr

closes #1311

4 years agoDRILL-6454: Native MapR DB plugin support for Hive MapR-DB json table
Vitalii Diravka [Fri, 1 Jun 2018 03:39:44 +0000 (20:39 -0700)] 
DRILL-6454: Native MapR DB plugin support for Hive MapR-DB json table

closes #1314

4 years agoDRILL-6340 Output Batch Control in Project using the RecordBatchSizer
karthik [Wed, 21 Mar 2018 20:52:03 +0000 (13:52 -0700)] 
DRILL-6340 Output Batch Control in Project using the RecordBatchSizer

Changes required to implement Output Batch Sizing in Project using the RecordBatchSizer.

closes #1302

4 years agoDRILL-6486: BitVector split and transfer does not work correctly for non byte-multipl...
karthik [Thu, 7 Jun 2018 20:23:04 +0000 (13:23 -0700)] 
DRILL-6486: BitVector split and transfer does not work correctly for non byte-multiple transfer lengths

Fix for the bug in BitVector splitAndTransfer. The logic for handling copy of last-n bits was incorrect for none byte-multiple transfer lengths.

closes #1316

4 years agoDRILL-6523: Fix NPE for describe of partial schema
Arina Ielchiieva [Thu, 21 Jun 2018 14:04:13 +0000 (17:04 +0300)] 
DRILL-6523: Fix NPE for describe of partial schema

closes #1332

4 years agoDRILL-6513: Max query memory per node set to Drillbit's maximum direct memory
Salim Achouche [Tue, 19 Jun 2018 20:10:06 +0000 (13:10 -0700)] 
DRILL-6513: Max query memory per node set to Drillbit's maximum direct memory

closes #1329

4 years agoDRILL-6502: Rename CorrelatePrel to LateralJoinPrel.
Hanumath Rao Maduri [Fri, 15 Jun 2018 19:05:09 +0000 (12:05 -0700)] 
DRILL-6502: Rename CorrelatePrel to LateralJoinPrel.

closes #1325

4 years agoDRILL-6491: Prevent merge join for full outer join at planning stage
Bohdan Kazydub [Tue, 12 Jun 2018 15:58:24 +0000 (18:58 +0300)] 
DRILL-6491: Prevent merge join for full outer join at planning stage

closes #1320

4 years agoDRILL-6470: Remove defunct repository
Kunal Khatua [Mon, 18 Jun 2018 17:36:43 +0000 (10:36 -0700)] 
DRILL-6470: Remove defunct repository

When build the Drill source code, Parquet jars from the repository hosted at http://repo.dremio.com/release/  is inaccessible.
These artifacts (parquet libraries) are now available within http://maven.corp.maprtech.com/nexus/content/groups/public/org/apache/parquet/

closes #1326

4 years agoDRILL-5188: Expand sub-queries using rules
Volodymyr Vysotskyi [Fri, 18 May 2018 12:54:16 +0000 (15:54 +0300)] 
DRILL-5188: Expand sub-queries using rules

- Add check for agg with group by literal
- Allow NLJ for limit 1
- Implement single_value aggregate function

closes #1321

4 years agoDRILL-6212: Prevent recursive cast expressions
chunhui-shi [Wed, 6 Jun 2018 19:25:31 +0000 (12:25 -0700)] 
DRILL-6212: Prevent recursive cast expressions

closes #1319

4 years agoDRILL-6455: Add missing JDBC Scan Operator for profiles
Kunal Khatua [Mon, 11 Jun 2018 06:36:10 +0000 (23:36 -0700)] 
DRILL-6455: Add missing JDBC Scan Operator for profiles

The operator is missing in the profile protobuf. This commit introduces that.
1. Added protobuf files (incl generated C++ and Java)
2. Updated JdbcSubScan's getOperatorType API

closes #1297

4 years agoDRILL-6468: CatastrophicFailures should not do a graceful shutdown of drill when...
Timothy Farkas [Tue, 5 Jun 2018 20:50:35 +0000 (13:50 -0700)] 
DRILL-6468: CatastrophicFailures should not do a graceful shutdown of drill when terminating the JVM.

closes #1306

4 years agoDRILL-6487: Limit estimateRowCount should not return negative rowcount
Gautam Parai [Fri, 15 Jun 2018 00:26:32 +0000 (17:26 -0700)] 
DRILL-6487: Limit estimateRowCount should not return negative rowcount

closes #1322

4 years agoDRILL-6353: Upgrade Parquet MR dependencies
Vlad Rozov [Wed, 9 May 2018 20:24:11 +0000 (13:24 -0700)] 
DRILL-6353: Upgrade Parquet MR dependencies

closes #1259

4 years agoDRILL-6474: Don't use TopN when order by and offset are used without a limit specified.
Timothy Farkas [Wed, 6 Jun 2018 19:04:39 +0000 (12:04 -0700)] 
DRILL-6474: Don't use TopN when order by and offset are used without a limit specified.

closes #1313

4 years agoDRILL-6488 - change instances of "template inline" to just "template"
Patrick Wong [Tue, 12 Jun 2018 00:05:19 +0000 (17:05 -0700)] 
DRILL-6488 - change instances of "template inline" to just "template"

closes #1317

4 years agoDRILL-6476: Generate explain plan which shows relation between Lateral and the corres...
Hanumath Rao Maduri [Thu, 7 Jun 2018 03:48:12 +0000 (20:48 -0700)] 
DRILL-6476: Generate explain plan which shows relation between Lateral and the corresponding Unnest.

closes #1308

4 years agoDRILL-6477: Drillbit crashes with OOME (Heap) for a large WebUI query
Kunal Khatua [Thu, 7 Jun 2018 20:32:00 +0000 (13:32 -0700)] 
DRILL-6477: Drillbit crashes with OOME (Heap) for a large WebUI query

For queries submitted through the WebUI and retrieving a large result-set, the Drillbit often hangs or crashes due to the (foreman) Drillbit running out of Heap memory.

This is because the Web client translates the result set into a massive object in the heap-space and tries to send that back to the browser. This results in the VM thread actively trying to perform GC if the memory is not sufficient.

The workaround is to have the active webConnection of the query periodically timeout to allow for checking the consumed heap-space. A level of 0.85 (i.e. 85%) is set as default threshold, crossing which, a query submitted through the REST API is marked and failed.
In addition, a user exception is thrown, indicting the cause of the query failing, along with alternative suggestions for re-executing the query.

closes #1309

4 years agoDRILL-6489: Fix filter push down for Hbase & Mapr-DB binary tables when convert funct...
Arina Ielchiieva [Tue, 12 Jun 2018 14:46:34 +0000 (14:46 +0000)] 
DRILL-6489: Fix filter push down for Hbase & Mapr-DB binary tables when convert function is used in a view

4 years agoDRILL-6478: enhance debug logs for batch sizing
Padma Penumarthy [Thu, 7 Jun 2018 21:04:47 +0000 (14:04 -0700)] 
DRILL-6478: enhance debug logs for batch sizing

closes #1310

4 years agoDRILL-6481: Rename ParquetXXXPredicates to ParquetXXXPredicate
Vlad Rozov [Fri, 8 Jun 2018 01:14:01 +0000 (18:14 -0700)] 
DRILL-6481: Rename ParquetXXXPredicates to ParquetXXXPredicate

closes #1312

4 years agoDRILL-6481: Refactor ParquetXXXPredicate classes
Vlad Rozov [Wed, 6 Jun 2018 22:03:42 +0000 (15:03 -0700)] 
DRILL-6481: Refactor ParquetXXXPredicate classes

closes #1312

4 years agoDRILL-6459: Unable to view profile of a running query
Kunal Khatua [Sat, 2 Jun 2018 00:02:58 +0000 (17:02 -0700)] 
DRILL-6459: Unable to view profile of a running query

Fixes the missing text component of the QueryId that causes lookups to fail in `WorkManager.queries` map.
This got introduced with the fix (#1265) for DRILL-5305

Reverting change to QueryIdHelper and DRILL-5305

Removing the changes done, based on inputs from @vrozov and @sohami . The correct approach would be to have this as part of the profile to avoid serialization of the queryIdText for each RPC making use of the QueryId

UX Changes
Set the query ID string and display in WebUI

closes #1301

4 years agoDRILL-6485: Fix typo in drill-env.sh
Paul Rogers [Mon, 11 Jun 2018 04:25:04 +0000 (21:25 -0700)] 
DRILL-6485: Fix typo in drill-env.sh

closes #1315

4 years agoDRILL-6446: Support for EMIT outcome in TopN
Sorabh Hamirwasia [Wed, 9 May 2018 02:09:38 +0000 (19:09 -0700)] 
DRILL-6446: Support for EMIT outcome in TopN
- Added comments for TopNBatch and PriorityQueueTemplate
- Adding support for SchemaChange across next() call with HyperVector in incoming container. This is achieved by adding a new method in HyperVectorWrapper which just updates the vector[] array
  holding multiple vectors with provided input ValueVector array. And also modifying RemovingRecordBatch GenericSV4Copier to hold reference to VectorWrapper instead of ValueVector[] for each column in incoming batch
- Handling empty batches. Two cases like empty batches in the begining with EMIT outcome and empty batches between consecutive EMIT outcome but after receiving some batches with data and EMIT outcome.
   Note: In first case of empty batch it was only returning EMIT outcome without properly creating the output container and SV4 vector. Because of that there could be a case where let's say first batch with EMIT outcome  is empty then TopN will return an empty batch with SV mode NONE and if later batch comes with some records and EMIT outcome, that will generate output batch with OK_NEW_SCHEMA (since TopN always
   generate first output batch with records with OK_NEW_SCHEMA as it returns output with SV4 mode). Also let's consider both batch with EMIT outcome were produced after processing first 2 rows of an input batch.
This is a problem as this is simulating schema change across rows of same incoming batch which will never be the case.

Note: In second case of empty batches priority queue will not be null and will be uninitialized. Also optimize to send EMIT outcome with output batch which has all the data to return for current iteration
                rather than sending it with OK followed by empty batch with EMIT outcome.

closes #1293

4 years agoDRILL-6463 : Fix integer overflow in MockGroupScanPOP
Gautam Parai [Sat, 2 Jun 2018 19:06:21 +0000 (12:06 -0700)] 
DRILL-6463 : Fix integer overflow in MockGroupScanPOP

closes #1303

4 years agoDRILL-6375 : Support for ANY_VALUE aggregate function
Gautam Parai [Wed, 4 Apr 2018 02:18:31 +0000 (19:18 -0700)] 
DRILL-6375 : Support for ANY_VALUE aggregate function

closes #1256

4 years agoDRILL-6466: Add HttpOnly flag to response cookies
Arina Ielchiieva [Tue, 5 Jun 2018 14:35:24 +0000 (17:35 +0300)] 
DRILL-6466: Add HttpOnly flag to response cookies

4 years agoDRILL-6467: Percentage usage of memory is reported as zero by the WebUI
Kunal Khatua [Thu, 7 Jun 2018 04:30:07 +0000 (21:30 -0700)] 
DRILL-6467: Percentage usage of memory is reported as zero by the WebUI

The memory reported as a percentage is incorrectly reported as 0%, irrespective of the actual usage.
closes #1305

4 years agoDRILL-6438: Remove excess logging form the tests.
Timothy Farkas [Tue, 22 May 2018 23:04:02 +0000 (16:04 -0700)] 
DRILL-6438: Remove excess logging form the tests.
 - Removed usages of System.out and System.err from the test and replaced with loggers

closes #1284

4 years agoDRILL-6389: Fixed building javadocs
Timothy Farkas [Tue, 8 May 2018 18:11:33 +0000 (11:11 -0700)] 
DRILL-6389: Fixed building javadocs
 - Added documentation about how to build javadocs
 - Fixed some of the javadoc warnings

closes #1276

4 years agoDRILL-5700: Adding nohup support for Sqlline script
kr-arjun [Fri, 1 Jun 2018 07:09:03 +0000 (00:09 -0700)] 
DRILL-5700: Adding nohup support for Sqlline script

closes #1300

4 years agoDRILL-6432: Show Button to print visualized query plan (#1278)
Kunal Khatua [Wed, 6 Jun 2018 15:43:27 +0000 (08:43 -0700)] 
DRILL-6432: Show Button to print visualized query plan (#1278)

* DRILL-6432: Show Button to print visualized query plan

Having the capability to only print the visualized query plan means the user can specifyi the zoom level (in the printer settings) when printing large complex plans that might span multiple pages.

* Update based on review comment

Padded button by 15px to avoid sticking to pane

closes #1278

4 years agoRevert "DRILL-4020: The not-equal operator returns incorrect results when used on...
Parth Chandra [Sun, 3 Jun 2018 00:41:45 +0000 (17:41 -0700)] 
Revert "DRILL-4020: The not-equal operator returns incorrect results when used on the HBase row key"

This reverts commit 0d5eda83fe34928ff60629e6a4903d43a1d82582.

4 years agoDRILL-5584: Add branding and versioning information for windows C++ Client.
Rob Wu [Thu, 16 Nov 2017 02:21:48 +0000 (18:21 -0800)] 
DRILL-5584: Add branding and versioning information for windows C++ Client.

This closes #5584

4 years agoDRILL-5924: native-client: Support user-specified CXX_FLAGS
Korn, Uwe [Sat, 28 Oct 2017 19:45:51 +0000 (21:45 +0200)] 
DRILL-5924: native-client: Support user-specified CXX_FLAGS

This closes #1022

4 years agoDRILL-4020: The not-equal operator returns incorrect results when used on the HBase...
Akihiko Kusanagi [Wed, 4 Nov 2015 12:33:06 +0000 (21:33 +0900)] 
DRILL-4020: The not-equal operator returns incorrect results when used on the HBase row key

- Added a condition that checks if the filter to the scan specification doesn't have NOT_EQUAL operator
- Added testFilterPushDownRowKeyNotEqual() to TestHBaseFilterPushDown

This closes #309

4 years agoDRILL-6456: Planner shouldn't create any exchanges on the right side of Lateral Join.
Hanumath Rao Maduri [Thu, 31 May 2018 00:59:51 +0000 (17:59 -0700)] 
DRILL-6456: Planner shouldn't create any exchanges on the right side of Lateral Join.

This closes #1299

4 years agoDRILL-6450: Visualized plans for profiles querying JDBC sources is broken
Kunal Khatua [Tue, 29 May 2018 18:20:56 +0000 (11:20 -0700)] 
DRILL-6450: Visualized plans for profiles querying JDBC sources is broken

When viewing a profile for a query against a JDBC source, the visualized plan is not rendered. This is because the generated SQL pushed down to the JDBC source has a line break injected just before the FROM clause.

The workaround is to strip away any injected newlines ('\\n') at least for the SQL defined in the text plan, so that the backend Javascript can render it correctly.
In addition, any single line comments are also removed, but any block comments (i.e. /* .. */ ) are retained as they might carry hints.

This closes #1295

4 years agoDRILL-6445: Fix existing test cases in TestScripts.java and add new test case for...
Sorabh Hamirwasia [Thu, 24 May 2018 18:58:01 +0000 (11:58 -0700)] 
DRILL-6445: Fix existing test cases in TestScripts.java and add new test case for DRILLBIT_CONTEXT variable

This closes #1289

4 years agoDRILL-6236:Batch sizing for hash join
Padma Penumarthy [Wed, 30 May 2018 21:00:16 +0000 (14:00 -0700)] 
DRILL-6236:Batch sizing for hash join

This closes #1227

4 years agoDRILL-6343: bit vector copyFromSafe is not doing realloc
Padma Penumarthy [Thu, 19 Apr 2018 21:36:38 +0000 (14:36 -0700)] 
DRILL-6343: bit vector copyFromSafe is not doing realloc

4 years agoDRILL-4364: Image Metadata Format Plugin - Initial commit of Image Metadata Format...
Akihiko Kusanagi [Sat, 6 Feb 2016 18:55:57 +0000 (03:55 +0900)] 
DRILL-4364: Image Metadata Format Plugin - Initial commit of Image Metadata Format Plugin - See https://issues.apache.org/jira/browse/DRILL-4364

This closes #367

4 years agoDRILL-4091: Adding support for additional gis operations in gis contrib module
potocki [Mon, 16 Nov 2015 13:05:18 +0000 (14:05 +0100)] 
DRILL-4091: Adding support for additional gis operations in gis contrib module

This closes #1201

4 years agoDRILL-6356: batch sizing for union all
Padma Penumarthy [Thu, 10 May 2018 22:28:40 +0000 (15:28 -0700)] 
DRILL-6356: batch sizing for union all

closes #1255

4 years agoDRILL-6402: Repeated Value Vectors copyFrom methods are not updating
Padma Penumarthy [Thu, 10 May 2018 22:29:27 +0000 (15:29 -0700)] 
DRILL-6402: Repeated Value Vectors copyFrom methods are not updating
the value count and writer index correctly for values vector

4 years agoDRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out (#1281)
Venkata Jyothsna Donapati [Wed, 30 May 2018 18:03:58 +0000 (11:03 -0700)] 
DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out (#1281)

DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out

4 years agoDRILL-3964 : Fix NPE in WriterRecordBatch when 0 rows
Gautam Parai [Fri, 25 May 2018 02:00:21 +0000 (19:00 -0700)] 
DRILL-3964 : Fix NPE in WriterRecordBatch when 0 rows

closes #1290

4 years agoDRILL-6442: Adjust Hbase disk cost & row count estimation when filter push down is...
Arina Ielchiieva [Wed, 23 May 2018 16:17:11 +0000 (16:17 +0000)] 
DRILL-6442: Adjust Hbase disk cost & row count estimation when filter push down is applied

closes #1288

4 years agoDRILL-5990: Drill should not startup with JRE sourced java executable
Kunal Khatua [Mon, 21 May 2018 21:19:53 +0000 (14:19 -0700)] 
DRILL-5990: Drill should not startup with JRE sourced java executable

Currently, the check for the Java compiler is made only during query execution.
This commit ensures that if the java executable used for instantiating the Drillbit is from a JRE (i.e. the Java compiler is not available), the Drillbit will fail with the exception reported in the logs:
```
Exception in thread "main" org.apache.drill.exec.exception.DrillbitStartupException: JDK Java compiler not available. Ensure Drill is running with the java executable from a JDK and not a JRE
        at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:138)
        at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:409)
        at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:389)
        at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:385)
```

4 years agoDRILL-6443: Enable Search for both running AND completed queries
Kunal Khatua [Wed, 23 May 2018 22:55:24 +0000 (15:55 -0700)] 
DRILL-6443: Enable Search for both running AND completed queries

When running a query in Drill, the `/profiles` page will show the search (and pagination) capabilities only for the top most visible table (i.e. Running Queries ).

The Completed Queries table will show the search feature only when there are no running queries. This is because the backend uses a generalized freemarker macro to define the seach capabilities for the tables being rendered. With running queries, both, running and completed queries tables have the same element ID, resulting in the search capability only being applied to the first table.

This modifies the Freemarker macro to take an additional argument for distinguishing between Running and Completed Queries table.

4 years agoDRILL-6437: Removed excess maven logging when downloading dependencies. This fixed...
Timothy Farkas [Wed, 23 May 2018 21:14:15 +0000 (14:14 -0700)] 
DRILL-6437: Removed excess maven logging when downloading dependencies. This fixed Travis failures due to log overflow.

4 years agoDRILL-6435: MappingSet is stateful, so it can't be shared between threads
Vlad Rozov [Wed, 23 May 2018 16:14:13 +0000 (09:14 -0700)] 
DRILL-6435: MappingSet is stateful, so it can't be shared between threads

4 years agoDRILL-6436: Storage Plugin to have name and context moved to AbstractStoragePlugin
chunhui-shi [Thu, 15 Feb 2018 20:21:33 +0000 (12:21 -0800)] 
DRILL-6436: Storage Plugin to have name and context moved to AbstractStoragePlugin

closes #1282

4 years agoDRILL-6440: Unnest unit tests and fixes for stats
Parth Chandra [Mon, 16 Apr 2018 23:42:20 +0000 (16:42 -0700)] 
DRILL-6440: Unnest unit tests and fixes for stats

  - Add unit test with mock input, nested lateral and unnest and project.
  - Fix unit test involving batch limits, ignore map tests
  - Fix input row count stats.

closes #1283

4 years agoDRILL-6374: Transitive Closure leads to TPCH Queries regressions and OOM when run...
Vitalii Diravka [Mon, 14 May 2018 12:27:16 +0000 (15:27 +0300)] 
DRILL-6374: Transitive Closure leads to TPCH Queries regressions and OOM when run concurrency test

- Test case for Directory Pruning with Transitive Predicates
- Improving description for STRICT_EQUAL_IS_DISTINCT_FROM predicate

closes #1262

4 years agoDRILL-6431: Unnest operator requires table and a single column alias to be specified.
hmaduri [Tue, 15 May 2018 20:51:45 +0000 (13:51 -0700)] 
DRILL-6431: Unnest operator requires table and a single column alias to be specified.
Fixing the issues related to star column renaming, same field name renaming
and also enforcing that an alias column is required for the unnest operator.

4 years agoDRILL-6419: E2E Integration test for Lateral&Unnest
Sorabh Hamirwasia [Tue, 15 May 2018 21:30:05 +0000 (14:30 -0700)] 
DRILL-6419: E2E Integration test for Lateral&Unnest

4 years agoDRILL-5846: Improve parquet performance for Flat Data Types
Salim Achouche [Wed, 18 Apr 2018 03:12:03 +0000 (20:12 -0700)] 
DRILL-5846: Improve parquet performance for Flat Data Types

closes #1060

4 years agoDRILL-6423: Export query result as a CSV file
Kunal Khatua [Wed, 16 May 2018 17:35:51 +0000 (10:35 -0700)] 
DRILL-6423: Export query result as a CSV file

Added option for user specified delimiter
Also, we show query Id on results page as alink for popping out a new window

closes #1266

4 years agoDRILL-5305: Query Profile must display Query ID
Kunal Khatua [Wed, 16 May 2018 17:22:41 +0000 (10:22 -0700)] 
DRILL-5305: Query Profile must display Query ID

Introduced change to the Protobuf to inject the text-equivalent of the QueryID into the profile. This way, the profile's file name can be changed, but restored back based on this new field.
The Profile UI also shows the Query ID, though this is not inferred from this new field, for sake of backward compatibility with older profiles.

closes #1265

4 years agoDRILL-6418: Handle Schema change in Unnest And Lateral for unnest field / non-unnest...
Sorabh Hamirwasia [Tue, 15 May 2018 21:27:31 +0000 (14:27 -0700)] 
DRILL-6418: Handle Schema change in Unnest And Lateral for unnest field / non-unnest field

Note: Changed Lateral to handle non-empty right batch with OK_NEW_SCHEMA

closes #1271

4 years agoDRILL-6348: Received batches are now owned by the receive operators instead of the...
Salim Achouche [Mon, 23 Apr 2018 01:02:35 +0000 (18:02 -0700)] 
DRILL-6348: Received batches are now owned by the receive operators instead of the parent

closes #1237

4 years agoDRILL-4525: Code cleanup
Bohdan Kazydub [Thu, 17 May 2018 10:08:31 +0000 (13:08 +0300)] 
DRILL-4525: Code cleanup

- Fixed failing test

closes #1268

4 years agoDRILL-4525: Allow SqlBetweenOperator to accept LOWER_OPERAND and UPPER_OPERAND with...
Hsuan-Yi Chu [Mon, 21 Mar 2016 21:43:54 +0000 (14:43 -0700)] 
DRILL-4525: Allow SqlBetweenOperator to accept LOWER_OPERAND and UPPER_OPERAND with different types

4 years agoDRILL-6424: Updating FasterXML Jackson libraries
Vitalii Diravka [Thu, 17 May 2018 09:59:08 +0000 (12:59 +0300)] 
DRILL-6424: Updating FasterXML Jackson libraries

closes #1274

4 years agoDRILL-6413: Update ParquetFilterBuilder.visitBooleanOperator to handle simplified...
Arina Ielchiieva [Thu, 17 May 2018 15:24:35 +0000 (18:24 +0300)] 
DRILL-6413: Update ParquetFilterBuilder.visitBooleanOperator to handle simplified boolean expression

closes #1269

4 years agoDRILL-6426: Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit...
Arina Ielchiieva [Fri, 18 May 2018 10:11:05 +0000 (13:11 +0300)] 
DRILL-6426: Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option

closes #1273

4 years agoDRILL-6421: Refactor DecimalUtility and CoreDecimalUtility classes
Volodymyr Vysotskyi [Wed, 16 May 2018 18:34:56 +0000 (21:34 +0300)] 
DRILL-6421: Refactor DecimalUtility and CoreDecimalUtility classes

closes #1267

4 years agoDRILL-6255: Drillbit while sending control message to itself creates a connection...
Sorabh Hamirwasia [Tue, 8 May 2018 20:06:20 +0000 (13:06 -0700)] 
DRILL-6255: Drillbit while sending control message to itself creates a connection instead of submitting locally

closes #1253

4 years agoDRILL-6420: Add Lateral and Unnest Keyword for highlighting on WebUI
Sorabh Hamirwasia [Fri, 4 May 2018 17:47:28 +0000 (10:47 -0700)] 
DRILL-6420: Add Lateral and Unnest Keyword for highlighting on WebUI

closes #1261

4 years agoDRILL-6027:
Ben-Zvi [Sat, 28 Apr 2018 04:59:25 +0000 (21:59 -0700)] 
DRILL-6027:
  - Added fallback option for HashJoin.
  - No copy of incoming for single partition, and avoid HT resize.
  - Fix memory leak when cancelling while spill file is read
  - get correct schema when probe side is empty
  - Re-create the HashJoinProbe

4 years agoDRILL-6027:
Timothy Farkas [Tue, 13 Feb 2018 00:49:29 +0000 (16:49 -0800)] 
DRILL-6027:
 - Added memory claculator
 - Added unit tests and docs.
 - Fixed IOB caused by output vector allocation.
 - Don't double count records that were spilled in HashJoin

4 years agoDRILL-6027: Initial implementation of HashJoin spill, without memory limits checks yet
Ben-Zvi [Wed, 14 Feb 2018 20:36:39 +0000 (12:36 -0800)] 
DRILL-6027: Initial implementation of HashJoin spill, without memory limits checks yet

4 years agocode review comments
Padma Penumarthy [Wed, 16 May 2018 00:58:13 +0000 (17:58 -0700)] 
code review comments

4 years agoDRILL-6411: Make batch memory sizing logs uniform across all operators
Padma Penumarthy [Mon, 14 May 2018 22:01:25 +0000 (15:01 -0700)] 
DRILL-6411: Make batch memory sizing logs uniform across all operators

4 years agoDRILL-6321: Lateral Join and Unnest - rules, options, logical plan supports
chunhui-shi [Mon, 30 Apr 2018 17:24:06 +0000 (10:24 -0700)] 
DRILL-6321: Lateral Join and Unnest - rules, options, logical plan supports

Included changes:
* Add planner.enable_unnest_lateral option. Default value set to false.
* Enable FilterCorrectRule
* Add support to logical plan
* Fix rebase errors for DRILL-6321 commits