drill.git
4 years agoDRILL-6626: Fixed an IndexOutOfBoundException during aggregator rehash
Salim Achouche [Mon, 23 Jul 2018 20:17:08 +0000 (13:17 -0700)] 
DRILL-6626: Fixed an IndexOutOfBoundException during aggregator rehash

4 years agoDRILL-6622: Fixed a NullPointerException in a query with Union
Salim Achouche [Sun, 22 Jul 2018 05:55:25 +0000 (22:55 -0700)] 
DRILL-6622: Fixed a NullPointerException in a query with Union

closes #1391

4 years agoDRILL-6627: Adding REGEX_SUB_SCAN operator to protobuf file
Vitalii Diravka [Mon, 23 Jul 2018 11:44:48 +0000 (14:44 +0300)] 
DRILL-6627: Adding REGEX_SUB_SCAN operator to protobuf file

- The operator is added to Java based UserBitShared.proto and C++ based UserBitShared.pb.h
- Java and C++ protobuf files are regenerated

4 years agoDRILL-6574: Code cleanup
Bohdan Kazydub [Fri, 13 Jul 2018 14:13:45 +0000 (17:13 +0300)] 
DRILL-6574: Code cleanup

- Fixed failing test

4 years agoDRILL-6574: Add option to push LIMIT(0) on top of SCAN (late limit 0 optimization)
Sudheesh Katkam [Mon, 21 Mar 2016 22:32:47 +0000 (15:32 -0700)] 
DRILL-6574: Add option to push LIMIT(0) on top of SCAN (late limit 0 optimization)

4 years agoDRILL-6606: Fixed bug in HashJoin that caused it not to return OK_NEW_SCHEMA in some...
Timothy Farkas [Sat, 21 Jul 2018 00:23:26 +0000 (17:23 -0700)] 
DRILL-6606: Fixed bug in HashJoin that caused it not to return OK_NEW_SCHEMA in some cases.

closes #1384

4 years agoDRILL-6624: Fix loss of the table row type when the same schema name was specified...
Volodymyr Vysotskyi [Sat, 21 Jul 2018 00:22:28 +0000 (03:22 +0300)] 
DRILL-6624: Fix loss of the table row type when the same schema name was specified as single path and as a complex path in the same query

closes #1390

4 years agoAdd public GPG key for user boaz to KEYS
Ben-Zvi [Fri, 20 Jul 2018 00:55:29 +0000 (17:55 -0700)] 
Add public GPG key for user boaz to KEYS

4 years agoDRILL-6614: Allow usage of MapRDBFormatPlugin for HiveStoragePlugin
Vitalii Diravka [Thu, 19 Jul 2018 13:00:40 +0000 (16:00 +0300)] 
DRILL-6614: Allow usage of MapRDBFormatPlugin for HiveStoragePlugin

4 years agoDRILL-6603: Set num_nulls for parquet statistics to -1 when actual number is not...
Arina Ielchiieva [Tue, 17 Jul 2018 13:41:48 +0000 (16:41 +0300)] 
DRILL-6603: Set num_nulls for parquet statistics to -1 when actual number is not defined.

4 years agoDRILL-6104: Add Log/Regex Format Plugin
Charles S. Givre [Wed, 18 Jul 2018 14:44:42 +0000 (10:44 -0400)] 
DRILL-6104: Add Log/Regex Format Plugin

closes #1114

4 years agoDRILL-6475: Unnest: Null fieldId Pointer.
HanumathRao [Fri, 29 Jun 2018 15:46:41 +0000 (08:46 -0700)] 
DRILL-6475: Unnest: Null fieldId Pointer.

closes #1381

4 years agoDRILL-6496: Added print methods for debugging tests, and fixed missing log statement...
Timothy Farkas [Mon, 25 Jun 2018 20:50:37 +0000 (13:50 -0700)] 
DRILL-6496: Added print methods for debugging tests, and fixed missing log statement in VectorUtils.

closes #1336

4 years agoDRILL-6612: Query fails with AssertionError when joining persistent and temporary...
Volodymyr Vysotskyi [Tue, 17 Jul 2018 15:11:58 +0000 (18:11 +0300)] 
DRILL-6612: Query fails with AssertionError when joining persistent and temporary tables

4 years agoDRILL-6588: Make Sys tables of nullable datatypes
Kunal Khatua [Sat, 14 Jul 2018 17:43:17 +0000 (10:43 -0700)] 
DRILL-6588: Make Sys tables of nullable datatypes

This is to address the generic problem of NULL values being projected as a string because of all datatypes being non-nullable.
This patch only applies to tables backed by the PojoReader (in our case, System tables). Added NonNullable annotations wherever application in any of the System tables, along with a unit test that verifies both nullable and non-nullable datatypes exist in the system tables.

closes #1371

4 years agoDRILL-6591: Show Exception for failed queries submitted in WebUI
Kunal Khatua [Fri, 13 Jul 2018 22:21:14 +0000 (15:21 -0700)] 
DRILL-6591: Show Exception for failed queries submitted in WebUI

* DRILL-6591: Show Exception for failed queries submitted in WebUI

When query fails on Web UI result page no error is shown, only "No result found."
This was because DRILL-6477 (PR #1309) switched to `WebUserConnection.await(long timeoutInMillis)` . Unlike the original `WebUserConnection.await()`, this method did not throw any UserException generated by a query failure. The fix was to use WebUserConnection.getError() method to know about failure of the query and throw UserRemoteException with that.

closes #1379

4 years agoDRILL-5495: convert_from function on top of int96 data results in ArrayIndexOutOfBoun...
Vitalii Diravka [Fri, 13 Jul 2018 14:29:59 +0000 (17:29 +0300)] 
DRILL-5495: convert_from function on top of int96 data results in ArrayIndexOutOfBoundsException

4 years agoDRILL-6472: Prevent using zero precision in CAST function
Volodymyr Vysotskyi [Thu, 12 Jul 2018 16:00:48 +0000 (19:00 +0300)] 
DRILL-6472: Prevent using zero precision in CAST function

- Add check for the correctness of scale value;
- Add check for fitting the value to the value with the concrete scale and precision;
- Implement negative UDF for VarDecimal
- Add unit tests for new checks and UDF.

4 years agoDRILL-6601 LageFileCompilation testProject times out
karthik [Thu, 12 Jul 2018 18:29:01 +0000 (11:29 -0700)] 
DRILL-6601 LageFileCompilation testProject times out

closes #1378

4 years agoDRILL-6542 : IndexOutOfBoundsException for multilevel lateral queries with schema...
Sorabh Hamirwasia [Tue, 10 Jul 2018 00:58:08 +0000 (17:58 -0700)] 
DRILL-6542 : IndexOutOfBoundsException for multilevel lateral queries with schema changed partitioned complex data

closes #1374

4 years ago[DRILL-6587] Added support for custom SSL CTX Options
superbstreak [Mon, 9 Jul 2018 19:07:22 +0000 (12:07 -0700)] 
[DRILL-6587] Added support for custom SSL CTX Options

closes #1366

4 years ago[DRILL-6586] Add SSL Hostname verification with zookeeper connection mode support
superbstreak [Mon, 9 Jul 2018 08:33:55 +0000 (01:33 -0700)] 
[DRILL-6586]  Add SSL Hostname verification with zookeeper connection mode support

4 years ago[DRILL-6581] C++ Client SSL Implementation Fixes/Improvements
superbstreak [Fri, 6 Jul 2018 01:11:47 +0000 (18:11 -0700)] 
[DRILL-6581] C++ Client SSL Implementation Fixes/Improvements

4 years agoDRILL-6579: Added sanity checks to the Parquet reader to avoid infinite loops
Salim Achouche [Tue, 3 Jul 2018 05:04:20 +0000 (22:04 -0700)] 
DRILL-6579: Added sanity checks to the Parquet reader to avoid infinite loops

closes #1361

4 years agoDRILL-6592: Unnest record batch size is called too frequently
Parth Chandra [Fri, 6 Jul 2018 23:23:51 +0000 (16:23 -0700)] 
DRILL-6592: Unnest record batch size is called too frequently

closes #1376

4 years agoDRILL-6596: Fix fillEmpties and set methods for Nullable variable length
Parth Chandra [Fri, 6 Jul 2018 23:23:51 +0000 (16:23 -0700)] 
DRILL-6596: Fix fillEmpties and set methods for Nullable variable length
vectors to not use emptyByteArray

closes #1377

4 years agoDRILL-6346: Create an Official Drill Docker Container
Abhishek Girish [Mon, 25 Jun 2018 20:01:04 +0000 (13:01 -0700)] 
DRILL-6346: Create an Official Drill Docker Container

closes #1348

4 years agoDRILL-6559: Travis timing out
Vitalii Diravka [Sun, 1 Jul 2018 21:47:13 +0000 (00:47 +0300)] 
DRILL-6559: Travis timing out

* Excluding contrib module tests and all exec TPCH unit tests for travis full build.
* Introducing new TRAVIS profile
* Travis build is faster for 4-5 mins

closes #1364

4 years agoDRILL-6560: Enhanced the batch statistics logging enablement
Salim Achouche [Sat, 30 Jun 2018 00:31:40 +0000 (17:31 -0700)] 
DRILL-6560: Enhanced the batch statistics logging enablement

closes #1355

4 years agoDRILL-6578: Handle query cancellation in Parquet reader
Salim Achouche [Tue, 3 Jul 2018 02:13:26 +0000 (19:13 -0700)] 
DRILL-6578: Handle query cancellation in Parquet reader

closes #1360

4 years agoDRILL-6594: Data batches for Project operator are not being split properly and exceed...
karthik [Tue, 5 Jun 2018 00:00:31 +0000 (17:00 -0700)] 
DRILL-6594: Data batches for Project operator are not being split properly and exceed the maximum specified

This change fixes the incorrect accounting in the case where a columns is being projected more than once

closes #1375

4 years agoDRILL-6516: EMIT support in streaming agg
Parth Chandra [Tue, 19 Jun 2018 04:34:20 +0000 (21:34 -0700)] 
DRILL-6516: EMIT support in streaming agg

This closes #1358

4 years agoDRILL-6516: Fix memory leak issue with Sort and StreamingAgg together
Sorabh Hamirwasia [Sun, 1 Jul 2018 06:02:55 +0000 (00:02 -0600)] 
DRILL-6516: Fix memory leak issue with Sort and StreamingAgg together

4 years agoDRILL-6583: Add space between pagination links in Profiles (WebUI) list
Kunal Khatua [Wed, 11 Jul 2018 17:36:55 +0000 (10:36 -0700)] 
DRILL-6583: Add space between pagination links in Profiles (WebUI) list

Inject a small space to improve readability of the pagination links (CSS)

closes #1369

4 years agoDRILL-5797: Use Parquet new reader on all non-complex columns queries
Oleksandr Kalinin [Tue, 10 Jul 2018 07:41:48 +0000 (09:41 +0200)] 
DRILL-5797: Use Parquet new reader on all non-complex columns queries

4 years agoDRILL-6529 Project Batch Sizing causes two LargeFileCompilation tests to timeout
karthik [Sat, 23 Jun 2018 01:26:37 +0000 (18:26 -0700)] 
DRILL-6529 Project Batch Sizing causes two LargeFileCompilation tests to timeout

closes #1335

Changed Project test columns to 10000 and reduces columns projected to SORT tests to 2500

4 years agoDRILL-6549: batch sizing for nested loop join
Padma Penumarthy [Thu, 28 Jun 2018 10:50:36 +0000 (03:50 -0700)] 
DRILL-6549: batch sizing for nested loop join

closes #1363

4 years agoDRILL-6519: Add String Distance and Phonetic Functions
Charles S. Givre [Thu, 5 Jul 2018 15:45:52 +0000 (11:45 -0400)] 
DRILL-6519: Add String Distance and Phonetic Functions

closes #1331

4 years agoDRILL-6575: Add store.hive.conf.properties option to allow set Hive properties at...
Arina Ielchiieva [Thu, 5 Jul 2018 20:48:37 +0000 (20:48 +0000)] 
DRILL-6575: Add store.hive.conf.properties option to allow set Hive properties at session level

closes #1365

4 years agoDRILL-6570: Fixed IndexOutofBoundException in Parquet Reader
Salim Achouche [Sat, 30 Jun 2018 02:58:12 +0000 (19:58 -0700)] 
DRILL-6570: Fixed IndexOutofBoundException in Parquet Reader

4 years agoDRILL-5977: Implement Filter Pushdown in Drill-Kafka plugin
Abhishek Ravi [Sun, 8 Apr 2018 21:01:31 +0000 (14:01 -0700)] 
DRILL-5977: Implement Filter Pushdown in Drill-Kafka plugin

closes #1272

4 years agoDRILL-6534: Upgrade ZooKeeper patch version to 3.4.12 and add Apache Curator to depen...
Bohdan Kazydub [Tue, 26 Jun 2018 09:30:53 +0000 (12:30 +0300)] 
DRILL-6534: Upgrade ZooKeeper patch version to 3.4.12 and add Apache Curator to dependencyManagement

closes #1337

4 years agoDRILL-6577: Change Hash-Join fallback default to false
Ben-Zvi [Tue, 3 Jul 2018 01:30:47 +0000 (18:30 -0700)] 
DRILL-6577: Change Hash-Join fallback default to false

closes #1359

4 years agoDRILL-6553: Fix TopN for unnest operator
Volodymyr Vysotskyi [Thu, 28 Jun 2018 17:20:38 +0000 (20:20 +0300)] 
DRILL-6553: Fix TopN for unnest operator

closes #1353

4 years agoDRILL-6557: Use size in bytes during Hive statistics calculation if present
Arina Ielchiieva [Fri, 29 Jun 2018 15:25:59 +0000 (18:25 +0300)] 
DRILL-6557: Use size in bytes during Hive statistics calculation if present

1. Check size in bytes presence in stats before fetching input splits and use it if present.
2. Add log trace suggesting to use ANALYZE command before running queries if statistics is unavailable and Drill had to fetch all input splits.
3. Minor refactoring /  cleanup in HiveMetadataProvider class.

closes #1357

4 years agoDRILL-6546: Allow unnest function with nested columns and complex expressions
Volodymyr Vysotskyi [Thu, 14 Jun 2018 16:32:43 +0000 (19:32 +0300)] 
DRILL-6546: Allow unnest function with nested columns and complex expressions

Fix loss of projected names in right side of correlate when single field is projected

4 years agoDRILL-6576: Unnest reports incoming record counts incorrectly
Parth Chandra [Thu, 7 Jun 2018 20:30:01 +0000 (13:30 -0700)] 
DRILL-6576: Unnest reports incoming record counts incorrectly

This closes #1362

4 years agoDRILL-6494: Drill Plugins Handler
Vitalii Diravka [Tue, 3 Jul 2018 17:23:03 +0000 (20:23 +0300)] 
DRILL-6494: Drill Plugins Handler

- Storage Plugins Handler service is used op the Drill start-up stage and it updates storage plugins configs from
  storage-plugins-override.conf file. If plugins configs are present in the persistence store - they are updated,
  otherwise bootstrap plugins are updated and the result configs are loaded to persistence store. If the enabled
  status is absent in the storage-plugins-override.conf file, the last plugin config enabled status persists.
- 'drill.exec.storage.action_on_plugins_override_file' Boot option is added. This is the action, which should be
  performed on the storage-plugins-override.conf file after successful updating storage plugins configs.
  Possible values are: "none" (default), "rename" and "remove".
- The "NULL" issue with updating Hive plugin config by REST is solved. But clients are still being instantiated for disabled
  plugins - DRILL-6412.
- "org.honton.chas.hocon:jackson-dataformat-hocon" library is added for the proper deserializing HOCON conf file
- additional refactoring: "com.typesafe:config" and "org.apache.commons:commons-lang3" are placed into DependencyManagement
  block with proper versions; correct properties for metrics in "drill-override-example.conf" are specified

closes #1345

4 years agoDRILL-6561: Lateral excluding the columns from output container provided by projectio...
Sorabh Hamirwasia [Fri, 29 Jun 2018 17:27:55 +0000 (10:27 -0700)] 
DRILL-6561: Lateral excluding the columns from output container provided by projection push into rules

This closes #1356

4 years agoDRILL-6530: JVM crash with a query involving multiple json files with one file having...
Sorabh Hamirwasia [Tue, 26 Jun 2018 17:53:53 +0000 (10:53 -0700)] 
DRILL-6530: JVM crash with a query involving multiple json files with one file having a schema change of one column from string to list

This closes #1343

4 years agoDRILL-6535: ClassCastException in Lateral Unnest queries when dealing with schema...
Sorabh Hamirwasia [Fri, 22 Jun 2018 05:57:00 +0000 (22:57 -0700)] 
DRILL-6535: ClassCastException in Lateral Unnest queries when dealing with schema changed json data Note: The issue was happening because for a left incoming all right batches were filtered and hence outputIndex was still 0 when new left incoming came with OK_NEW_SCHEMA. The OK_NEW_SCHEMA change was consumed without updating output container schema.

This closes #1339

4 years agoDRILL-6310: limit batch size for hash aggregate
Padma Penumarthy [Tue, 26 Jun 2018 21:00:07 +0000 (14:00 -0700)] 
DRILL-6310: limit batch size for hash aggregate

closes #1324

4 years agoDRILL-6537: Limit the batch size for buffering operators based on how much memory...
Padma Penumarthy [Sun, 1 Jul 2018 16:43:40 +0000 (09:43 -0700)] 
DRILL-6537: Limit the batch size for buffering operators based on how much memory they get

closes #1342

4 years agoDRILL-6554: Minor code improvements in parquet statistics handling
Vlad Rozov [Wed, 27 Jun 2018 17:45:49 +0000 (10:45 -0700)] 
DRILL-6554: Minor code improvements in parquet statistics handling

closes #1349

4 years agoDRILL-6548: IllegalStateException: Unexpected EMIT outcome received in buildSchema...
Sorabh Hamirwasia [Fri, 29 Jun 2018 05:20:32 +0000 (22:20 -0700)] 
DRILL-6548: IllegalStateException: Unexpected EMIT outcome received in buildSchema phase

closes #1352

4 years agoDRILL-6545: Projection Push down into Lateral Join operator.
HanumathRao [Fri, 22 Jun 2018 01:42:24 +0000 (18:42 -0700)] 
DRILL-6545: Projection Push down into Lateral Join operator.

closes #1347

4 years agoDRILL-4580: Support for exporting storage plugin configurations
agirish [Mon, 10 Apr 2017 00:42:57 +0000 (17:42 -0700)] 
DRILL-4580: Support for exporting storage plugin configurations

closes #1350

4 years agoDRILL-4020: The not-equal operator returns incorrect results when used on the HBase...
Akihiko Kusanagi [Tue, 5 Jun 2018 14:12:13 +0000 (23:12 +0900)] 
DRILL-4020: The not-equal operator returns incorrect results when used on the HBase row key

- Added a condition that checks if the filter to the scan specification doesn't have NOT_EQUAL operator
- Added testFilterPushDownRowKeyNotEqual() to TestHBaseFilterPushDown

closes #309

4 years agoDRILL-6147: Adding Columnar Parquet Batch Sizing functionality
Salim Achouche [Wed, 20 Jun 2018 02:23:41 +0000 (19:23 -0700)] 
DRILL-6147: Adding Columnar Parquet Batch Sizing functionality

closes #1330

4 years agoDRILL-6539: Record count not set for this vector container error
Padma Penumarthy [Tue, 26 Jun 2018 04:27:48 +0000 (21:27 -0700)] 
DRILL-6539: Record count not set for this vector container error

closes #1340

4 years agoDRILL-6512: Remove unnecessary processing overhead from RecordBatchSizer
Padma Penumarthy [Tue, 26 Jun 2018 02:06:51 +0000 (19:06 -0700)] 
DRILL-6512: Remove unnecessary processing overhead from RecordBatchSizer

closes #1341

4 years agoDRILL-6498: Support for EMIT outcome in ExternalSortBatch
Sorabh Hamirwasia [Thu, 28 Jun 2018 17:54:31 +0000 (10:54 -0700)] 
DRILL-6498: Support for EMIT outcome in ExternalSortBatch

* DRILL-6498: Support for EMIT outcome in ExternalSortBatch
* Updated TestTopNEmitOutcome to use RowSetComparison for comparing expected and actual output batches produced

closes #1323

4 years agoDRILL-6526: Refactor FileSystemConfig to disallow direct access from the code to...
Arina Ielchiieva [Fri, 22 Jun 2018 16:36:41 +0000 (19:36 +0300)] 
DRILL-6526: Refactor FileSystemConfig to disallow direct access from the code to its variables

4 years agoDRILL-6503: Performance improvements in lateral
Sorabh Hamirwasia [Wed, 27 Jun 2018 06:19:25 +0000 (23:19 -0700)] 
DRILL-6503: Performance improvements in lateral

closes #1328

4 years agoDRILL-6515: Render a link between the Unnest operator and it's source
Kunal Khatua [Tue, 26 Jun 2018 20:21:13 +0000 (13:21 -0700)] 
DRILL-6515: Render a link between the Unnest operator and it's source

* DRILL-6515: Render a link between the Unnest operator and it's source

1. Unnest operator is expected to specify srcOp=##-## to help identify the source of its input.
2. Implicit Source Map is leveraged to capture implicit data flow pipelines between Unnest and Lateral. (Can be expanded to other operators too).
3. In addition, code refactored for more readability.

* Minor comment updates

* Minor Update

Converted implicitSrc to a local variable

closes #1327

4 years agoDRILL-6479: Support EMIT for the Hash Aggr 1311/head
Ben-Zvi [Mon, 4 Jun 2018 21:07:47 +0000 (14:07 -0700)] 
DRILL-6479: Support EMIT for the Hash Aggr

closes #1311

4 years agoDRILL-6454: Native MapR DB plugin support for Hive MapR-DB json table
Vitalii Diravka [Fri, 1 Jun 2018 03:39:44 +0000 (20:39 -0700)] 
DRILL-6454: Native MapR DB plugin support for Hive MapR-DB json table

closes #1314

4 years agoDRILL-6340 Output Batch Control in Project using the RecordBatchSizer
karthik [Wed, 21 Mar 2018 20:52:03 +0000 (13:52 -0700)] 
DRILL-6340 Output Batch Control in Project using the RecordBatchSizer

Changes required to implement Output Batch Sizing in Project using the RecordBatchSizer.

closes #1302

4 years agoDRILL-6486: BitVector split and transfer does not work correctly for non byte-multipl...
karthik [Thu, 7 Jun 2018 20:23:04 +0000 (13:23 -0700)] 
DRILL-6486: BitVector split and transfer does not work correctly for non byte-multiple transfer lengths

Fix for the bug in BitVector splitAndTransfer. The logic for handling copy of last-n bits was incorrect for none byte-multiple transfer lengths.

closes #1316

4 years agoDRILL-6523: Fix NPE for describe of partial schema
Arina Ielchiieva [Thu, 21 Jun 2018 14:04:13 +0000 (17:04 +0300)] 
DRILL-6523: Fix NPE for describe of partial schema

closes #1332

4 years agoDRILL-6513: Max query memory per node set to Drillbit's maximum direct memory
Salim Achouche [Tue, 19 Jun 2018 20:10:06 +0000 (13:10 -0700)] 
DRILL-6513: Max query memory per node set to Drillbit's maximum direct memory

closes #1329

4 years agoDRILL-6502: Rename CorrelatePrel to LateralJoinPrel.
Hanumath Rao Maduri [Fri, 15 Jun 2018 19:05:09 +0000 (12:05 -0700)] 
DRILL-6502: Rename CorrelatePrel to LateralJoinPrel.

closes #1325

4 years agoDRILL-6491: Prevent merge join for full outer join at planning stage
Bohdan Kazydub [Tue, 12 Jun 2018 15:58:24 +0000 (18:58 +0300)] 
DRILL-6491: Prevent merge join for full outer join at planning stage

closes #1320

4 years agoDRILL-6470: Remove defunct repository
Kunal Khatua [Mon, 18 Jun 2018 17:36:43 +0000 (10:36 -0700)] 
DRILL-6470: Remove defunct repository

When build the Drill source code, Parquet jars from the repository hosted at http://repo.dremio.com/release/  is inaccessible.
These artifacts (parquet libraries) are now available within http://maven.corp.maprtech.com/nexus/content/groups/public/org/apache/parquet/

closes #1326

4 years agoDRILL-5188: Expand sub-queries using rules
Volodymyr Vysotskyi [Fri, 18 May 2018 12:54:16 +0000 (15:54 +0300)] 
DRILL-5188: Expand sub-queries using rules

- Add check for agg with group by literal
- Allow NLJ for limit 1
- Implement single_value aggregate function

closes #1321

4 years agoDRILL-6212: Prevent recursive cast expressions
chunhui-shi [Wed, 6 Jun 2018 19:25:31 +0000 (12:25 -0700)] 
DRILL-6212: Prevent recursive cast expressions

closes #1319

4 years agoDRILL-6455: Add missing JDBC Scan Operator for profiles
Kunal Khatua [Mon, 11 Jun 2018 06:36:10 +0000 (23:36 -0700)] 
DRILL-6455: Add missing JDBC Scan Operator for profiles

The operator is missing in the profile protobuf. This commit introduces that.
1. Added protobuf files (incl generated C++ and Java)
2. Updated JdbcSubScan's getOperatorType API

closes #1297

4 years agoDRILL-6468: CatastrophicFailures should not do a graceful shutdown of drill when...
Timothy Farkas [Tue, 5 Jun 2018 20:50:35 +0000 (13:50 -0700)] 
DRILL-6468: CatastrophicFailures should not do a graceful shutdown of drill when terminating the JVM.

closes #1306

4 years agoDRILL-6487: Limit estimateRowCount should not return negative rowcount
Gautam Parai [Fri, 15 Jun 2018 00:26:32 +0000 (17:26 -0700)] 
DRILL-6487: Limit estimateRowCount should not return negative rowcount

closes #1322

4 years agoDRILL-6353: Upgrade Parquet MR dependencies
Vlad Rozov [Wed, 9 May 2018 20:24:11 +0000 (13:24 -0700)] 
DRILL-6353: Upgrade Parquet MR dependencies

closes #1259

4 years agoDRILL-6474: Don't use TopN when order by and offset are used without a limit specified.
Timothy Farkas [Wed, 6 Jun 2018 19:04:39 +0000 (12:04 -0700)] 
DRILL-6474: Don't use TopN when order by and offset are used without a limit specified.

closes #1313

4 years agoDRILL-6488 - change instances of "template inline" to just "template"
Patrick Wong [Tue, 12 Jun 2018 00:05:19 +0000 (17:05 -0700)] 
DRILL-6488 - change instances of "template inline" to just "template"

closes #1317

4 years agoDRILL-6476: Generate explain plan which shows relation between Lateral and the corres...
Hanumath Rao Maduri [Thu, 7 Jun 2018 03:48:12 +0000 (20:48 -0700)] 
DRILL-6476: Generate explain plan which shows relation between Lateral and the corresponding Unnest.

closes #1308

4 years agoDRILL-6477: Drillbit crashes with OOME (Heap) for a large WebUI query
Kunal Khatua [Thu, 7 Jun 2018 20:32:00 +0000 (13:32 -0700)] 
DRILL-6477: Drillbit crashes with OOME (Heap) for a large WebUI query

For queries submitted through the WebUI and retrieving a large result-set, the Drillbit often hangs or crashes due to the (foreman) Drillbit running out of Heap memory.

This is because the Web client translates the result set into a massive object in the heap-space and tries to send that back to the browser. This results in the VM thread actively trying to perform GC if the memory is not sufficient.

The workaround is to have the active webConnection of the query periodically timeout to allow for checking the consumed heap-space. A level of 0.85 (i.e. 85%) is set as default threshold, crossing which, a query submitted through the REST API is marked and failed.
In addition, a user exception is thrown, indicting the cause of the query failing, along with alternative suggestions for re-executing the query.

closes #1309

4 years agoDRILL-6489: Fix filter push down for Hbase & Mapr-DB binary tables when convert funct...
Arina Ielchiieva [Tue, 12 Jun 2018 14:46:34 +0000 (14:46 +0000)] 
DRILL-6489: Fix filter push down for Hbase & Mapr-DB binary tables when convert function is used in a view

4 years agoDRILL-6478: enhance debug logs for batch sizing
Padma Penumarthy [Thu, 7 Jun 2018 21:04:47 +0000 (14:04 -0700)] 
DRILL-6478: enhance debug logs for batch sizing

closes #1310

4 years agoDRILL-6481: Rename ParquetXXXPredicates to ParquetXXXPredicate
Vlad Rozov [Fri, 8 Jun 2018 01:14:01 +0000 (18:14 -0700)] 
DRILL-6481: Rename ParquetXXXPredicates to ParquetXXXPredicate

closes #1312

4 years agoDRILL-6481: Refactor ParquetXXXPredicate classes
Vlad Rozov [Wed, 6 Jun 2018 22:03:42 +0000 (15:03 -0700)] 
DRILL-6481: Refactor ParquetXXXPredicate classes

closes #1312

4 years agoDRILL-6459: Unable to view profile of a running query
Kunal Khatua [Sat, 2 Jun 2018 00:02:58 +0000 (17:02 -0700)] 
DRILL-6459: Unable to view profile of a running query

Fixes the missing text component of the QueryId that causes lookups to fail in `WorkManager.queries` map.
This got introduced with the fix (#1265) for DRILL-5305

Reverting change to QueryIdHelper and DRILL-5305

Removing the changes done, based on inputs from @vrozov and @sohami . The correct approach would be to have this as part of the profile to avoid serialization of the queryIdText for each RPC making use of the QueryId

UX Changes
Set the query ID string and display in WebUI

closes #1301

4 years agoDRILL-6485: Fix typo in drill-env.sh
Paul Rogers [Mon, 11 Jun 2018 04:25:04 +0000 (21:25 -0700)] 
DRILL-6485: Fix typo in drill-env.sh

closes #1315

4 years agoDRILL-6446: Support for EMIT outcome in TopN
Sorabh Hamirwasia [Wed, 9 May 2018 02:09:38 +0000 (19:09 -0700)] 
DRILL-6446: Support for EMIT outcome in TopN
- Added comments for TopNBatch and PriorityQueueTemplate
- Adding support for SchemaChange across next() call with HyperVector in incoming container. This is achieved by adding a new method in HyperVectorWrapper which just updates the vector[] array
  holding multiple vectors with provided input ValueVector array. And also modifying RemovingRecordBatch GenericSV4Copier to hold reference to VectorWrapper instead of ValueVector[] for each column in incoming batch
- Handling empty batches. Two cases like empty batches in the begining with EMIT outcome and empty batches between consecutive EMIT outcome but after receiving some batches with data and EMIT outcome.
   Note: In first case of empty batch it was only returning EMIT outcome without properly creating the output container and SV4 vector. Because of that there could be a case where let's say first batch with EMIT outcome  is empty then TopN will return an empty batch with SV mode NONE and if later batch comes with some records and EMIT outcome, that will generate output batch with OK_NEW_SCHEMA (since TopN always
   generate first output batch with records with OK_NEW_SCHEMA as it returns output with SV4 mode). Also let's consider both batch with EMIT outcome were produced after processing first 2 rows of an input batch.
This is a problem as this is simulating schema change across rows of same incoming batch which will never be the case.

Note: In second case of empty batches priority queue will not be null and will be uninitialized. Also optimize to send EMIT outcome with output batch which has all the data to return for current iteration
                rather than sending it with OK followed by empty batch with EMIT outcome.

closes #1293

4 years agoDRILL-6463 : Fix integer overflow in MockGroupScanPOP
Gautam Parai [Sat, 2 Jun 2018 19:06:21 +0000 (12:06 -0700)] 
DRILL-6463 : Fix integer overflow in MockGroupScanPOP

closes #1303

4 years agoDRILL-6375 : Support for ANY_VALUE aggregate function
Gautam Parai [Wed, 4 Apr 2018 02:18:31 +0000 (19:18 -0700)] 
DRILL-6375 : Support for ANY_VALUE aggregate function

closes #1256

4 years agoDRILL-6466: Add HttpOnly flag to response cookies
Arina Ielchiieva [Tue, 5 Jun 2018 14:35:24 +0000 (17:35 +0300)] 
DRILL-6466: Add HttpOnly flag to response cookies

4 years agoDRILL-6467: Percentage usage of memory is reported as zero by the WebUI
Kunal Khatua [Thu, 7 Jun 2018 04:30:07 +0000 (21:30 -0700)] 
DRILL-6467: Percentage usage of memory is reported as zero by the WebUI

The memory reported as a percentage is incorrectly reported as 0%, irrespective of the actual usage.
closes #1305

4 years agoDRILL-6438: Remove excess logging form the tests.
Timothy Farkas [Tue, 22 May 2018 23:04:02 +0000 (16:04 -0700)] 
DRILL-6438: Remove excess logging form the tests.
 - Removed usages of System.out and System.err from the test and replaced with loggers

closes #1284

4 years agoDRILL-6389: Fixed building javadocs
Timothy Farkas [Tue, 8 May 2018 18:11:33 +0000 (11:11 -0700)] 
DRILL-6389: Fixed building javadocs
 - Added documentation about how to build javadocs
 - Fixed some of the javadoc warnings

closes #1276

4 years agoDRILL-5700: Adding nohup support for Sqlline script
kr-arjun [Fri, 1 Jun 2018 07:09:03 +0000 (00:09 -0700)] 
DRILL-5700: Adding nohup support for Sqlline script

closes #1300

4 years agoDRILL-6432: Show Button to print visualized query plan (#1278)
Kunal Khatua [Wed, 6 Jun 2018 15:43:27 +0000 (08:43 -0700)] 
DRILL-6432: Show Button to print visualized query plan (#1278)

* DRILL-6432: Show Button to print visualized query plan

Having the capability to only print the visualized query plan means the user can specifyi the zoom level (in the printer settings) when printing large complex plans that might span multiple pages.

* Update based on review comment

Padded button by 15px to avoid sticking to pane

closes #1278

4 years agoRevert "DRILL-4020: The not-equal operator returns incorrect results when used on...
Parth Chandra [Sun, 3 Jun 2018 00:41:45 +0000 (17:41 -0700)] 
Revert "DRILL-4020: The not-equal operator returns incorrect results when used on the HBase row key"

This reverts commit 0d5eda83fe34928ff60629e6a4903d43a1d82582.