drill.git
4 years agoDRILL-4020: The not-equal operator returns incorrect results when used on the HBase...
Akihiko Kusanagi [Tue, 5 Jun 2018 14:12:13 +0000 (23:12 +0900)] 
DRILL-4020: The not-equal operator returns incorrect results when used on the HBase row key

- Added a condition that checks if the filter to the scan specification doesn't have NOT_EQUAL operator
- Added testFilterPushDownRowKeyNotEqual() to TestHBaseFilterPushDown

closes #309

4 years agoDRILL-6147: Adding Columnar Parquet Batch Sizing functionality
Salim Achouche [Wed, 20 Jun 2018 02:23:41 +0000 (19:23 -0700)] 
DRILL-6147: Adding Columnar Parquet Batch Sizing functionality

closes #1330

4 years agoDRILL-6539: Record count not set for this vector container error
Padma Penumarthy [Tue, 26 Jun 2018 04:27:48 +0000 (21:27 -0700)] 
DRILL-6539: Record count not set for this vector container error

closes #1340

4 years agoDRILL-6512: Remove unnecessary processing overhead from RecordBatchSizer
Padma Penumarthy [Tue, 26 Jun 2018 02:06:51 +0000 (19:06 -0700)] 
DRILL-6512: Remove unnecessary processing overhead from RecordBatchSizer

closes #1341

4 years agoDRILL-6498: Support for EMIT outcome in ExternalSortBatch
Sorabh Hamirwasia [Thu, 28 Jun 2018 17:54:31 +0000 (10:54 -0700)] 
DRILL-6498: Support for EMIT outcome in ExternalSortBatch

* DRILL-6498: Support for EMIT outcome in ExternalSortBatch
* Updated TestTopNEmitOutcome to use RowSetComparison for comparing expected and actual output batches produced

closes #1323

4 years agoDRILL-6526: Refactor FileSystemConfig to disallow direct access from the code to...
Arina Ielchiieva [Fri, 22 Jun 2018 16:36:41 +0000 (19:36 +0300)] 
DRILL-6526: Refactor FileSystemConfig to disallow direct access from the code to its variables

4 years agoDRILL-6503: Performance improvements in lateral
Sorabh Hamirwasia [Wed, 27 Jun 2018 06:19:25 +0000 (23:19 -0700)] 
DRILL-6503: Performance improvements in lateral

closes #1328

4 years agoDRILL-6515: Render a link between the Unnest operator and it's source
Kunal Khatua [Tue, 26 Jun 2018 20:21:13 +0000 (13:21 -0700)] 
DRILL-6515: Render a link between the Unnest operator and it's source

* DRILL-6515: Render a link between the Unnest operator and it's source

1. Unnest operator is expected to specify srcOp=##-## to help identify the source of its input.
2. Implicit Source Map is leveraged to capture implicit data flow pipelines between Unnest and Lateral. (Can be expanded to other operators too).
3. In addition, code refactored for more readability.

* Minor comment updates

* Minor Update

Converted implicitSrc to a local variable

closes #1327

4 years agoDRILL-6479: Support EMIT for the Hash Aggr 1311/head
Ben-Zvi [Mon, 4 Jun 2018 21:07:47 +0000 (14:07 -0700)] 
DRILL-6479: Support EMIT for the Hash Aggr

closes #1311

4 years agoDRILL-6454: Native MapR DB plugin support for Hive MapR-DB json table
Vitalii Diravka [Fri, 1 Jun 2018 03:39:44 +0000 (20:39 -0700)] 
DRILL-6454: Native MapR DB plugin support for Hive MapR-DB json table

closes #1314

4 years agoDRILL-6340 Output Batch Control in Project using the RecordBatchSizer
karthik [Wed, 21 Mar 2018 20:52:03 +0000 (13:52 -0700)] 
DRILL-6340 Output Batch Control in Project using the RecordBatchSizer

Changes required to implement Output Batch Sizing in Project using the RecordBatchSizer.

closes #1302

4 years agoDRILL-6486: BitVector split and transfer does not work correctly for non byte-multipl...
karthik [Thu, 7 Jun 2018 20:23:04 +0000 (13:23 -0700)] 
DRILL-6486: BitVector split and transfer does not work correctly for non byte-multiple transfer lengths

Fix for the bug in BitVector splitAndTransfer. The logic for handling copy of last-n bits was incorrect for none byte-multiple transfer lengths.

closes #1316

4 years agoDRILL-6523: Fix NPE for describe of partial schema
Arina Ielchiieva [Thu, 21 Jun 2018 14:04:13 +0000 (17:04 +0300)] 
DRILL-6523: Fix NPE for describe of partial schema

closes #1332

4 years agoDRILL-6513: Max query memory per node set to Drillbit's maximum direct memory
Salim Achouche [Tue, 19 Jun 2018 20:10:06 +0000 (13:10 -0700)] 
DRILL-6513: Max query memory per node set to Drillbit's maximum direct memory

closes #1329

4 years agoDRILL-6502: Rename CorrelatePrel to LateralJoinPrel.
Hanumath Rao Maduri [Fri, 15 Jun 2018 19:05:09 +0000 (12:05 -0700)] 
DRILL-6502: Rename CorrelatePrel to LateralJoinPrel.

closes #1325

4 years agoDRILL-6491: Prevent merge join for full outer join at planning stage
Bohdan Kazydub [Tue, 12 Jun 2018 15:58:24 +0000 (18:58 +0300)] 
DRILL-6491: Prevent merge join for full outer join at planning stage

closes #1320

4 years agoDRILL-6470: Remove defunct repository
Kunal Khatua [Mon, 18 Jun 2018 17:36:43 +0000 (10:36 -0700)] 
DRILL-6470: Remove defunct repository

When build the Drill source code, Parquet jars from the repository hosted at http://repo.dremio.com/release/  is inaccessible.
These artifacts (parquet libraries) are now available within http://maven.corp.maprtech.com/nexus/content/groups/public/org/apache/parquet/

closes #1326

4 years agoDRILL-5188: Expand sub-queries using rules
Volodymyr Vysotskyi [Fri, 18 May 2018 12:54:16 +0000 (15:54 +0300)] 
DRILL-5188: Expand sub-queries using rules

- Add check for agg with group by literal
- Allow NLJ for limit 1
- Implement single_value aggregate function

closes #1321

4 years agoDRILL-6212: Prevent recursive cast expressions
chunhui-shi [Wed, 6 Jun 2018 19:25:31 +0000 (12:25 -0700)] 
DRILL-6212: Prevent recursive cast expressions

closes #1319

4 years agoDRILL-6455: Add missing JDBC Scan Operator for profiles
Kunal Khatua [Mon, 11 Jun 2018 06:36:10 +0000 (23:36 -0700)] 
DRILL-6455: Add missing JDBC Scan Operator for profiles

The operator is missing in the profile protobuf. This commit introduces that.
1. Added protobuf files (incl generated C++ and Java)
2. Updated JdbcSubScan's getOperatorType API

closes #1297

4 years agoDRILL-6468: CatastrophicFailures should not do a graceful shutdown of drill when...
Timothy Farkas [Tue, 5 Jun 2018 20:50:35 +0000 (13:50 -0700)] 
DRILL-6468: CatastrophicFailures should not do a graceful shutdown of drill when terminating the JVM.

closes #1306

4 years agoDRILL-6487: Limit estimateRowCount should not return negative rowcount
Gautam Parai [Fri, 15 Jun 2018 00:26:32 +0000 (17:26 -0700)] 
DRILL-6487: Limit estimateRowCount should not return negative rowcount

closes #1322

4 years agoDRILL-6353: Upgrade Parquet MR dependencies
Vlad Rozov [Wed, 9 May 2018 20:24:11 +0000 (13:24 -0700)] 
DRILL-6353: Upgrade Parquet MR dependencies

closes #1259

4 years agoDRILL-6474: Don't use TopN when order by and offset are used without a limit specified.
Timothy Farkas [Wed, 6 Jun 2018 19:04:39 +0000 (12:04 -0700)] 
DRILL-6474: Don't use TopN when order by and offset are used without a limit specified.

closes #1313

4 years agoDRILL-6488 - change instances of "template inline" to just "template"
Patrick Wong [Tue, 12 Jun 2018 00:05:19 +0000 (17:05 -0700)] 
DRILL-6488 - change instances of "template inline" to just "template"

closes #1317

4 years agoDRILL-6476: Generate explain plan which shows relation between Lateral and the corres...
Hanumath Rao Maduri [Thu, 7 Jun 2018 03:48:12 +0000 (20:48 -0700)] 
DRILL-6476: Generate explain plan which shows relation between Lateral and the corresponding Unnest.

closes #1308

4 years agoDRILL-6477: Drillbit crashes with OOME (Heap) for a large WebUI query
Kunal Khatua [Thu, 7 Jun 2018 20:32:00 +0000 (13:32 -0700)] 
DRILL-6477: Drillbit crashes with OOME (Heap) for a large WebUI query

For queries submitted through the WebUI and retrieving a large result-set, the Drillbit often hangs or crashes due to the (foreman) Drillbit running out of Heap memory.

This is because the Web client translates the result set into a massive object in the heap-space and tries to send that back to the browser. This results in the VM thread actively trying to perform GC if the memory is not sufficient.

The workaround is to have the active webConnection of the query periodically timeout to allow for checking the consumed heap-space. A level of 0.85 (i.e. 85%) is set as default threshold, crossing which, a query submitted through the REST API is marked and failed.
In addition, a user exception is thrown, indicting the cause of the query failing, along with alternative suggestions for re-executing the query.

closes #1309

4 years agoDRILL-6489: Fix filter push down for Hbase & Mapr-DB binary tables when convert funct...
Arina Ielchiieva [Tue, 12 Jun 2018 14:46:34 +0000 (14:46 +0000)] 
DRILL-6489: Fix filter push down for Hbase & Mapr-DB binary tables when convert function is used in a view

4 years agoDRILL-6478: enhance debug logs for batch sizing
Padma Penumarthy [Thu, 7 Jun 2018 21:04:47 +0000 (14:04 -0700)] 
DRILL-6478: enhance debug logs for batch sizing

closes #1310

4 years agoDRILL-6481: Rename ParquetXXXPredicates to ParquetXXXPredicate
Vlad Rozov [Fri, 8 Jun 2018 01:14:01 +0000 (18:14 -0700)] 
DRILL-6481: Rename ParquetXXXPredicates to ParquetXXXPredicate

closes #1312

4 years agoDRILL-6481: Refactor ParquetXXXPredicate classes
Vlad Rozov [Wed, 6 Jun 2018 22:03:42 +0000 (15:03 -0700)] 
DRILL-6481: Refactor ParquetXXXPredicate classes

closes #1312

4 years agoDRILL-6459: Unable to view profile of a running query
Kunal Khatua [Sat, 2 Jun 2018 00:02:58 +0000 (17:02 -0700)] 
DRILL-6459: Unable to view profile of a running query

Fixes the missing text component of the QueryId that causes lookups to fail in `WorkManager.queries` map.
This got introduced with the fix (#1265) for DRILL-5305

Reverting change to QueryIdHelper and DRILL-5305

Removing the changes done, based on inputs from @vrozov and @sohami . The correct approach would be to have this as part of the profile to avoid serialization of the queryIdText for each RPC making use of the QueryId

UX Changes
Set the query ID string and display in WebUI

closes #1301

4 years agoDRILL-6485: Fix typo in drill-env.sh
Paul Rogers [Mon, 11 Jun 2018 04:25:04 +0000 (21:25 -0700)] 
DRILL-6485: Fix typo in drill-env.sh

closes #1315

4 years agoDRILL-6446: Support for EMIT outcome in TopN
Sorabh Hamirwasia [Wed, 9 May 2018 02:09:38 +0000 (19:09 -0700)] 
DRILL-6446: Support for EMIT outcome in TopN
- Added comments for TopNBatch and PriorityQueueTemplate
- Adding support for SchemaChange across next() call with HyperVector in incoming container. This is achieved by adding a new method in HyperVectorWrapper which just updates the vector[] array
  holding multiple vectors with provided input ValueVector array. And also modifying RemovingRecordBatch GenericSV4Copier to hold reference to VectorWrapper instead of ValueVector[] for each column in incoming batch
- Handling empty batches. Two cases like empty batches in the begining with EMIT outcome and empty batches between consecutive EMIT outcome but after receiving some batches with data and EMIT outcome.
   Note: In first case of empty batch it was only returning EMIT outcome without properly creating the output container and SV4 vector. Because of that there could be a case where let's say first batch with EMIT outcome  is empty then TopN will return an empty batch with SV mode NONE and if later batch comes with some records and EMIT outcome, that will generate output batch with OK_NEW_SCHEMA (since TopN always
   generate first output batch with records with OK_NEW_SCHEMA as it returns output with SV4 mode). Also let's consider both batch with EMIT outcome were produced after processing first 2 rows of an input batch.
This is a problem as this is simulating schema change across rows of same incoming batch which will never be the case.

Note: In second case of empty batches priority queue will not be null and will be uninitialized. Also optimize to send EMIT outcome with output batch which has all the data to return for current iteration
                rather than sending it with OK followed by empty batch with EMIT outcome.

closes #1293

4 years agoDRILL-6463 : Fix integer overflow in MockGroupScanPOP
Gautam Parai [Sat, 2 Jun 2018 19:06:21 +0000 (12:06 -0700)] 
DRILL-6463 : Fix integer overflow in MockGroupScanPOP

closes #1303

4 years agoDRILL-6375 : Support for ANY_VALUE aggregate function
Gautam Parai [Wed, 4 Apr 2018 02:18:31 +0000 (19:18 -0700)] 
DRILL-6375 : Support for ANY_VALUE aggregate function

closes #1256

4 years agoDRILL-6466: Add HttpOnly flag to response cookies
Arina Ielchiieva [Tue, 5 Jun 2018 14:35:24 +0000 (17:35 +0300)] 
DRILL-6466: Add HttpOnly flag to response cookies

4 years agoDRILL-6467: Percentage usage of memory is reported as zero by the WebUI
Kunal Khatua [Thu, 7 Jun 2018 04:30:07 +0000 (21:30 -0700)] 
DRILL-6467: Percentage usage of memory is reported as zero by the WebUI

The memory reported as a percentage is incorrectly reported as 0%, irrespective of the actual usage.
closes #1305

4 years agoDRILL-6438: Remove excess logging form the tests.
Timothy Farkas [Tue, 22 May 2018 23:04:02 +0000 (16:04 -0700)] 
DRILL-6438: Remove excess logging form the tests.
 - Removed usages of System.out and System.err from the test and replaced with loggers

closes #1284

4 years agoDRILL-6389: Fixed building javadocs
Timothy Farkas [Tue, 8 May 2018 18:11:33 +0000 (11:11 -0700)] 
DRILL-6389: Fixed building javadocs
 - Added documentation about how to build javadocs
 - Fixed some of the javadoc warnings

closes #1276

4 years agoDRILL-5700: Adding nohup support for Sqlline script
kr-arjun [Fri, 1 Jun 2018 07:09:03 +0000 (00:09 -0700)] 
DRILL-5700: Adding nohup support for Sqlline script

closes #1300

4 years agoDRILL-6432: Show Button to print visualized query plan (#1278)
Kunal Khatua [Wed, 6 Jun 2018 15:43:27 +0000 (08:43 -0700)] 
DRILL-6432: Show Button to print visualized query plan (#1278)

* DRILL-6432: Show Button to print visualized query plan

Having the capability to only print the visualized query plan means the user can specifyi the zoom level (in the printer settings) when printing large complex plans that might span multiple pages.

* Update based on review comment

Padded button by 15px to avoid sticking to pane

closes #1278

4 years agoRevert "DRILL-4020: The not-equal operator returns incorrect results when used on...
Parth Chandra [Sun, 3 Jun 2018 00:41:45 +0000 (17:41 -0700)] 
Revert "DRILL-4020: The not-equal operator returns incorrect results when used on the HBase row key"

This reverts commit 0d5eda83fe34928ff60629e6a4903d43a1d82582.

4 years agoDRILL-5584: Add branding and versioning information for windows C++ Client.
Rob Wu [Thu, 16 Nov 2017 02:21:48 +0000 (18:21 -0800)] 
DRILL-5584: Add branding and versioning information for windows C++ Client.

This closes #5584

4 years agoDRILL-5924: native-client: Support user-specified CXX_FLAGS
Korn, Uwe [Sat, 28 Oct 2017 19:45:51 +0000 (21:45 +0200)] 
DRILL-5924: native-client: Support user-specified CXX_FLAGS

This closes #1022

4 years agoDRILL-4020: The not-equal operator returns incorrect results when used on the HBase...
Akihiko Kusanagi [Wed, 4 Nov 2015 12:33:06 +0000 (21:33 +0900)] 
DRILL-4020: The not-equal operator returns incorrect results when used on the HBase row key

- Added a condition that checks if the filter to the scan specification doesn't have NOT_EQUAL operator
- Added testFilterPushDownRowKeyNotEqual() to TestHBaseFilterPushDown

This closes #309

4 years agoDRILL-6456: Planner shouldn't create any exchanges on the right side of Lateral Join.
Hanumath Rao Maduri [Thu, 31 May 2018 00:59:51 +0000 (17:59 -0700)] 
DRILL-6456: Planner shouldn't create any exchanges on the right side of Lateral Join.

This closes #1299

4 years agoDRILL-6450: Visualized plans for profiles querying JDBC sources is broken
Kunal Khatua [Tue, 29 May 2018 18:20:56 +0000 (11:20 -0700)] 
DRILL-6450: Visualized plans for profiles querying JDBC sources is broken

When viewing a profile for a query against a JDBC source, the visualized plan is not rendered. This is because the generated SQL pushed down to the JDBC source has a line break injected just before the FROM clause.

The workaround is to strip away any injected newlines ('\\n') at least for the SQL defined in the text plan, so that the backend Javascript can render it correctly.
In addition, any single line comments are also removed, but any block comments (i.e. /* .. */ ) are retained as they might carry hints.

This closes #1295

4 years agoDRILL-6445: Fix existing test cases in TestScripts.java and add new test case for...
Sorabh Hamirwasia [Thu, 24 May 2018 18:58:01 +0000 (11:58 -0700)] 
DRILL-6445: Fix existing test cases in TestScripts.java and add new test case for DRILLBIT_CONTEXT variable

This closes #1289

4 years agoDRILL-6236:Batch sizing for hash join
Padma Penumarthy [Wed, 30 May 2018 21:00:16 +0000 (14:00 -0700)] 
DRILL-6236:Batch sizing for hash join

This closes #1227

4 years agoDRILL-6343: bit vector copyFromSafe is not doing realloc
Padma Penumarthy [Thu, 19 Apr 2018 21:36:38 +0000 (14:36 -0700)] 
DRILL-6343: bit vector copyFromSafe is not doing realloc

4 years agoDRILL-4364: Image Metadata Format Plugin - Initial commit of Image Metadata Format...
Akihiko Kusanagi [Sat, 6 Feb 2016 18:55:57 +0000 (03:55 +0900)] 
DRILL-4364: Image Metadata Format Plugin - Initial commit of Image Metadata Format Plugin - See https://issues.apache.org/jira/browse/DRILL-4364

This closes #367

4 years agoDRILL-4091: Adding support for additional gis operations in gis contrib module
potocki [Mon, 16 Nov 2015 13:05:18 +0000 (14:05 +0100)] 
DRILL-4091: Adding support for additional gis operations in gis contrib module

This closes #1201

4 years agoDRILL-6356: batch sizing for union all
Padma Penumarthy [Thu, 10 May 2018 22:28:40 +0000 (15:28 -0700)] 
DRILL-6356: batch sizing for union all

closes #1255

4 years agoDRILL-6402: Repeated Value Vectors copyFrom methods are not updating
Padma Penumarthy [Thu, 10 May 2018 22:29:27 +0000 (15:29 -0700)] 
DRILL-6402: Repeated Value Vectors copyFrom methods are not updating
the value count and writer index correctly for values vector

4 years agoDRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out (#1281)
Venkata Jyothsna Donapati [Wed, 30 May 2018 18:03:58 +0000 (11:03 -0700)] 
DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out (#1281)

DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out

4 years agoDRILL-3964 : Fix NPE in WriterRecordBatch when 0 rows
Gautam Parai [Fri, 25 May 2018 02:00:21 +0000 (19:00 -0700)] 
DRILL-3964 : Fix NPE in WriterRecordBatch when 0 rows

closes #1290

4 years agoDRILL-6442: Adjust Hbase disk cost & row count estimation when filter push down is...
Arina Ielchiieva [Wed, 23 May 2018 16:17:11 +0000 (16:17 +0000)] 
DRILL-6442: Adjust Hbase disk cost & row count estimation when filter push down is applied

closes #1288

4 years agoDRILL-5990: Drill should not startup with JRE sourced java executable
Kunal Khatua [Mon, 21 May 2018 21:19:53 +0000 (14:19 -0700)] 
DRILL-5990: Drill should not startup with JRE sourced java executable

Currently, the check for the Java compiler is made only during query execution.
This commit ensures that if the java executable used for instantiating the Drillbit is from a JRE (i.e. the Java compiler is not available), the Drillbit will fail with the exception reported in the logs:
```
Exception in thread "main" org.apache.drill.exec.exception.DrillbitStartupException: JDK Java compiler not available. Ensure Drill is running with the java executable from a JDK and not a JRE
        at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:138)
        at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:409)
        at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:389)
        at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:385)
```

4 years agoDRILL-6443: Enable Search for both running AND completed queries
Kunal Khatua [Wed, 23 May 2018 22:55:24 +0000 (15:55 -0700)] 
DRILL-6443: Enable Search for both running AND completed queries

When running a query in Drill, the `/profiles` page will show the search (and pagination) capabilities only for the top most visible table (i.e. Running Queries ).

The Completed Queries table will show the search feature only when there are no running queries. This is because the backend uses a generalized freemarker macro to define the seach capabilities for the tables being rendered. With running queries, both, running and completed queries tables have the same element ID, resulting in the search capability only being applied to the first table.

This modifies the Freemarker macro to take an additional argument for distinguishing between Running and Completed Queries table.

4 years agoDRILL-6437: Removed excess maven logging when downloading dependencies. This fixed...
Timothy Farkas [Wed, 23 May 2018 21:14:15 +0000 (14:14 -0700)] 
DRILL-6437: Removed excess maven logging when downloading dependencies. This fixed Travis failures due to log overflow.

4 years agoDRILL-6435: MappingSet is stateful, so it can't be shared between threads
Vlad Rozov [Wed, 23 May 2018 16:14:13 +0000 (09:14 -0700)] 
DRILL-6435: MappingSet is stateful, so it can't be shared between threads

4 years agoDRILL-6436: Storage Plugin to have name and context moved to AbstractStoragePlugin
chunhui-shi [Thu, 15 Feb 2018 20:21:33 +0000 (12:21 -0800)] 
DRILL-6436: Storage Plugin to have name and context moved to AbstractStoragePlugin

closes #1282

4 years agoDRILL-6440: Unnest unit tests and fixes for stats
Parth Chandra [Mon, 16 Apr 2018 23:42:20 +0000 (16:42 -0700)] 
DRILL-6440: Unnest unit tests and fixes for stats

  - Add unit test with mock input, nested lateral and unnest and project.
  - Fix unit test involving batch limits, ignore map tests
  - Fix input row count stats.

closes #1283

4 years agoDRILL-6374: Transitive Closure leads to TPCH Queries regressions and OOM when run...
Vitalii Diravka [Mon, 14 May 2018 12:27:16 +0000 (15:27 +0300)] 
DRILL-6374: Transitive Closure leads to TPCH Queries regressions and OOM when run concurrency test

- Test case for Directory Pruning with Transitive Predicates
- Improving description for STRICT_EQUAL_IS_DISTINCT_FROM predicate

closes #1262

4 years agoDRILL-6431: Unnest operator requires table and a single column alias to be specified.
hmaduri [Tue, 15 May 2018 20:51:45 +0000 (13:51 -0700)] 
DRILL-6431: Unnest operator requires table and a single column alias to be specified.
Fixing the issues related to star column renaming, same field name renaming
and also enforcing that an alias column is required for the unnest operator.

4 years agoDRILL-6419: E2E Integration test for Lateral&Unnest
Sorabh Hamirwasia [Tue, 15 May 2018 21:30:05 +0000 (14:30 -0700)] 
DRILL-6419: E2E Integration test for Lateral&Unnest

4 years agoDRILL-5846: Improve parquet performance for Flat Data Types
Salim Achouche [Wed, 18 Apr 2018 03:12:03 +0000 (20:12 -0700)] 
DRILL-5846: Improve parquet performance for Flat Data Types

closes #1060

4 years agoDRILL-6423: Export query result as a CSV file
Kunal Khatua [Wed, 16 May 2018 17:35:51 +0000 (10:35 -0700)] 
DRILL-6423: Export query result as a CSV file

Added option for user specified delimiter
Also, we show query Id on results page as alink for popping out a new window

closes #1266

4 years agoDRILL-5305: Query Profile must display Query ID
Kunal Khatua [Wed, 16 May 2018 17:22:41 +0000 (10:22 -0700)] 
DRILL-5305: Query Profile must display Query ID

Introduced change to the Protobuf to inject the text-equivalent of the QueryID into the profile. This way, the profile's file name can be changed, but restored back based on this new field.
The Profile UI also shows the Query ID, though this is not inferred from this new field, for sake of backward compatibility with older profiles.

closes #1265

4 years agoDRILL-6418: Handle Schema change in Unnest And Lateral for unnest field / non-unnest...
Sorabh Hamirwasia [Tue, 15 May 2018 21:27:31 +0000 (14:27 -0700)] 
DRILL-6418: Handle Schema change in Unnest And Lateral for unnest field / non-unnest field

Note: Changed Lateral to handle non-empty right batch with OK_NEW_SCHEMA

closes #1271

4 years agoDRILL-6348: Received batches are now owned by the receive operators instead of the...
Salim Achouche [Mon, 23 Apr 2018 01:02:35 +0000 (18:02 -0700)] 
DRILL-6348: Received batches are now owned by the receive operators instead of the parent

closes #1237

4 years agoDRILL-4525: Code cleanup
Bohdan Kazydub [Thu, 17 May 2018 10:08:31 +0000 (13:08 +0300)] 
DRILL-4525: Code cleanup

- Fixed failing test

closes #1268

4 years agoDRILL-4525: Allow SqlBetweenOperator to accept LOWER_OPERAND and UPPER_OPERAND with...
Hsuan-Yi Chu [Mon, 21 Mar 2016 21:43:54 +0000 (14:43 -0700)] 
DRILL-4525: Allow SqlBetweenOperator to accept LOWER_OPERAND and UPPER_OPERAND with different types

4 years agoDRILL-6424: Updating FasterXML Jackson libraries
Vitalii Diravka [Thu, 17 May 2018 09:59:08 +0000 (12:59 +0300)] 
DRILL-6424: Updating FasterXML Jackson libraries

closes #1274

4 years agoDRILL-6413: Update ParquetFilterBuilder.visitBooleanOperator to handle simplified...
Arina Ielchiieva [Thu, 17 May 2018 15:24:35 +0000 (18:24 +0300)] 
DRILL-6413: Update ParquetFilterBuilder.visitBooleanOperator to handle simplified boolean expression

closes #1269

4 years agoDRILL-6426: Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit...
Arina Ielchiieva [Fri, 18 May 2018 10:11:05 +0000 (13:11 +0300)] 
DRILL-6426: Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option

closes #1273

4 years agoDRILL-6421: Refactor DecimalUtility and CoreDecimalUtility classes
Volodymyr Vysotskyi [Wed, 16 May 2018 18:34:56 +0000 (21:34 +0300)] 
DRILL-6421: Refactor DecimalUtility and CoreDecimalUtility classes

closes #1267

4 years agoDRILL-6255: Drillbit while sending control message to itself creates a connection...
Sorabh Hamirwasia [Tue, 8 May 2018 20:06:20 +0000 (13:06 -0700)] 
DRILL-6255: Drillbit while sending control message to itself creates a connection instead of submitting locally

closes #1253

4 years agoDRILL-6420: Add Lateral and Unnest Keyword for highlighting on WebUI
Sorabh Hamirwasia [Fri, 4 May 2018 17:47:28 +0000 (10:47 -0700)] 
DRILL-6420: Add Lateral and Unnest Keyword for highlighting on WebUI

closes #1261

4 years agoDRILL-6027:
Ben-Zvi [Sat, 28 Apr 2018 04:59:25 +0000 (21:59 -0700)] 
DRILL-6027:
  - Added fallback option for HashJoin.
  - No copy of incoming for single partition, and avoid HT resize.
  - Fix memory leak when cancelling while spill file is read
  - get correct schema when probe side is empty
  - Re-create the HashJoinProbe

4 years agoDRILL-6027:
Timothy Farkas [Tue, 13 Feb 2018 00:49:29 +0000 (16:49 -0800)] 
DRILL-6027:
 - Added memory claculator
 - Added unit tests and docs.
 - Fixed IOB caused by output vector allocation.
 - Don't double count records that were spilled in HashJoin

4 years agoDRILL-6027: Initial implementation of HashJoin spill, without memory limits checks yet
Ben-Zvi [Wed, 14 Feb 2018 20:36:39 +0000 (12:36 -0800)] 
DRILL-6027: Initial implementation of HashJoin spill, without memory limits checks yet

4 years agocode review comments
Padma Penumarthy [Wed, 16 May 2018 00:58:13 +0000 (17:58 -0700)] 
code review comments

4 years agoDRILL-6411: Make batch memory sizing logs uniform across all operators
Padma Penumarthy [Mon, 14 May 2018 22:01:25 +0000 (15:01 -0700)] 
DRILL-6411: Make batch memory sizing logs uniform across all operators

4 years agoDRILL-6321: Lateral Join and Unnest - rules, options, logical plan supports
chunhui-shi [Mon, 30 Apr 2018 17:24:06 +0000 (10:24 -0700)] 
DRILL-6321: Lateral Join and Unnest - rules, options, logical plan supports

Included changes:
* Add planner.enable_unnest_lateral option. Default value set to false.
* Enable FilterCorrectRule
* Add support to logical plan
* Fix rebase errors for DRILL-6321 commits

4 years agoDRILL-6321: Lateral Join and Unnest - initial implementation for parser and planning
chunhui-shi [Mon, 26 Mar 2018 19:15:09 +0000 (12:15 -0700)] 
DRILL-6321: Lateral Join and Unnest - initial implementation for parser and planning

4 years agoDRILL-6272: Refactor dynamic UDFs and function initializer tests to generate needed...
Arina Ielchiieva [Sat, 5 May 2018 12:32:07 +0000 (15:32 +0300)] 
DRILL-6272: Refactor dynamic UDFs and function initializer tests to generate needed binary and source jars at runtime

close apache/drill#1225

4 years agoDRILL-6363: Upgrade jmockit and mockito libs
Arina Ielchiieva [Sat, 5 May 2018 12:31:10 +0000 (15:31 +0300)] 
DRILL-6363: Upgrade jmockit and mockito libs

4 years agoDRILL-6242 Use java.time.Local{Date|Time|DateTime} for Drill Date, Time, Timestamp...
jiang-wu [Tue, 1 May 2018 21:48:26 +0000 (14:48 -0700)] 
DRILL-6242 Use java.time.Local{Date|Time|DateTime} for Drill Date, Time, Timestamp types. (#3)

close apache/drill#1247

* DRILL-6242 - Use java.time.Local{Date|Time|DateTime} classes to hold values from corresponding Drill date, time, and timestamp types.
Conflicts:
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/ExtendedJsonOutput.java

Fix merge conflicts and check style.

4 years agoDRILL-6401: Precision for decimal data types may be lost for the case when cast with...
Volodymyr Vysotskyi [Thu, 10 May 2018 14:06:28 +0000 (17:06 +0300)] 
DRILL-6401: Precision for decimal data types may be lost for the case when cast with literal is used

close apache/drill#1254

4 years agoDRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart
Kunal Khatua [Fri, 27 Apr 2018 23:27:45 +0000 (16:27 -0700)] 
DRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart

As a follow up to DRILL-6289, the following improvements have been done:
1. When loading the page for the first time, the WebUI enables the shutdown button without actually checking the state of the Drillbits.
   The ideal behaviour should be to disable the button till the state is verified. [Done]
   If a Drillbit is confirmed down (i.e. not in `/state` response), it is marked as OFFLINE and button is disabled.
2. When shutting down the current Drillbit, the WebUI no more has access to the cluster state.
   The ideal behaviour here should be to fetch the state from any of the other Drillbits to update the status. [Done]
   With the current Drillbit down, the other bits are requested for cluster state info and update accordingly.
3. When a new, previously unseen Drillbit comes up, the WebUI will never render it because the table is statically generated during the first page load.
   The idea behaviour should be to append to the table on discovery of a new node. [Done]
   The new Drillbit info is injected and a prompt appears to refresh the page to re-populate any missing info. This also works with feature (2) mentioned above.

The only Java code change was to have the state response carry the address and http-port as a tuple, instead of the user-port (which is never used).

Updates based on review

1. Added descriptive comments to functions
2. Handled possible race condition from multiple httpRequests for cluster state
3. Eliminated unused stateKey variable
4. Best practice of using local (let) instead of global (var) objects, and substituting currentRow variable instead of the jQuery search.

Additional changes based on review

1. Random selection of Drillbits to query for state when primary Drillbit is down (limited to 3; with a timeout of 3 sec)
2. Indicate when an Offline Drillbit is de-registered, versus just Offline as per Zookeeper
3. Hide shutdown column when Authentication is enabled, but the user is NOT an Admin.
   When the column is visible, remote bits are disabled because
4. Metrics will be shown all the time (except HTTPS), because the information is available and a non-admin user would anyway not have actionable capabilities
5. Basic clean up on unused variables

Hide shutdown buttons for HTTPS scenarios

Fixed check for 'https'

Since, `location.protocol` returns a trailing ':' as well

Handle metrics lookup on local Drillbit for HTTPS

If the webpage is accessed via IP, the certificate is tied to the IP and not the FQDN, which is what is used for fetching the metrics.

close apache/drill#1241

4 years agoDRILL-6321: Customize Drill's conformance. Allow support to APPLY keywords
chunhui-shi [Fri, 2 Feb 2018 18:03:38 +0000 (10:03 -0800)] 
DRILL-6321: Customize Drill's conformance. Allow support to APPLY keywords

close apache/drill#1224

Conflicts:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java

Fix checkstyle

4 years agoDRILL-6249: Adding more unit testing documentation.
Timothy Farkas [Thu, 15 Mar 2018 21:27:55 +0000 (14:27 -0700)] 
DRILL-6249: Adding more unit testing documentation.

close apache/drill#1251

4 years agoDRILL-6386: Remove unused imports and star imports.
Drill Dev [Tue, 8 May 2018 20:57:37 +0000 (13:57 -0700)] 
DRILL-6386: Remove unused imports and star imports.

4 years agoDRILL-6386: Disallow unused imports and star imports.
Timothy Farkas [Tue, 8 May 2018 20:55:40 +0000 (13:55 -0700)] 
DRILL-6386: Disallow unused imports and star imports.

4 years agoDRILL-143: CGroup Support for Drill-on-YARN 440/head
Kunal Khatua [Mon, 23 Apr 2018 17:31:21 +0000 (10:31 -0700)] 
DRILL-143: CGroup Support for Drill-on-YARN

Initial patch
1. Minor fix up
2. Updated variable name for pid file
Revert changes to yarn-drillbit.sh
Based on discussion for PR #1239 (Ref: https://github.com/apache/drill/pull/1239)

closes #1239

4 years agoDRILL-6361: Revised typeOf() function versions
Paul Rogers [Sun, 29 Apr 2018 06:04:26 +0000 (23:04 -0700)] 
DRILL-6361: Revised typeOf() function versions

Added more unit tests.

Updated to handle VARDECIMAL
The VARDECIMAL type was recently added to Drill. Added support for this type. The sqlTypeOf() function now returns DECIMAL(p, s) for precision p, scale s.

closes #1242

4 years agoDRILL-6380: Fix sporadic mongo db hangs.
Timothy Farkas [Wed, 2 May 2018 08:21:10 +0000 (01:21 -0700)] 
DRILL-6380: Fix sporadic mongo db hangs.

closes #1249

4 years agoDRILL-6281: Introduce Collectors class for internal iterators
Vlad Rozov [Mon, 30 Apr 2018 14:11:20 +0000 (07:11 -0700)] 
DRILL-6281: Introduce Collectors class for internal iterators

closes #1238