drill.git
4 years agoDRILL-6481: Refactor ParquetXXXPredicate classes
Vlad Rozov [Wed, 6 Jun 2018 22:03:42 +0000 (15:03 -0700)] 
DRILL-6481: Refactor ParquetXXXPredicate classes

closes #1312

4 years agoDRILL-6459: Unable to view profile of a running query
Kunal Khatua [Sat, 2 Jun 2018 00:02:58 +0000 (17:02 -0700)] 
DRILL-6459: Unable to view profile of a running query

Fixes the missing text component of the QueryId that causes lookups to fail in `WorkManager.queries` map.
This got introduced with the fix (#1265) for DRILL-5305

Reverting change to QueryIdHelper and DRILL-5305

Removing the changes done, based on inputs from @vrozov and @sohami . The correct approach would be to have this as part of the profile to avoid serialization of the queryIdText for each RPC making use of the QueryId

UX Changes
Set the query ID string and display in WebUI

closes #1301

4 years agoDRILL-6485: Fix typo in drill-env.sh
Paul Rogers [Mon, 11 Jun 2018 04:25:04 +0000 (21:25 -0700)] 
DRILL-6485: Fix typo in drill-env.sh

closes #1315

4 years agoDRILL-6446: Support for EMIT outcome in TopN
Sorabh Hamirwasia [Wed, 9 May 2018 02:09:38 +0000 (19:09 -0700)] 
DRILL-6446: Support for EMIT outcome in TopN
- Added comments for TopNBatch and PriorityQueueTemplate
- Adding support for SchemaChange across next() call with HyperVector in incoming container. This is achieved by adding a new method in HyperVectorWrapper which just updates the vector[] array
  holding multiple vectors with provided input ValueVector array. And also modifying RemovingRecordBatch GenericSV4Copier to hold reference to VectorWrapper instead of ValueVector[] for each column in incoming batch
- Handling empty batches. Two cases like empty batches in the begining with EMIT outcome and empty batches between consecutive EMIT outcome but after receiving some batches with data and EMIT outcome.
   Note: In first case of empty batch it was only returning EMIT outcome without properly creating the output container and SV4 vector. Because of that there could be a case where let's say first batch with EMIT outcome  is empty then TopN will return an empty batch with SV mode NONE and if later batch comes with some records and EMIT outcome, that will generate output batch with OK_NEW_SCHEMA (since TopN always
   generate first output batch with records with OK_NEW_SCHEMA as it returns output with SV4 mode). Also let's consider both batch with EMIT outcome were produced after processing first 2 rows of an input batch.
This is a problem as this is simulating schema change across rows of same incoming batch which will never be the case.

Note: In second case of empty batches priority queue will not be null and will be uninitialized. Also optimize to send EMIT outcome with output batch which has all the data to return for current iteration
                rather than sending it with OK followed by empty batch with EMIT outcome.

closes #1293

4 years agoDRILL-6463 : Fix integer overflow in MockGroupScanPOP
Gautam Parai [Sat, 2 Jun 2018 19:06:21 +0000 (12:06 -0700)] 
DRILL-6463 : Fix integer overflow in MockGroupScanPOP

closes #1303

4 years agoDRILL-6375 : Support for ANY_VALUE aggregate function
Gautam Parai [Wed, 4 Apr 2018 02:18:31 +0000 (19:18 -0700)] 
DRILL-6375 : Support for ANY_VALUE aggregate function

closes #1256

4 years agoDRILL-6466: Add HttpOnly flag to response cookies
Arina Ielchiieva [Tue, 5 Jun 2018 14:35:24 +0000 (17:35 +0300)] 
DRILL-6466: Add HttpOnly flag to response cookies

4 years agoDRILL-6467: Percentage usage of memory is reported as zero by the WebUI
Kunal Khatua [Thu, 7 Jun 2018 04:30:07 +0000 (21:30 -0700)] 
DRILL-6467: Percentage usage of memory is reported as zero by the WebUI

The memory reported as a percentage is incorrectly reported as 0%, irrespective of the actual usage.
closes #1305

4 years agoDRILL-6438: Remove excess logging form the tests.
Timothy Farkas [Tue, 22 May 2018 23:04:02 +0000 (16:04 -0700)] 
DRILL-6438: Remove excess logging form the tests.
 - Removed usages of System.out and System.err from the test and replaced with loggers

closes #1284

4 years agoDRILL-6389: Fixed building javadocs
Timothy Farkas [Tue, 8 May 2018 18:11:33 +0000 (11:11 -0700)] 
DRILL-6389: Fixed building javadocs
 - Added documentation about how to build javadocs
 - Fixed some of the javadoc warnings

closes #1276

4 years agoDRILL-5700: Adding nohup support for Sqlline script
kr-arjun [Fri, 1 Jun 2018 07:09:03 +0000 (00:09 -0700)] 
DRILL-5700: Adding nohup support for Sqlline script

closes #1300

4 years agoDRILL-6432: Show Button to print visualized query plan (#1278)
Kunal Khatua [Wed, 6 Jun 2018 15:43:27 +0000 (08:43 -0700)] 
DRILL-6432: Show Button to print visualized query plan (#1278)

* DRILL-6432: Show Button to print visualized query plan

Having the capability to only print the visualized query plan means the user can specifyi the zoom level (in the printer settings) when printing large complex plans that might span multiple pages.

* Update based on review comment

Padded button by 15px to avoid sticking to pane

closes #1278

4 years agoRevert "DRILL-4020: The not-equal operator returns incorrect results when used on...
Parth Chandra [Sun, 3 Jun 2018 00:41:45 +0000 (17:41 -0700)] 
Revert "DRILL-4020: The not-equal operator returns incorrect results when used on the HBase row key"

This reverts commit 0d5eda83fe34928ff60629e6a4903d43a1d82582.

4 years agoDRILL-5584: Add branding and versioning information for windows C++ Client.
Rob Wu [Thu, 16 Nov 2017 02:21:48 +0000 (18:21 -0800)] 
DRILL-5584: Add branding and versioning information for windows C++ Client.

This closes #5584

4 years agoDRILL-5924: native-client: Support user-specified CXX_FLAGS
Korn, Uwe [Sat, 28 Oct 2017 19:45:51 +0000 (21:45 +0200)] 
DRILL-5924: native-client: Support user-specified CXX_FLAGS

This closes #1022

4 years agoDRILL-4020: The not-equal operator returns incorrect results when used on the HBase...
Akihiko Kusanagi [Wed, 4 Nov 2015 12:33:06 +0000 (21:33 +0900)] 
DRILL-4020: The not-equal operator returns incorrect results when used on the HBase row key

- Added a condition that checks if the filter to the scan specification doesn't have NOT_EQUAL operator
- Added testFilterPushDownRowKeyNotEqual() to TestHBaseFilterPushDown

This closes #309

4 years agoDRILL-6456: Planner shouldn't create any exchanges on the right side of Lateral Join.
Hanumath Rao Maduri [Thu, 31 May 2018 00:59:51 +0000 (17:59 -0700)] 
DRILL-6456: Planner shouldn't create any exchanges on the right side of Lateral Join.

This closes #1299

4 years agoDRILL-6450: Visualized plans for profiles querying JDBC sources is broken
Kunal Khatua [Tue, 29 May 2018 18:20:56 +0000 (11:20 -0700)] 
DRILL-6450: Visualized plans for profiles querying JDBC sources is broken

When viewing a profile for a query against a JDBC source, the visualized plan is not rendered. This is because the generated SQL pushed down to the JDBC source has a line break injected just before the FROM clause.

The workaround is to strip away any injected newlines ('\\n') at least for the SQL defined in the text plan, so that the backend Javascript can render it correctly.
In addition, any single line comments are also removed, but any block comments (i.e. /* .. */ ) are retained as they might carry hints.

This closes #1295

4 years agoDRILL-6445: Fix existing test cases in TestScripts.java and add new test case for...
Sorabh Hamirwasia [Thu, 24 May 2018 18:58:01 +0000 (11:58 -0700)] 
DRILL-6445: Fix existing test cases in TestScripts.java and add new test case for DRILLBIT_CONTEXT variable

This closes #1289

4 years agoDRILL-6236:Batch sizing for hash join
Padma Penumarthy [Wed, 30 May 2018 21:00:16 +0000 (14:00 -0700)] 
DRILL-6236:Batch sizing for hash join

This closes #1227

4 years agoDRILL-6343: bit vector copyFromSafe is not doing realloc
Padma Penumarthy [Thu, 19 Apr 2018 21:36:38 +0000 (14:36 -0700)] 
DRILL-6343: bit vector copyFromSafe is not doing realloc

4 years agoDRILL-4364: Image Metadata Format Plugin - Initial commit of Image Metadata Format...
Akihiko Kusanagi [Sat, 6 Feb 2016 18:55:57 +0000 (03:55 +0900)] 
DRILL-4364: Image Metadata Format Plugin - Initial commit of Image Metadata Format Plugin - See https://issues.apache.org/jira/browse/DRILL-4364

This closes #367

4 years agoDRILL-4091: Adding support for additional gis operations in gis contrib module
potocki [Mon, 16 Nov 2015 13:05:18 +0000 (14:05 +0100)] 
DRILL-4091: Adding support for additional gis operations in gis contrib module

This closes #1201

4 years agoDRILL-6356: batch sizing for union all
Padma Penumarthy [Thu, 10 May 2018 22:28:40 +0000 (15:28 -0700)] 
DRILL-6356: batch sizing for union all

closes #1255

4 years agoDRILL-6402: Repeated Value Vectors copyFrom methods are not updating
Padma Penumarthy [Thu, 10 May 2018 22:29:27 +0000 (15:29 -0700)] 
DRILL-6402: Repeated Value Vectors copyFrom methods are not updating
the value count and writer index correctly for values vector

4 years agoDRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out (#1281)
Venkata Jyothsna Donapati [Wed, 30 May 2018 18:03:58 +0000 (11:03 -0700)] 
DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out (#1281)

DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out

4 years agoDRILL-3964 : Fix NPE in WriterRecordBatch when 0 rows
Gautam Parai [Fri, 25 May 2018 02:00:21 +0000 (19:00 -0700)] 
DRILL-3964 : Fix NPE in WriterRecordBatch when 0 rows

closes #1290

4 years agoDRILL-6442: Adjust Hbase disk cost & row count estimation when filter push down is...
Arina Ielchiieva [Wed, 23 May 2018 16:17:11 +0000 (16:17 +0000)] 
DRILL-6442: Adjust Hbase disk cost & row count estimation when filter push down is applied

closes #1288

4 years agoDRILL-5990: Drill should not startup with JRE sourced java executable
Kunal Khatua [Mon, 21 May 2018 21:19:53 +0000 (14:19 -0700)] 
DRILL-5990: Drill should not startup with JRE sourced java executable

Currently, the check for the Java compiler is made only during query execution.
This commit ensures that if the java executable used for instantiating the Drillbit is from a JRE (i.e. the Java compiler is not available), the Drillbit will fail with the exception reported in the logs:
```
Exception in thread "main" org.apache.drill.exec.exception.DrillbitStartupException: JDK Java compiler not available. Ensure Drill is running with the java executable from a JDK and not a JRE
        at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:138)
        at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:409)
        at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:389)
        at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:385)
```

4 years agoDRILL-6443: Enable Search for both running AND completed queries
Kunal Khatua [Wed, 23 May 2018 22:55:24 +0000 (15:55 -0700)] 
DRILL-6443: Enable Search for both running AND completed queries

When running a query in Drill, the `/profiles` page will show the search (and pagination) capabilities only for the top most visible table (i.e. Running Queries ).

The Completed Queries table will show the search feature only when there are no running queries. This is because the backend uses a generalized freemarker macro to define the seach capabilities for the tables being rendered. With running queries, both, running and completed queries tables have the same element ID, resulting in the search capability only being applied to the first table.

This modifies the Freemarker macro to take an additional argument for distinguishing between Running and Completed Queries table.

4 years agoDRILL-6437: Removed excess maven logging when downloading dependencies. This fixed...
Timothy Farkas [Wed, 23 May 2018 21:14:15 +0000 (14:14 -0700)] 
DRILL-6437: Removed excess maven logging when downloading dependencies. This fixed Travis failures due to log overflow.

4 years agoDRILL-6435: MappingSet is stateful, so it can't be shared between threads
Vlad Rozov [Wed, 23 May 2018 16:14:13 +0000 (09:14 -0700)] 
DRILL-6435: MappingSet is stateful, so it can't be shared between threads

4 years agoDRILL-6436: Storage Plugin to have name and context moved to AbstractStoragePlugin
chunhui-shi [Thu, 15 Feb 2018 20:21:33 +0000 (12:21 -0800)] 
DRILL-6436: Storage Plugin to have name and context moved to AbstractStoragePlugin

closes #1282

4 years agoDRILL-6440: Unnest unit tests and fixes for stats
Parth Chandra [Mon, 16 Apr 2018 23:42:20 +0000 (16:42 -0700)] 
DRILL-6440: Unnest unit tests and fixes for stats

  - Add unit test with mock input, nested lateral and unnest and project.
  - Fix unit test involving batch limits, ignore map tests
  - Fix input row count stats.

closes #1283

4 years agoDRILL-6374: Transitive Closure leads to TPCH Queries regressions and OOM when run...
Vitalii Diravka [Mon, 14 May 2018 12:27:16 +0000 (15:27 +0300)] 
DRILL-6374: Transitive Closure leads to TPCH Queries regressions and OOM when run concurrency test

- Test case for Directory Pruning with Transitive Predicates
- Improving description for STRICT_EQUAL_IS_DISTINCT_FROM predicate

closes #1262

4 years agoDRILL-6431: Unnest operator requires table and a single column alias to be specified.
hmaduri [Tue, 15 May 2018 20:51:45 +0000 (13:51 -0700)] 
DRILL-6431: Unnest operator requires table and a single column alias to be specified.
Fixing the issues related to star column renaming, same field name renaming
and also enforcing that an alias column is required for the unnest operator.

4 years agoDRILL-6419: E2E Integration test for Lateral&Unnest
Sorabh Hamirwasia [Tue, 15 May 2018 21:30:05 +0000 (14:30 -0700)] 
DRILL-6419: E2E Integration test for Lateral&Unnest

4 years agoDRILL-5846: Improve parquet performance for Flat Data Types
Salim Achouche [Wed, 18 Apr 2018 03:12:03 +0000 (20:12 -0700)] 
DRILL-5846: Improve parquet performance for Flat Data Types

closes #1060

4 years agoDRILL-6423: Export query result as a CSV file
Kunal Khatua [Wed, 16 May 2018 17:35:51 +0000 (10:35 -0700)] 
DRILL-6423: Export query result as a CSV file

Added option for user specified delimiter
Also, we show query Id on results page as alink for popping out a new window

closes #1266

4 years agoDRILL-5305: Query Profile must display Query ID
Kunal Khatua [Wed, 16 May 2018 17:22:41 +0000 (10:22 -0700)] 
DRILL-5305: Query Profile must display Query ID

Introduced change to the Protobuf to inject the text-equivalent of the QueryID into the profile. This way, the profile's file name can be changed, but restored back based on this new field.
The Profile UI also shows the Query ID, though this is not inferred from this new field, for sake of backward compatibility with older profiles.

closes #1265

4 years agoDRILL-6418: Handle Schema change in Unnest And Lateral for unnest field / non-unnest...
Sorabh Hamirwasia [Tue, 15 May 2018 21:27:31 +0000 (14:27 -0700)] 
DRILL-6418: Handle Schema change in Unnest And Lateral for unnest field / non-unnest field

Note: Changed Lateral to handle non-empty right batch with OK_NEW_SCHEMA

closes #1271

4 years agoDRILL-6348: Received batches are now owned by the receive operators instead of the...
Salim Achouche [Mon, 23 Apr 2018 01:02:35 +0000 (18:02 -0700)] 
DRILL-6348: Received batches are now owned by the receive operators instead of the parent

closes #1237

4 years agoDRILL-4525: Code cleanup
Bohdan Kazydub [Thu, 17 May 2018 10:08:31 +0000 (13:08 +0300)] 
DRILL-4525: Code cleanup

- Fixed failing test

closes #1268

4 years agoDRILL-4525: Allow SqlBetweenOperator to accept LOWER_OPERAND and UPPER_OPERAND with...
Hsuan-Yi Chu [Mon, 21 Mar 2016 21:43:54 +0000 (14:43 -0700)] 
DRILL-4525: Allow SqlBetweenOperator to accept LOWER_OPERAND and UPPER_OPERAND with different types

4 years agoDRILL-6424: Updating FasterXML Jackson libraries
Vitalii Diravka [Thu, 17 May 2018 09:59:08 +0000 (12:59 +0300)] 
DRILL-6424: Updating FasterXML Jackson libraries

closes #1274

4 years agoDRILL-6413: Update ParquetFilterBuilder.visitBooleanOperator to handle simplified...
Arina Ielchiieva [Thu, 17 May 2018 15:24:35 +0000 (18:24 +0300)] 
DRILL-6413: Update ParquetFilterBuilder.visitBooleanOperator to handle simplified boolean expression

closes #1269

4 years agoDRILL-6426: Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit...
Arina Ielchiieva [Fri, 18 May 2018 10:11:05 +0000 (13:11 +0300)] 
DRILL-6426: Refactor TestFunctionsWithTypeExpoQueries test to be independent on limit 0 optimization option

closes #1273

4 years agoDRILL-6421: Refactor DecimalUtility and CoreDecimalUtility classes
Volodymyr Vysotskyi [Wed, 16 May 2018 18:34:56 +0000 (21:34 +0300)] 
DRILL-6421: Refactor DecimalUtility and CoreDecimalUtility classes

closes #1267

4 years agoDRILL-6255: Drillbit while sending control message to itself creates a connection...
Sorabh Hamirwasia [Tue, 8 May 2018 20:06:20 +0000 (13:06 -0700)] 
DRILL-6255: Drillbit while sending control message to itself creates a connection instead of submitting locally

closes #1253

4 years agoDRILL-6420: Add Lateral and Unnest Keyword for highlighting on WebUI
Sorabh Hamirwasia [Fri, 4 May 2018 17:47:28 +0000 (10:47 -0700)] 
DRILL-6420: Add Lateral and Unnest Keyword for highlighting on WebUI

closes #1261

4 years agoDRILL-6027:
Ben-Zvi [Sat, 28 Apr 2018 04:59:25 +0000 (21:59 -0700)] 
DRILL-6027:
  - Added fallback option for HashJoin.
  - No copy of incoming for single partition, and avoid HT resize.
  - Fix memory leak when cancelling while spill file is read
  - get correct schema when probe side is empty
  - Re-create the HashJoinProbe

4 years agoDRILL-6027:
Timothy Farkas [Tue, 13 Feb 2018 00:49:29 +0000 (16:49 -0800)] 
DRILL-6027:
 - Added memory claculator
 - Added unit tests and docs.
 - Fixed IOB caused by output vector allocation.
 - Don't double count records that were spilled in HashJoin

4 years agoDRILL-6027: Initial implementation of HashJoin spill, without memory limits checks yet
Ben-Zvi [Wed, 14 Feb 2018 20:36:39 +0000 (12:36 -0800)] 
DRILL-6027: Initial implementation of HashJoin spill, without memory limits checks yet

4 years agocode review comments
Padma Penumarthy [Wed, 16 May 2018 00:58:13 +0000 (17:58 -0700)] 
code review comments

4 years agoDRILL-6411: Make batch memory sizing logs uniform across all operators
Padma Penumarthy [Mon, 14 May 2018 22:01:25 +0000 (15:01 -0700)] 
DRILL-6411: Make batch memory sizing logs uniform across all operators

4 years agoDRILL-6321: Lateral Join and Unnest - rules, options, logical plan supports
chunhui-shi [Mon, 30 Apr 2018 17:24:06 +0000 (10:24 -0700)] 
DRILL-6321: Lateral Join and Unnest - rules, options, logical plan supports

Included changes:
* Add planner.enable_unnest_lateral option. Default value set to false.
* Enable FilterCorrectRule
* Add support to logical plan
* Fix rebase errors for DRILL-6321 commits

4 years agoDRILL-6321: Lateral Join and Unnest - initial implementation for parser and planning
chunhui-shi [Mon, 26 Mar 2018 19:15:09 +0000 (12:15 -0700)] 
DRILL-6321: Lateral Join and Unnest - initial implementation for parser and planning

4 years agoDRILL-6272: Refactor dynamic UDFs and function initializer tests to generate needed...
Arina Ielchiieva [Sat, 5 May 2018 12:32:07 +0000 (15:32 +0300)] 
DRILL-6272: Refactor dynamic UDFs and function initializer tests to generate needed binary and source jars at runtime

close apache/drill#1225

4 years agoDRILL-6363: Upgrade jmockit and mockito libs
Arina Ielchiieva [Sat, 5 May 2018 12:31:10 +0000 (15:31 +0300)] 
DRILL-6363: Upgrade jmockit and mockito libs

4 years agoDRILL-6242 Use java.time.Local{Date|Time|DateTime} for Drill Date, Time, Timestamp...
jiang-wu [Tue, 1 May 2018 21:48:26 +0000 (14:48 -0700)] 
DRILL-6242 Use java.time.Local{Date|Time|DateTime} for Drill Date, Time, Timestamp types. (#3)

close apache/drill#1247

* DRILL-6242 - Use java.time.Local{Date|Time|DateTime} classes to hold values from corresponding Drill date, time, and timestamp types.
Conflicts:
exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/ExtendedJsonOutput.java

Fix merge conflicts and check style.

4 years agoDRILL-6401: Precision for decimal data types may be lost for the case when cast with...
Volodymyr Vysotskyi [Thu, 10 May 2018 14:06:28 +0000 (17:06 +0300)] 
DRILL-6401: Precision for decimal data types may be lost for the case when cast with literal is used

close apache/drill#1254

4 years agoDRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart
Kunal Khatua [Fri, 27 Apr 2018 23:27:45 +0000 (16:27 -0700)] 
DRILL-6364: Handle Cluster Info in WebUI when existing/new bits restart

As a follow up to DRILL-6289, the following improvements have been done:
1. When loading the page for the first time, the WebUI enables the shutdown button without actually checking the state of the Drillbits.
   The ideal behaviour should be to disable the button till the state is verified. [Done]
   If a Drillbit is confirmed down (i.e. not in `/state` response), it is marked as OFFLINE and button is disabled.
2. When shutting down the current Drillbit, the WebUI no more has access to the cluster state.
   The ideal behaviour here should be to fetch the state from any of the other Drillbits to update the status. [Done]
   With the current Drillbit down, the other bits are requested for cluster state info and update accordingly.
3. When a new, previously unseen Drillbit comes up, the WebUI will never render it because the table is statically generated during the first page load.
   The idea behaviour should be to append to the table on discovery of a new node. [Done]
   The new Drillbit info is injected and a prompt appears to refresh the page to re-populate any missing info. This also works with feature (2) mentioned above.

The only Java code change was to have the state response carry the address and http-port as a tuple, instead of the user-port (which is never used).

Updates based on review

1. Added descriptive comments to functions
2. Handled possible race condition from multiple httpRequests for cluster state
3. Eliminated unused stateKey variable
4. Best practice of using local (let) instead of global (var) objects, and substituting currentRow variable instead of the jQuery search.

Additional changes based on review

1. Random selection of Drillbits to query for state when primary Drillbit is down (limited to 3; with a timeout of 3 sec)
2. Indicate when an Offline Drillbit is de-registered, versus just Offline as per Zookeeper
3. Hide shutdown column when Authentication is enabled, but the user is NOT an Admin.
   When the column is visible, remote bits are disabled because
4. Metrics will be shown all the time (except HTTPS), because the information is available and a non-admin user would anyway not have actionable capabilities
5. Basic clean up on unused variables

Hide shutdown buttons for HTTPS scenarios

Fixed check for 'https'

Since, `location.protocol` returns a trailing ':' as well

Handle metrics lookup on local Drillbit for HTTPS

If the webpage is accessed via IP, the certificate is tied to the IP and not the FQDN, which is what is used for fetching the metrics.

close apache/drill#1241

4 years agoDRILL-6321: Customize Drill's conformance. Allow support to APPLY keywords
chunhui-shi [Fri, 2 Feb 2018 18:03:38 +0000 (10:03 -0800)] 
DRILL-6321: Customize Drill's conformance. Allow support to APPLY keywords

close apache/drill#1224

Conflicts:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/SqlConverter.java

Fix checkstyle

4 years agoDRILL-6249: Adding more unit testing documentation.
Timothy Farkas [Thu, 15 Mar 2018 21:27:55 +0000 (14:27 -0700)] 
DRILL-6249: Adding more unit testing documentation.

close apache/drill#1251

4 years agoDRILL-6386: Remove unused imports and star imports.
Drill Dev [Tue, 8 May 2018 20:57:37 +0000 (13:57 -0700)] 
DRILL-6386: Remove unused imports and star imports.

4 years agoDRILL-6386: Disallow unused imports and star imports.
Timothy Farkas [Tue, 8 May 2018 20:55:40 +0000 (13:55 -0700)] 
DRILL-6386: Disallow unused imports and star imports.

4 years agoDRILL-143: CGroup Support for Drill-on-YARN 440/head
Kunal Khatua [Mon, 23 Apr 2018 17:31:21 +0000 (10:31 -0700)] 
DRILL-143: CGroup Support for Drill-on-YARN

Initial patch
1. Minor fix up
2. Updated variable name for pid file
Revert changes to yarn-drillbit.sh
Based on discussion for PR #1239 (Ref: https://github.com/apache/drill/pull/1239)

closes #1239

4 years agoDRILL-6361: Revised typeOf() function versions
Paul Rogers [Sun, 29 Apr 2018 06:04:26 +0000 (23:04 -0700)] 
DRILL-6361: Revised typeOf() function versions

Added more unit tests.

Updated to handle VARDECIMAL
The VARDECIMAL type was recently added to Drill. Added support for this type. The sqlTypeOf() function now returns DECIMAL(p, s) for precision p, scale s.

closes #1242

4 years agoDRILL-6380: Fix sporadic mongo db hangs.
Timothy Farkas [Wed, 2 May 2018 08:21:10 +0000 (01:21 -0700)] 
DRILL-6380: Fix sporadic mongo db hangs.

closes #1249

4 years agoDRILL-6281: Introduce Collectors class for internal iterators
Vlad Rozov [Mon, 30 Apr 2018 14:11:20 +0000 (07:11 -0700)] 
DRILL-6281: Introduce Collectors class for internal iterators

closes #1238

4 years agoDRILL-6281: Refactor TimedRunnable
Vlad Rozov [Sat, 21 Apr 2018 15:07:42 +0000 (08:07 -0700)] 
DRILL-6281: Refactor TimedRunnable

4 years agoDRILL-6281: Refactor TimedRunnable (rename TimedRunnable to TimedCallable)
Vlad Rozov [Sat, 21 Apr 2018 01:00:20 +0000 (18:00 -0700)] 
DRILL-6281: Refactor TimedRunnable (rename TimedRunnable to TimedCallable)

4 years agoDRILL-6347: Change method names to "visitField".
Kui LIU [Mon, 30 Apr 2018 06:43:00 +0000 (08:43 +0200)] 
DRILL-6347: Change method names to "visitField".

Further change the method names to "visitField" with Vlad Rozov's comments.

closes #1236

4 years agoDRILL-6318: Push down limit past flatten is incorrect
Oleg [Tue, 10 Apr 2018 08:26:25 +0000 (11:26 +0300)] 
DRILL-6318: Push down limit past flatten is incorrect

closes #1204

4 years agoDRILL-6094: Decimal data type enhancements
Volodymyr Vysotskyi [Thu, 5 Apr 2018 12:35:42 +0000 (15:35 +0300)] 
DRILL-6094: Decimal data type enhancements

Add ExprVisitors for VARDECIMAL
Modify writers/readers to support VARDECIMAL
- Added usage of VarDecimal for parquet, hive, maprdb, jdbc;
- Added options to store decimals as int32 and int64 or fixed_len_byte_array or binary;
Add UDFs for VARDECIMAL data type
- modify type inference rules
- remove UDFs for obsolete DECIMAL types
Enable DECIMAL data type by default
Add unit tests for DECIMAL data type
Fix mapping for NLJ when literal with non-primitive type is used in join conditions
Refresh protobuf C++ source files
Changes in C++ files
Add support for decimal logical type in Avro.
Add support for date, time and timestamp logical types.
Update Avro version to 1.8.2.

4 years agoDRILL-4184: Support variable length decimal fields in parquet
Dave Oshinsky [Tue, 9 Feb 2016 22:37:47 +0000 (17:37 -0500)] 
DRILL-4184: Support variable length decimal fields in parquet

4 years agoDRILL-6341: Fixed failing tests for mongodb storage plugin by upgrading MongoDB version.
Timothy Farkas [Tue, 17 Apr 2018 20:29:14 +0000 (13:29 -0700)] 
DRILL-6341: Fixed failing tests for mongodb storage plugin by upgrading MongoDB version.

closes #1222

4 years agoDRILL-6327: Update unary operators to handle IterOutcome.EMIT
Sorabh Hamirwasia [Thu, 5 Apr 2018 00:54:58 +0000 (17:54 -0700)] 
DRILL-6327: Update unary operators to handle IterOutcome.EMIT
            Note: Handles for Non-Blocking Unary operators (like Filter/Project/etc) with EMIT Iter.Outcome

closes #1240

4 years agoDRILL-6307: Handle empty batches in record batch sizer correctly
Padma Penumarthy [Fri, 6 Apr 2018 19:56:06 +0000 (12:56 -0700)] 
DRILL-6307: Handle empty batches in record batch sizer correctly

closes #1228

4 years agoDRILL-6202: Deprecate usage of IndexOutOfBoundsException to re-alloc vectors
Vlad Rozov [Thu, 1 Mar 2018 17:36:05 +0000 (09:36 -0800)] 
DRILL-6202: Deprecate usage of IndexOutOfBoundsException to re-alloc vectors

closes #1144

4 years agoDRILL-6345: DRILL Query fails on Function LOG10
vladimir tkach [Fri, 27 Apr 2018 10:24:56 +0000 (13:24 +0300)] 
DRILL-6345: DRILL Query fails on Function LOG10

 - Added log10 function implementation

closes #1230

4 years agoDRILL-6335: Column accessor refactoring
Paul Rogers [Tue, 17 Apr 2018 04:44:10 +0000 (21:44 -0700)] 
DRILL-6335: Column accessor refactoring

closes #1218

4 years agoDRILL-6328: Adding unit testing docs.
Paul Rogers [Wed, 18 Apr 2018 00:43:02 +0000 (17:43 -0700)] 
DRILL-6328: Adding unit testing docs.

closes #1220

4 years agoDRILL-6328: Consolidated developer docs in the docs folder.
Timothy Farkas [Wed, 18 Apr 2018 00:36:57 +0000 (17:36 -0700)] 
DRILL-6328: Consolidated developer docs in the docs folder.

closes #1220

4 years agoDRILL-6302: Fixed NPE in Drillbit close method
dvjyothsna [Mon, 16 Apr 2018 18:45:30 +0000 (11:45 -0700)] 
DRILL-6302: Fixed NPE in Drillbit close method

closes #1217

4 years agoDRILL-6286: Fixed incorrect reference to shutdown in drillbit.log
dvjyothsna [Fri, 30 Mar 2018 00:11:49 +0000 (17:11 -0700)] 
DRILL-6286: Fixed incorrect reference to shutdown in drillbit.log

closes #1196

4 years agoDRILL-6282: Update Drill's Metrics dependencies
Vitalii Diravka [Wed, 21 Mar 2018 14:16:10 +0000 (16:16 +0200)] 
DRILL-6282: Update Drill's Metrics dependencies

- Replacing com.codahale.metrics with last io.dropwizard.metrics Metrics for Drill
- com.yammer.metrics is removed, since isn't used directly by Drill

closes #1189

4 years agoDRILL-6270: Added general option --jvm, which can be used to set the remote debug...
Anton Gozhiy [Tue, 27 Mar 2018 10:29:14 +0000 (13:29 +0300)] 
DRILL-6270: Added general option --jvm, which can be used to set the remote debug jvm options.

The option is not needed for drillbit startup, you can set the jvm properties directly after the start command.
Usage:
bin/drill-embedded --jvm "[JVM options]"
bin/sqlline -u "jdbc:drill:zk=zkaddr:port" --jvm "[JVM options]"
bin/drillbit.sh start [JVM options]
sqlline.bat -u "jdbc:drill:zk=local" --jvm "[JVM options]"

closes #1210

4 years agoDRILL-6173: Support transitive closure during filter push down and partition pruning
Vitalii Diravka [Tue, 17 Apr 2018 11:38:03 +0000 (14:38 +0300)] 
DRILL-6173: Support transitive closure during filter push down and partition pruning

closes #1216

4 years agoDRILL-5927: Fixed memory leak in TestBsonRecordReader, and sped up the test.
Timothy Farkas [Fri, 20 Apr 2018 23:52:59 +0000 (16:52 -0700)] 
DRILL-5927: Fixed memory leak in TestBsonRecordReader, and sped up the test.

closes #1234

4 years agoDRILL-3855: Enable FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule
Vitalii Diravka [Thu, 19 Apr 2018 11:31:17 +0000 (14:31 +0300)] 
DRILL-3855: Enable FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule

closes #1226

4 years agoDRILL-6331: Revisit Hive Drill native parquet implementation to be exposed to Drill...
Arina Ielchiieva [Tue, 20 Mar 2018 18:29:45 +0000 (18:29 +0000)] 
DRILL-6331: Revisit Hive Drill native parquet implementation to be exposed to Drill optimizations (filter / limit push down, count to direct scan)

1. Factored out common logic for Drill parquet reader and Hive Drill native parquet readers: AbstractParquetGroupScan, AbstractParquetRowGroupScan, AbstractParquetScanBatchCreator.
2. Rules that worked previously only with ParquetGroupScan, now can be applied for any class that extends AbstractParquetGroupScan: DrillFilterItemStarReWriterRule, ParquetPruneScanRule, PruneScanRule.
3. Hive populated partition values based on information returned from Hive metastore. Drill populates partition values based on path difference between selection root and actual file path.
   Before ColumnExplorer populated partition values based on Drill approach. Since now ColumnExplorer populates values for parquet files from Hive tables,
   `populateImplicitColumns` method logic was changed to populated partition columns only based on given partition values.
4. Refactored ParquetPartitionDescriptor to be responsible for populating partition values rather than storing this logic in parquet group scan class.
5. Metadata class was moved to separate metadata package (org.apache.drill.exec.store.parquet.metadata). Factored out several inner classed to improve code readability.
6. Collected all Drill native parquet reader unit tests into one class TestHiveDrillNativeParquetReader, also added new tests to cover new functionality.
7. Reduced excessive logging when parquet files metadata is read

closes #1214

4 years agoDRILL-6342: Fix schema path unIndexed method to return correct value
Arina Ielchiieva [Fri, 20 Apr 2018 11:38:14 +0000 (14:38 +0300)] 
DRILL-6342: Fix schema path unIndexed method to return correct value

closes #1231

4 years agoDRILL-6324: Unnest - Add tests with real Unnest and real Lateral.
Parth Chandra [Wed, 7 Mar 2018 08:23:14 +0000 (13:53 +0530)] 
DRILL-6324: Unnest - Add tests with real Unnest and real Lateral.

Code cleanup, more comments, fix license headers, and more logging.
Refactor Unnest to allow setting in incoming batch after construction
fix compilation after rebase

This closes #1223

4 years agoDRILL-6324: Unnest - Initial Implementation
Parth Chandra [Thu, 8 Feb 2018 05:13:13 +0000 (10:43 +0530)] 
DRILL-6324: Unnest - Initial Implementation

- Based on Flatten
- Implement unnestRecords in UnnestTemplate
- Remove unnecessary code inherited from Flatten/Project. Add schema change handling.
- Fix build failure after rebase since RecordBatchSizer used by UNNEST was relocated to a different package
- Add unit tests
- Handling of input row splitting across multiple batches. Also do not kill incoming in killIncoming.
- Schema change generated by Unnest

4 years agoDRILL-6339: Add a new option to disable TopN (for testing)
Ben-Zvi [Wed, 18 Apr 2018 00:05:50 +0000 (17:05 -0700)] 
DRILL-6339: Add a new option to disable TopN (for testing)

This closes #1219

4 years agoDRILL-6338: Do not skip license maven plugin when formatting generated protobuf files.
Timothy Farkas [Tue, 17 Apr 2018 18:57:40 +0000 (11:57 -0700)] 
DRILL-6338: Do not skip license maven plugin when formatting generated protobuf files.

This closes #1215

4 years agoDRILL-6334: Minor code cleanup
Paul Rogers [Tue, 17 Apr 2018 01:24:07 +0000 (18:24 -0700)] 
DRILL-6334: Minor code cleanup

This closes #1213

4 years agoDRILL-6333: Fixed Quotation marks
Kunal Khatua [Fri, 20 Apr 2018 18:01:51 +0000 (11:01 -0700)] 
DRILL-6333: Fixed Quotation marks

Initial step to making the source-code ready for Javadoc generation

This closes #1229

4 years agoDRILL-6289: Cluster view should show more relevant information
Kunal Khatua [Mon, 26 Mar 2018 05:03:12 +0000 (22:03 -0700)] 
DRILL-6289: Cluster view should show more relevant information

Protobuf change to carry HTTP port info
Allow CORS for access to remote Drillbit metrics
Cross-origin resource sharing (CORS) is required to ensure that the WebServer is able serve REST calls for status pages.
Materialize relevant metrics
  1. Heap memory (incl usage)
  2. Heap memory (incl usage)
  3. Average System Load (last 1 min)
  4. Option to view from other nodes (pop out)
  5. Added Glyphicons
Update System Table and related tests
  1. Updated System Table to show HTTP port
  2. Updated unit tests
Skip updating remote bit info when HTTPS (SSL) or Authentication is enabled.
Default CpuGaugeSet is public; Added Gauges
  * CPU Utiization by Drill
  * Uptime
Show ALL Buttons, but do HTTPS Check
Reduce power button to icon
Allowing CORS for /status/metrics only
Accounting for situations when JVM does not report Process CPU Load
  i.e. returned value is negative.
  See https://docs.oracle.com/javase/7/docs/jre/api/management/extension/com/sun/management/OperatingSystemMXBean.html#getProcessCpuLoad()
Addressed shutdown security conditions
Added C++ Client Protobuf
Added steps for Protobuf generation to protocol/readme.txt

This closes #1203