drill.git
2 days agoDRILL-6895: Warnings for license headers checks and wrong license headers in some... master
Vitalii Diravka [Tue, 11 Dec 2018 12:30:47 +0000 (14:30 +0200)] 
DRILL-6895: Warnings for license headers checks and wrong license headers in some files

- update license-maven-plugin plugin custom mappings
- update files licenses headers
- update from "/usr/bin/env bash" to "/bin/bash" shebang line in shell script files

2 days agoDRILL-6878: Use DrillPushRowKeyJoinToScan rule on DrillJoin pattern to account for...
Gautam Parai [Tue, 4 Dec 2018 02:03:23 +0000 (18:03 -0800)] 
DRILL-6878: Use DrillPushRowKeyJoinToScan rule on DrillJoin pattern to account for DrillSemiJoin

closes #1568

4 days agoDRILL-6889: Exclude Drill unit tests to avoid Travis timing out
Vitalii Diravka [Fri, 7 Dec 2018 17:44:47 +0000 (19:44 +0200)] 
DRILL-6889: Exclude Drill unit tests to avoid Travis timing out

closes #1567

4 days agoDRILL-6877: NPE when starting Drillbit
Venkata Jyothsna Donapati [Mon, 3 Dec 2018 18:19:06 +0000 (10:19 -0800)] 
DRILL-6877: NPE when starting Drillbit

closes #1560

4 days agoDRILL-6791: Scan projection framework
Paul Rogers [Fri, 12 Oct 2018 05:47:49 +0000 (22:47 -0700)] 
DRILL-6791: Scan projection framework

The "schema projection" mechanism:

* Handles none (SELECT COUNT\(*)), some (SELECT a, b, x) and all (SELECT *) projection.
* Handles null columns (for projection a column "x" that does not exist in the base table.)
* Handles constant columns as used for file metadata (AKA "implicit" columns).
* Handle schema persistence: the need to reuse the same vectors across different scanners
* Provides a framework for consuming externally-supplied metadata
* Since we don't yet have a way to provide "real" metadata, obtains metadata hints from
  previous batches and from the projection list (a.b implies that "a" is a map, c[0]
  implies that "c" is an array, etc.)
* Handles merging the set of data source columns and null columns to create the final output batch.
* Running tests found a failure due to an uninialized "bits" vector. Added code to explicitly fill
  the bits vectors with zeros in the "result set loader."

5 days agoDRILL-6886: Cancel Query Button for running queries
Kunal Khatua [Mon, 10 Dec 2018 06:40:19 +0000 (22:40 -0800)] 
DRILL-6886: Cancel Query Button for running queries

The Freemarker template missed a check for a query's Running state to display the [Cancel] button.

The fix is trivial and only needs the check.

6 days agoDRILL-6887: Fix FunctionInitializerTest.init() failure
Vitalii Diravka [Fri, 7 Dec 2018 13:55:48 +0000 (15:55 +0200)] 
DRILL-6887: Fix FunctionInitializerTest.init() failure

close apache/drill#1566

6 days agoDRILL-6874: Close input stream after AsyncPageReaderTask is completed
Bohdan Kazydub [Thu, 6 Dec 2018 13:03:25 +0000 (15:03 +0200)] 
DRILL-6874: Close input stream after AsyncPageReaderTask is completed

close apache/drill#1565

6 days agoDRILL-6883: Force reload of options after resetting parameter
Kunal Khatua [Thu, 6 Dec 2018 07:34:26 +0000 (23:34 -0800)] 
DRILL-6883: Force reload of options after resetting parameter

If an update is initiated, the webpage loaded changes. Attempting a reset tries to reload this page, but it does not show the updated values from the back end.
This patch forces the reload of updated values by redirecting to /options page

Use AJAX call for Update and Reset

Eliminates the need for window.location=<redirectURL>

is Number check

Added comments to explain extraction

close apache/drill#1563

8 days agoDRILL-6751: Upgrade Apache parent POM to version 21
Vitalii Diravka [Fri, 16 Nov 2018 01:40:21 +0000 (17:40 -0800)] 
DRILL-6751: Upgrade Apache parent POM to version 21

- Update apache.pom file version to 21 (with updating some maven plugins versions)
- Include Drill's sources jars on assembly stage in <moduleSets> (not <dependencySets>)
  for properincluding jars with last apache-21.pom
- Separate "distro-assembly" to the two execution stages to avoid:
  [WARNING] Assembly file: <DRILL_HOME>/distribution/target/apache-drill-1.15.0-SNAPSHOT is not a regular
  file (it may be a directory). It cannot be attached to the project build for installation or deployment.
- Remove unsused <include>/<exclude> in assebly descriptor to avoid:
  [WARNING] The following patterns were never triggered in this artifact inclusion filter
- Update "maven-assembly-plugin" version
- Update "slf4j" version
- Update "mockito-core" version
- Update "bcpkix-jdk15on" (Bouncy Castle Cryptography APIs) version

close apache/drill#1561

8 days agoDRILL-4546: Only generate one zip archive when using apache-release profile
Laurent Goujon [Mon, 28 Mar 2016 20:31:53 +0000 (13:31 -0700)] 
DRILL-4546: Only generate one zip archive when using apache-release profile

Drill root pom doesn't override completely Apache parent pom configuration
regarding assemblies, which caused a zip archive of the project to be generated
twice, and deployed to a remote server twice too.

The fix uses the plugin properties to override the configuration. Also remove
Drill source assembly descriptor as the Apache parent project provides the same one.

8 days agoDRILL-6876: Enable CircleCI builds for JDK 9-11
Volodymyr Vysotskyi [Fri, 16 Nov 2018 13:43:27 +0000 (15:43 +0200)] 
DRILL-6876: Enable CircleCI builds for JDK 9-11

close apache/drill#1558

8 days agoDRILL-6882: Handle the cases where RowKeyJoin's left pipeline being called multiple...
Hanumath Rao Maduri [Sat, 30 Sep 2017 20:26:40 +0000 (13:26 -0700)] 
DRILL-6882: Handle the cases where RowKeyJoin's left pipeline being called multiple times.

close apache/drill#1562

9 days agoAdd Vitalii's GPG key
Vitalii Diravka [Wed, 5 Dec 2018 13:54:01 +0000 (15:54 +0200)] 
Add Vitalii's GPG key

2 weeks agoDRILL-6866: Upgrade to SqlLine 1.6.0
Arina Ielchiieva [Thu, 22 Nov 2018 15:27:15 +0000 (15:27 +0000)] 
DRILL-6866: Upgrade to SqlLine 1.6.0

1. Changed SqlLine version to 1.6.0.
2. Overridden new getVersion method in DrillSqlLineApplication.
3. Set maxColumnWidth to 80 to avoid issue described in DRILL-6769.
4. Changed colorScheme to obsidian.
5. Output null value for varchar / char / boolean types as null instead of empty string.
6. Changed access modifier from package default to public for JDBC classes that implement external interfaces to avoid issues when calling methods from these classes using reflection.

closes #1556

2 weeks agoDRILL-6867: WebUI Query editor cursor position
ozinoviev [Fri, 23 Nov 2018 09:34:36 +0000 (12:34 +0300)] 
DRILL-6867: WebUI Query editor cursor position

closes #1551

2 weeks agoDRILL-6863: Drop table is not working if path within workspace starts with "/"
Bohdan Kazydub [Wed, 28 Nov 2018 14:51:18 +0000 (16:51 +0200)] 
DRILL-6863: Drop table is not working if path within workspace starts with "/"

- Made workspace to be honored when table/view name starts with "/" for DROP TABLE, DROP VIEW, CREATE VIEW and SELECT from view queries;
- Made "/{name}" and "{name}" to be equivalent names (the leading "/" is removed) when creating temporary tables so that SELECT ... FROM "/{name}" ... and SELECT ... FROM "{name}" ... produce the same results and behave as regular tables in the context.

closes #1557

2 weeks agoDRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf reference count...
weijie.tong [Sun, 14 Oct 2018 11:41:51 +0000 (19:41 +0800)] 
DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf reference count bugs & tune the execution flow & support left deep tree

closes #1504

2 weeks agoDRILL-6039: Fixed drillbit.sh script to do graceful shutdown
dvjyothsna [Fri, 9 Nov 2018 21:56:39 +0000 (13:56 -0800)] 
DRILL-6039: Fixed drillbit.sh script to do graceful shutdown

closes #1536

2 weeks agoDRILL-6864: Upgrade the git-commit-id plugin to 2.2.5
Ben-Zvi [Tue, 27 Nov 2018 03:06:52 +0000 (19:06 -0800)] 
DRILL-6864: Upgrade the git-commit-id plugin to 2.2.5

2 weeks agoDRILL-6834: Introduce option to disable result set for DDL queries for JDBC connection
Bohdan Kazydub [Wed, 21 Nov 2018 01:56:49 +0000 (03:56 +0200)] 
DRILL-6834: Introduce option to disable result set for DDL queries for JDBC connection

- Added session-scoped option `drill.exec.fetch_resultset_for_ddl` to control whether update count or result set should be returned for JDBC connection session. By default the option is set to `true` which ensures that result set is returned;
- Updated Drill JDBC: `DrillCursor` and `DrillStatement` to achieve desired behaviour.

closes #1549

2 weeks agoDRILL-6865: Filter is not removed from the plan when parquet table fully matches...
Volodymyr Vysotskyi [Fri, 23 Nov 2018 09:34:13 +0000 (11:34 +0200)] 
DRILL-6865: Filter is not removed from the plan when parquet table fully matches the filter

closes #1552

2 weeks agoDRILL-6865: Query returns wrong result when filter pruning happens
Volodymyr Vysotskyi [Wed, 21 Nov 2018 12:08:17 +0000 (14:08 +0200)] 
DRILL-6865: Query returns wrong result when filter pruning happens

2 weeks agoDRILL-6870: Upgrade to ANTLR4
Arina Ielchiieva [Tue, 20 Nov 2018 12:27:55 +0000 (14:27 +0200)] 
DRILL-6870: Upgrade to ANTLR4

closes #1554

2 weeks agoDRILL-6868: Upgrade Janino compiler to 3.0.11
Volodymyr Vysotskyi [Fri, 19 Oct 2018 15:04:12 +0000 (18:04 +0300)] 
DRILL-6868: Upgrade Janino compiler to 3.0.11

- Remove workaround where removing adjacent ALOAD-POP instruction pairs
- Remove ModifiedUnparser and use DeepCopier for modifying methods instead of modifying it with custom Unparser implementation

closes #1553

2 weeks agoDRILL-6858: Add functionality to list directories / files with exceptions suppression
Arina Ielchiieva [Tue, 20 Nov 2018 15:06:07 +0000 (17:06 +0200)] 
DRILL-6858: Add functionality to list directories / files with exceptions suppression

1. Add listDirectoriesSafe, listFilesSafe, listAllSafe in FileSystemUtil and DrillFileSystemUtil classes.
2. Use FileSystemUtil.listAllSafe during listing files in show files command and information_schema.files table.

closes #1547

2 weeks agoDRILL-6857: Read only required row groups in a file when limit push down is applied
Arina Ielchiieva [Tue, 20 Nov 2018 18:08:38 +0000 (20:08 +0200)] 
DRILL-6857: Read only required row groups in a file when limit push down is applied

closes #1548

2 weeks agoDRILL-6850: Allow configuring table names case sensitivity for JDBC storage plugin
Volodymyr Vysotskyi [Thu, 15 Nov 2018 11:07:44 +0000 (13:07 +0200)] 
DRILL-6850: Allow configuring table names case sensitivity for JDBC storage plugin

closes #1542

2 weeks agoDRILL-6850: Force setting DRILL_LOGICAL Convention for DrillRelFactories and DrillFil...
Volodymyr Vysotskyi [Tue, 13 Nov 2018 13:21:39 +0000 (15:21 +0200)] 
DRILL-6850: Force setting DRILL_LOGICAL Convention for DrillRelFactories and DrillFilterRel

- Fix workspace case insensitivity for JDBC storage plugin

2 weeks agoDRILL-6850: JDBC integration tests failures
Vitalii Diravka [Fri, 2 Nov 2018 13:41:37 +0000 (15:41 +0200)] 
DRILL-6850: JDBC integration tests failures

- Fix RDBMS integration tests (expected decimal output and testCrossSourceMultiFragmentJoin)
- Update libraries versions
- Resolve NPE for empty result

2 weeks agoDRILL-6847: Add Query Metadata to RESTful Interface
Charles S. Givre [Wed, 21 Nov 2018 13:15:00 +0000 (08:15 -0500)] 
DRILL-6847: Add Query Metadata to RESTful Interface

closes #1539

2 weeks agoDRILL-6818: Add descriptions to secondary index options.
Aman Sinha [Mon, 19 Nov 2018 19:25:59 +0000 (11:25 -0800)] 
DRILL-6818: Add descriptions to secondary index options.

closes #1545

2 weeks agoDRILL-6691: Unify checkstyle-config.xml files.
Hanumath Rao Maduri [Wed, 31 Oct 2018 21:39:25 +0000 (14:39 -0700)] 
DRILL-6691: Unify checkstyle-config.xml files.

closes #1550

2 weeks agoDRILL-6668: In Web UI, highlight options that are not default values
Kunal Khatua [Thu, 15 Nov 2018 06:28:48 +0000 (22:28 -0800)] 
DRILL-6668: In Web UI, highlight options that are not default values

This commit introduces a new button on the options page that allows a user to reset an option to its system default value.
To simplify things, a tooltip is shown when the mouse hovers over the button. If the option value is already default, the button is disabled.

Currently, the Update button redirects to /option/optionName . This change reuses what we already are using to set the default (using AJAX) and auto-refreshing
Switch [Default] label to [Reset]
Patch To Pass StatusResourcesTest

closes #1543

2 weeks agoDRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM
ozinoviev [Mon, 27 Aug 2018 12:48:52 +0000 (15:48 +0300)] 
DRILL-6349: Drill JDBC driver fails on Java 1.9+ with NoClassDefFoundError: sun/misc/VM

closes #1446

3 weeks agoDRILL-6861: Hash-Join should not exit after an empty probe-side spilled partition
Ben-Zvi [Tue, 20 Nov 2018 02:55:45 +0000 (18:55 -0800)] 
DRILL-6861: Hash-Join should not exit after an empty probe-side spilled partition

4 weeks agoDRILL-6844: Query with ORDER BY DESC on indexed column does not pick secondary index.
Hanumath Rao Maduri [Fri, 19 Oct 2018 02:35:55 +0000 (19:35 -0700)] 
DRILL-6844: Query with ORDER BY DESC on indexed column does not pick secondary index.

4 weeks agoDRILL-6853: Make the complex parquet reader batch max row size configurable
Salim Achouche [Thu, 15 Nov 2018 22:40:48 +0000 (14:40 -0800)] 
DRILL-6853: Make the complex parquet reader batch max row size configurable

4 weeks agoDRILL-6833: Support for pushdown of rowkey based joins
Gautam Parai [Wed, 3 Jan 2018 19:06:41 +0000 (11:06 -0800)] 
DRILL-6833: Support for pushdown of rowkey based joins

closes #1532

4 weeks agoDRILL-6770: JsonTableGroupScan should use new MapRDB 6.1.0 APIs
Gautam Parai [Wed, 3 Oct 2018 23:21:04 +0000 (16:21 -0700)] 
DRILL-6770: JsonTableGroupScan should use new MapRDB 6.1.0 APIs

closes #1489

4 weeks agoDRILL-6744: Support varchar and decimal push down
Arina Ielchiieva [Wed, 31 Oct 2018 18:24:03 +0000 (20:24 +0200)] 
DRILL-6744: Support varchar and decimal push down

1. Added enableStringsSignedMinMax parquet format plugin config and store.parquet.reader.strings_signed_min_max session option to control reading binary statistics for files generated by prior versions of Parquet 1.10.0.
2. Added ParquetReaderConfig to store configuration needed during reading parquet statistics or files.
3. Provided mechanism to enable varchar / decimal filter push down.
4. Added VersionUtil to compare Drill versions in string representation.
5. Added appropriate unit tests.

closes #1537

4 weeks agoDRILL-3933: Surround $QUERY variable in double-quotes to avoid asterisk expansion...
Arina Ielchiieva [Tue, 13 Nov 2018 17:53:07 +0000 (19:53 +0200)] 
DRILL-3933: Surround $QUERY variable in double-quotes to avoid asterisk expansion in sqlline script

closes #1538

4 weeks agoDRILL-5671: Set secure ACLs (Access Control List) for Drill ZK nodes in a secure...
karthik [Wed, 11 Jul 2018 21:01:04 +0000 (14:01 -0700)] 
DRILL-5671: Set secure ACLs (Access Control List) for Drill ZK nodes in a secure cluster

closes #1467

4 weeks agoDRILL-6843: Update SchemaBuilder comment to match implementation
Paul Rogers [Sun, 11 Nov 2018 00:15:33 +0000 (16:15 -0800)] 
DRILL-6843: Update SchemaBuilder comment to match implementation

The schema builder API evolved a while back. The class provides a detailed comment to explain usage. That comment was out of date. This fix updates the comment to match the code.

closes #1533

4 weeks agoDRILL-786: Allow CROSS JOIN syntax
Igor Guzenko [Wed, 3 Oct 2018 11:55:31 +0000 (14:55 +0300)] 
DRILL-786: Allow CROSS JOIN syntax

 1. Removed throw statement in UnsupportedOperatorsVisitor
 2. Extended UnsupportedRelOperatorException's message

closes #1488

4 weeks agoDRILL-6848: Duration panel in a query profile's WebUI does not open
Kunal Khatua [Tue, 13 Nov 2018 22:11:51 +0000 (14:11 -0800)] 
DRILL-6848: Duration panel in a query profile's WebUI does not open

DRILL-5571 ( PR #1531 ) accidentally introduced a bug that permanently keeps the "Duration" panel closed instead of opening it. This trivial patch fixes that.

4 weeks agoDrill 6735: Implement Semi-Join for the Hash-Join operator (#1522)
Boaz Ben-Zvi [Tue, 13 Nov 2018 20:58:31 +0000 (12:58 -0800)] 
Drill 6735: Implement Semi-Join for the Hash-Join operator (#1522)

4 weeks agoDRILL-2035: Add ability to cancel multiple queries
Kunal Khatua [Tue, 13 Nov 2018 20:12:09 +0000 (12:12 -0800)] 
DRILL-2035: Add ability to cancel multiple queries

Currently Drill UI allows canceling one query at a time.
This commit (on lines of DRILL-5571 /  PR #1531) allows for cancelling multiple `running` queries.

4 weeks agoDRILL-5571: Cancel running query from its Web UI
Kunal Khatua [Tue, 13 Nov 2018 20:09:42 +0000 (12:09 -0800)] 
DRILL-5571: Cancel running query from its Web UI

Currently, a running query can be cancelled by navigating to the Edit Query Tab of the profile page, making it difficult to access. This commit allows the user to cancel.
In addition, the Duration segment is also made visible by default rather than collapsed.

5 weeks agoDRILL-6760: Retain original exception in Verbose Error Message
shuifeng lu [Wed, 31 Oct 2018 11:54:52 +0000 (19:54 +0800)] 
DRILL-6760: Retain original exception in Verbose Error Message

closes #1519

5 weeks agoDRILL-6717: lower and upper functions not works with national characters
ozinoviev [Tue, 28 Aug 2018 17:29:55 +0000 (20:29 +0300)] 
DRILL-6717: lower and upper functions not works with national characters

closes #1450

5 weeks agoDRILL-6610: Add support for Minimum TLS restriction.
superbstreak [Wed, 18 Jul 2018 21:26:09 +0000 (14:26 -0700)] 
DRILL-6610: Add support for Minimum TLS restriction.

closes #1388

5 weeks agoDRILL-4456: Add Hive translate UDF
Volodymyr Vysotskyi [Thu, 1 Nov 2018 16:31:16 +0000 (18:31 +0200)] 
DRILL-4456: Add Hive translate UDF

closes #1527

5 weeks agoDRILL-3610: Add TIMESTAMPADD and TIMESTAMPDIFF functions
Volodymyr Vysotskyi [Wed, 7 Nov 2018 18:03:40 +0000 (20:03 +0200)] 
DRILL-3610: Add TIMESTAMPADD and TIMESTAMPDIFF functions

closes #1528

5 weeks agoDRILL-6837: Missing reference for dynamically created Javascript library
Kunal Khatua [Thu, 8 Nov 2018 23:13:43 +0000 (15:13 -0800)] 
DRILL-6837: Missing reference for dynamically created Javascript library

A PR for DRILL-6084 broke the auto-complete library required for Edit Query page, with the removal of the static javascript library.
https://github.com/apache/drill/pull/1491/files#diff-2bb3c04f1a9eead1faf57212cce5d2a9

The dynamically loaded script is not defined in the Freemarker template page.
The fix is trivial.

5 weeks agoDRILL-6611: Add Ctrl+Enter support for query submission
Kunal Khatua [Mon, 5 Nov 2018 18:23:39 +0000 (10:23 -0800)] 
DRILL-6611: Add Ctrl+Enter support for query submission

1. BugFix on parent commit: Ensure query submission is done with user name when impersonation is enabled.
2. Support non-Mac browsers
3. Support keyboard submission for profile pages with Edit Query tab.

5 weeks agoDRILL-6611 to enable meta-enter query submission in web query interface
boB Rudis [Mon, 23 Jul 2018 12:16:45 +0000 (08:16 -0400)] 
DRILL-6611 to enable meta-enter query submission in web query interface

6 weeks agoDRILL-6768: Improve to_date, to_time and to_timestamp and corresponding cast function...
Bohdan Kazydub [Wed, 3 Oct 2018 17:07:46 +0000 (20:07 +0300)] 
DRILL-6768: Improve to_date, to_time and to_timestamp and corresponding cast functions to handle empty string when  option is enabled
closes #1494

6 weeks agoDRILL-6809: Handle repeated map in schema inference
Paul Rogers [Sun, 28 Oct 2018 05:47:58 +0000 (22:47 -0700)] 
DRILL-6809: Handle repeated map in schema inference

It turns out that the RowSet utilities build a repeated map without including the hidden $offsets$ vector in the metadata for the map. But, other parts in Drill do include this vector.

The RowSet behavior might be a bug which can be addressed in another PR.

This PR:

* Adds unit tests for map accessors at the row set level. Looks like these were never added originally. They are a simplified form of the ResultSetLoader map tests.
* Verified that the schema inference can infer a schema from a repeated map (using the RowSet style.)
* Added a test to reproduce the case from the bug.
* Made a tweak to the RowSetBuilder to allow access to the RowSetWriter which is needed by the new tests.
* Could of minor clean-ups.

closes #1513

6 weeks agoDRILL-6810: Disable NULL_IF_NULL NullHandling for functions with ComplexWriter
Bohdan Kazydub [Thu, 18 Oct 2018 15:15:23 +0000 (18:15 +0300)] 
DRILL-6810: Disable NULL_IF_NULL NullHandling for functions with ComplexWriter
closes #1509

6 weeks agoDRILL-6824: Handle schema changes in MapRDBJsonRecordReader
Gautam Parai [Sat, 22 Sep 2018 01:44:38 +0000 (18:44 -0700)] 
DRILL-6824: Handle schema changes in MapRDBJsonRecordReader
closes #1518

6 weeks agoDRILL-6798: Planner changes to support semi-join.
Hanumath Rao Maduri [Wed, 10 Oct 2018 00:33:43 +0000 (17:33 -0700)] 
DRILL-6798: Planner changes to support semi-join.

6 weeks agoDRILL-6819: Remove invisible back link in Drill WebUI
Denys Ordynskiy [Thu, 1 Nov 2018 12:04:11 +0000 (14:04 +0200)] 
DRILL-6819: Remove invisible back link in Drill WebUI

6 weeks agoFixed imports for DRILL-6381
dvjyothsna [Wed, 31 Oct 2018 16:58:38 +0000 (09:58 -0700)] 
Fixed imports for DRILL-6381

7 weeks agoDRILL-6763: Codegen optimization of SQL functions with constant values(#1481)
shuifeng lu [Wed, 26 Sep 2018 03:22:20 +0000 (11:22 +0800)] 
DRILL-6763: Codegen optimization of SQL functions with constant values(#1481)

closes #1481

7 weeks agoDRILL-6811: Fix type inference to return correct data mode for boolean functions
Volodymyr Vysotskyi [Wed, 24 Oct 2018 09:57:24 +0000 (12:57 +0300)] 
DRILL-6811: Fix type inference to return correct data mode for boolean functions

closes #1510

7 weeks agoDRILL-6381: Address code review comments (part 3).
Aman Sinha [Sun, 14 Oct 2018 06:38:17 +0000 (23:38 -0700)] 
DRILL-6381: Address code review comments (part 3).

DRILL-6381: Add missing joinControl logic for INTERSECT_DISTINCT.

- Modified HashJoin's probe phase to process INTERSECT_DISTINCT.

- NOTE: For build phase, the functionality will be same as for SemiJoin when it is added later.

DRILL-6381: Address code review comment for intersect_distinct.

DRILL-6381: Rebase on latest master and fix compilation issues.

DRILL-6381: Generate protobuf files for C++ native client.

DRILL-6381: Use shaded Guava classes.  Add more comments and Javadoc.

7 weeks agoDRILL-6381: Address review comments (part 2): fix formatting issues and add javadoc.
Aman Sinha [Wed, 10 Oct 2018 23:36:48 +0000 (16:36 -0700)] 
DRILL-6381: Address review comments (part 2): fix formatting issues and add javadoc.

7 weeks agoDRILL-6381: Address code review comments.
Aman Sinha [Mon, 1 Oct 2018 19:06:39 +0000 (12:06 -0700)] 
DRILL-6381: Address code review comments.

7 weeks agoDRILL-6381: (Part 5) Update Javadoc for a few interfaces.
Aman Sinha [Thu, 13 Sep 2018 10:55:47 +0000 (03:55 -0700)] 
DRILL-6381: (Part 5) Update Javadoc for a few interfaces.

7 weeks agoDRILL-6381: (Part 4) Enhance MapR-DB plugin to support querying secondary indexes
rebase [Fri, 16 Mar 2018 21:24:20 +0000 (14:24 -0700)] 
DRILL-6381: (Part 4) Enhance MapR-DB plugin to support querying secondary indexes

  1. Implementation of the index descriptor for MapR-DB.
  2. MapR-DB specific costing for covering and non-covering indexes.
  3. Discovery componenent to discover the indexes available for a MapR-DB table including CAST functional indexes.
  4. Utility functions to build a canonical index descriptor.
  5. Statistics: fetch and initialize statistcs from MapR-DB for a query condition. Maintain a query-scoped cache for the statistics. Utility functions to compute selectivity.
  6. Range Partitioning: partitioning function that takes into account the tablet map to find out where a particular rowkey belongs.
  7. Restricted Scan: support doing restricted (i.e skip) scan through lookups on the rowkey. Added a group-scan and record reader for this.
  8. MD-3726: Simple Order by queries (without limit) when an index is used are showing regression.
  9. MD-3995: Do not pushdown limit 0 past project with CONVERT_FROMJSON
  10. MD-4259 : Account for limit during hashcode computation

Co-authored-by: Aman Sinha <asinha@maprtech.com>
Co-authored-by: chunhui-shi <cshi@maprtech.com>
Co-authored-by: Gautam Parai <gparai@maprtech.com>
Co-authored-by: Padma Penumarthy <ppenumar97@yahoo.com>
Co-authored-by: Hanumath Rao Maduri <hmaduri@maprtech.com>
Conflicts:
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBFormatMatcher.java
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBPushProjectIntoScan.java
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/JsonTableGroupScan.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/index/rules/DbScanSortRemovalRule.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SortPrel.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/TopNPrel.java

Fix additional compilation issues.

7 weeks agoDRILL-6381: (Part 3) Planner and Execution implementation to support Secondary Indexes
rebase [Fri, 16 Mar 2018 21:25:26 +0000 (14:25 -0700)] 
DRILL-6381: (Part 3) Planner and Execution implementation to support Secondary Indexes

  1. Index Planning Rules and Plan generators
    - DbScanToIndexScanRule: Top level physical planning rule that drives index planning for several relational algebra patterns.
    - DbScanSortRemovalRule: Physical planning rule for index planning for Sort-based operations.
    - Plan Generators: Covering, Non-Covering and Intersect physical plan generators.
    - Support planning with functional indexes such as CAST functions.
    - Enhance PlannerSettings with several configuration options for indexes.
  2. Index Selection and Statistics
    - An IndexSelector that support cost-based index selection of covering and non-covering indexes using statistics and collation properties.
    - Costing of index intersection for comparison with single-index plans.
  3. Planning and execution operators
    - Support RangePartitioning physical operator during query planning and execution.
    - Support RowKeyJoin physical operator during query planning and execution.
    - HashTable and HashJoin changes to support RowKeyJoin and Index Intersection.
    - Enhance Materializer to keep track of subscan association with a particular rowkey join.
  4. Index Planning utilities
    - Utility classes to perform RexNode analysis, including conversion to and from SchemaPath.
    - Utility class to analyze filter condition and an input collation to determine output collation.
    - Helper classes to maintain index contexts for logical and physical planning phase.
    - IndexPlanUtils utility class for various helper methods.
  5. Miscellaneous
    - Separate physical rel for DirectScan.
    - Modify LimitExchangeTranspose rule to handle SingleMergeExchange.
    - MD-3880: Return correct status from RangePartitionRecordBatch setupNewSchema

Co-authored-by: Aman Sinha <asinha@maprtech.com>
Co-authored-by: chunhui-shi <cshi@maprtech.com>
Co-authored-by: Gautam Parai <gparai@maprtech.com>
Co-authored-by: Padma Penumarthy <ppenumar97@yahoo.com>
Co-authored-by: Hanumath Rao Maduri <hmaduri@maprtech.com>
Conflicts:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashJoinPOP.java
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashPartition.java
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTable.java
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Materializer.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillMergeProjectRule.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillOptiq.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushProjectIntoScanRule.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillScanRel.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/BroadcastExchangePrel.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DrillDistributionTrait.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashJoinPrel.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PrelUtil.java
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetPushDownFilter.java
exec/java-exec/src/main/resources/drill-module.conf
logical/src/main/java/org/apache/drill/common/logical/StoragePluginConfig.java

Resolve merge comflicts and compilation issues.

7 weeks agoDRILL-6381: (Part 2) MapRDB plugin update to 6.0.1
rebase [Mon, 12 Feb 2018 22:31:49 +0000 (14:31 -0800)] 
DRILL-6381: (Part 2) MapRDB plugin update to 6.0.1

  1. MD-3960: Update Drill to build with MapR-6.0.1 libraries
  2. MD-3995: Do not pushdown limit 0 past project with CONVERT_FROMJSON
  3. MD-4054: Restricted scan limit is changed to dynamically read rows using the rowcount of the rightside instead of 4096.
  4. MD-3688: Impersonating a view owner doesn't work with security disabled in 6.0
  5. MD-4492: Missing limit pushdown changes in JsonTableGroupScan

Co-authored-by: chunhui-shi <cshi@maprtech.com>
Co-authored-by: Gautam Parai <gparai@maprtech.com>
Co-authored-by: Vlad Rozov <vrozov@mapr.com>
Conflicts:
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBFormatPlugin.java
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/MapRDBSubScan.java
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/binary/BinaryTableGroupScan.java
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/CompareFunctionsProcessor.java
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/JsonConditionBuilder.java
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/JsonTableGroupScan.java
contrib/format-maprdb/src/main/java/org/apache/drill/exec/store/mapr/db/json/MaprDBJsonRecordReader.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PrelUtil.java
pom.xml

7 weeks agoDRILL-6381: (Part 1) Secondary Index framework
rebase [Mon, 12 Feb 2018 22:10:56 +0000 (14:10 -0800)] 
DRILL-6381: (Part 1) Secondary Index framework

  1. Secondary Index planning interfaces and abstract classes like DBGroupScan, DbSubScan, IndexDecriptor etc.
  2. Statistics and Cost model interfaces/classes: PluginCost, Statistics, StatisticsPayload, AbstractIndexStatistics
  3. ScanBatch and RecordReader to support repeatable scan
  4. Secondary Index execution related interfaces: RangePartitionSender, RowKeyJoin, PartitionFunction
  5. MD-3979: Query using cast index plan fails with NPE

Co-authored-by: Aman Sinha <asinha@maprtech.com>
Co-authored-by: chunhui-shi <cshi@maprtech.com>
Co-authored-by: Gautam Parai <gparai@maprtech.com>
Co-authored-by: Padma Penumarthy <ppenumar97@yahoo.com>
Co-authored-by: Hanumath Rao Maduri <hmaduri@maprtech.com>
Conflicts:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillTable.java
protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java
protocol/src/main/java/org/apache/drill/exec/proto/beans/CoreOperatorType.java
protocol/src/main/protobuf/UserBitShared.proto

7 weeks agoDRILL-6715: Update descriptions for System Options table
Kunal Khatua [Thu, 25 Oct 2018 19:08:57 +0000 (12:08 -0700)] 
DRILL-6715: Update descriptions for System Options table

With introduction of DRILL-5735 , the descriptions for about half the system options still remain missing. This commit collects descriptions review by @bbevens

1. Update options for HashAgg/Join (@Ben-Zvi )
2. Update options for Parquet Reader/Writer (@sachouche )
3. Update options for Planners (@HanumathRao , @vdiravka , @KazydubB )
4. Update options for BatchSizing (@bitblender )
5. Update options for Planner Optimizations (@arina-ielchiieva )
6. Update options for Security & Kafka (Krystal Nguyen)
7. Update options for Misc entries (@arina-ielchiieva , @vvysotskyi )

In additional, there is a patch for `org.apache.drill.exec.compile.ClassTransformer.scalar_replacement` , which appears to have replaced `exec.compile.scalar_replacement`. References to the latter have been removed to avoid confusion.
Additional changes include moving the `ClassTransformer` validator to `ExecConstants.java`
Adding support for internal options' descriptions
Removed mention of {{Will be removed in 1.15.0}}. (Refer DRILL-6527)

7 weeks agoDRILL-6804: Simplify usage of OperatorPhase in HashAgg.
Timothy Farkas [Thu, 18 Oct 2018 23:32:03 +0000 (16:32 -0700)] 
DRILL-6804: Simplify usage of OperatorPhase in HashAgg.

8 weeks agoDRILL-6719: Separate spilling queue logic from HashJoin and HashAgg.
Timothy Farkas [Tue, 28 Aug 2018 23:28:49 +0000 (16:28 -0700)] 
DRILL-6719: Separate spilling queue logic from HashJoin and HashAgg.

8 weeks agoDRILL-6795: Upgrade Janino compiler from 2.7.6 to 3.0.10
Volodymyr Vysotskyi [Sat, 13 Oct 2018 13:13:36 +0000 (16:13 +0300)] 
DRILL-6795: Upgrade Janino compiler from 2.7.6 to 3.0.10

closes #1503

8 weeks agoDRILL-6793: FragmentExecutor cannot send its final state for the case when RootExec...
Bohdan Kazydub [Tue, 16 Oct 2018 14:11:36 +0000 (17:11 +0300)] 
DRILL-6793: FragmentExecutor cannot send its final state for the case when RootExec root wasn't initialized

closes #1506

8 weeks agoDRILL-6785: DataClient is using RootAllocator in the bootstrap instead of dataPool
Sorabh Hamirwasia [Mon, 8 Oct 2018 18:28:27 +0000 (11:28 -0700)] 
DRILL-6785: DataClient is using RootAllocator in the bootstrap instead of dataPool

closes #1502

8 weeks agoDRILL-6084: Show Drill functions in WebUI for autocomplete
Kunal Khatua [Thu, 18 Oct 2018 00:30:03 +0000 (17:30 -0700)] 
DRILL-6084: Show Drill functions in WebUI for autocomplete

Building on top of DRILL-3988 and leveraging DRILL-5868, this allows support for Drill functions to be now available in the WebUI.
If users wants UDFs to show up, they should place the UDF jars in the `$DRILL_HOME/jars/3rdparty` directory so that this can be loaded during the Drillbit's startup.
Concept of internal Drill functions are introduced. With this, internal Drill functions like `ConvertToNullableXYZ` has been marked as internal.
The WebUI will not show these functions. However, they are still visible in `sys.functions` table with an additional column indicating that it  is an internal function.
Tests have been added as a part of this commit to verify the internal functions concept.

8 weeks agoDRILL-6797: Fix UntypedNull handling for complex types
Arina Ielchiieva [Thu, 11 Oct 2018 16:25:17 +0000 (19:25 +0300)] 
DRILL-6797: Fix UntypedNull handling for complex types

2 months agoDRILL-6775: The schema for empty output is not shown in Drill Web UI
Anton Gozhiy [Tue, 9 Oct 2018 12:59:16 +0000 (15:59 +0300)] 
DRILL-6775: The schema for empty output is not shown in Drill Web UI

Removed an excess check for the result emptiness that prevented retrieval of the column names.

close apache/drill#1498

2 months agoDRILL-6783: CAST string literal as INTERVAL MONTH/YEAR works inconsistently when...
Bohdan Kazydub [Mon, 8 Oct 2018 13:50:22 +0000 (16:50 +0300)] 
DRILL-6783: CAST string literal as INTERVAL MONTH/YEAR works inconsistently when selecting from a table with multiple rows

close apache/drill#1496

2 months agoDRILL-6777: Setup CircleCI configs for Drill
Vitalii Diravka [Tue, 11 Sep 2018 11:39:13 +0000 (14:39 +0300)] 
DRILL-6777: Setup CircleCI configs for Drill

- adding .circleci/config.yml to the project to launch CircleCI
- custom memory parameters
- usage of CircleCI machine
- excluding "SlowTest" and "UnlikelyTest" groups
- update maven version
- adding libaio.so library to solve MySQL integration tests
- update com.jcabi:jcabi-mysql-maven-plugin library version
- TODO descriptions for the future enhancements of CircleCI build for Drill

close apache/drill#1493

2 months agoDRILL-6473: Update MapR Hive
Bohdan Kazydub [Mon, 21 May 2018 16:11:23 +0000 (19:11 +0300)] 
DRILL-6473: Update MapR Hive

close apache/drill#1307

2 months agoDRILL-3988: Expose Drill built-in functions & UDFs in a system table (#1483)
Kunal Khatua [Thu, 11 Oct 2018 17:30:15 +0000 (10:30 -0700)] 
DRILL-3988: Expose Drill built-in functions & UDFs  in a system table (#1483)

This commit exposes available SQL functions in Drill and also detects UDFs that have been dynamically loaded into Drill.
An example is shown below for 2 UDFs dynamically loaded into the cluster, along side the existing built-in functions that come with Drill.

```
0: jdbc:drill:schema=sys> select source, count(*) as functionCount from sys.functions group by source;
+-----------------------------------------+----------------+
|                 source                  | functionCount  |
+-----------------------------------------+----------------+
| built-in                                | 2704           |
| simple-drill-function-1.0-SNAPSHOT.jar  | 12             |
| drill-url-tools-1.0.jar                 | 1              |
+-----------------------------------------+----------------+
3 rows selected (0.209 seconds)
```

The system table exposes information as shown. The UDF is initialized, making the `returnType` available.
The `random(FLOAT8-REQUIRED,FLOAT8-REQUIRED)` function is an example of a UDF that has overloaded arguments (see `signature`).
The `url_parse(VARCHAR-REQUIRED)` function is another example of an initialized UDF.
Rest are built-in functions that meet the query's filter criteria.
```
0: jdbc:drill:schema=sys> select * from sys.functions where name like 'random' or name like '%url%';
+-------------+----------------------------------+-------------+-----------------------------------------+
|    name     |            signature             | returnType  |                 source                  |
+-------------+----------------------------------+-------------+-----------------------------------------+
| parse_url   | VARCHAR-REQUIRED                 | LATE        | built-in                                |
| random      |                                  | FLOAT8      | built-in                                |
| random      | FLOAT8-REQUIRED,FLOAT8-REQUIRED  | FLOAT8      | simple-drill-function-1.0-SNAPSHOT.jar  |
| url_decode  | VARCHAR-REQUIRED                 | VARCHAR     | built-in                                |
| url_encode  | VARCHAR-REQUIRED                 | VARCHAR     | built-in                                |
| url_parse   | VARCHAR-REQUIRED                 | LATE        | drill-url-tools-1.0.jar                 |
+-------------+----------------------------------+-------------+-----------------------------------------+
6 rows selected (0.619 seconds)
```

2 months agoDRILL-6776: Removed external jQuery dependency
Igor Guzenko [Mon, 8 Oct 2018 10:55:03 +0000 (13:55 +0300)] 
DRILL-6776: Removed external jQuery dependency

2 months agoDRILL-6731: use thread pool to run the runtime filter aggregating work
weijie.tong [Tue, 9 Oct 2018 12:28:22 +0000 (20:28 +0800)] 
DRILL-6731: use thread pool to run the runtime filter aggregating work
closes #1459

2 months agoDRILL-6731: Resolving race conditions in RuntimeFilterSink
Sorabh Hamirwasia [Wed, 26 Sep 2018 20:20:02 +0000 (13:20 -0700)] 
DRILL-6731: Resolving race conditions in RuntimeFilterSink
Add condition variable to avoid starvation of producer thread while acquiring queue lock

2 months agoDRILL-6731: Move the BFs aggregating work from the Foreman to the RuntimeFilter
weijie.tong [Thu, 6 Sep 2018 11:23:35 +0000 (19:23 +0800)] 
DRILL-6731: Move the BFs aggregating work from the Foreman to the RuntimeFilter

2 months agoDRILL-6766: Lateral Unnest query : IllegalStateException - rowId in right batch of...
Sorabh Hamirwasia [Mon, 1 Oct 2018 21:15:33 +0000 (14:15 -0700)] 
DRILL-6766: Lateral Unnest query : IllegalStateException - rowId in right batch of lateral is smaller than rowId in left batch being processed
Note: Issue was in StreamingAgg where if output from one or multiple input batch was splitting into multiple output batch, then remaining input
records were discarded after producing first output batch
closes #1490

2 months agoDRILL-6788: Intermittent unit test failure TestDrillbitResilience.failsWhenParsing...
Sorabh Hamirwasia [Wed, 10 Oct 2018 16:19:33 +0000 (09:19 -0700)] 
DRILL-6788: Intermittent unit test failure TestDrillbitResilience.failsWhenParsing: Query state should be FAILED (and not COMPLETED)
closes #1499

2 months agoDRILL-6410: Fixed memory leak in flat Parquet reader
Salim Achouche [Mon, 8 Oct 2018 17:16:44 +0000 (10:16 -0700)] 
DRILL-6410: Fixed memory leak in flat Parquet reader

2 months agoDRILL-6541: Upgrade ZooKeeper patch version to 3.4.11 for mapr profile
Bohdan Kazydub [Tue, 26 Jun 2018 10:09:15 +0000 (13:09 +0300)] 
DRILL-6541: Upgrade ZooKeeper patch version to 3.4.11 for mapr profile

closes #1398

2 months agoDRILL-6773: The renamed schema with aliases is not shown for queries on empty directories
Vitalii Diravka [Fri, 5 Oct 2018 13:21:55 +0000 (16:21 +0300)] 
DRILL-6773: The renamed schema with aliases is not shown for queries on empty directories

closes #1492

2 months agoDRILL-6764: Query fails with IOB when Unnest has reference to deep nested field like...
Hanumath Rao Maduri [Tue, 2 Oct 2018 23:32:38 +0000 (16:32 -0700)] 
DRILL-6764: Query fails with IOB when Unnest has reference to deep nested field like (t.c_orders.o_lineitems).

closes #1487

2 months agoDRILL-6762: Fix dynamic UDFs versioning issue
Arina Ielchiieva [Fri, 28 Sep 2018 18:47:40 +0000 (20:47 +0200)] 
DRILL-6762: Fix dynamic UDFs versioning issue

1. Added UndefinedVersionDelegatingStore to serve as versioned wrapper for those stores that do not support versioning.
2. Aligned remote and local function registries version type. Type will be represented as int since ZK version is returned as int.
3. Added NOT_AVAILABLE and UNDEFINED versions to DataChangeVersion holder to indicate proper registry state.
4. Added additional trace logging.
5. Minor refactoring and clean up.

closes #1484

2 months agoDRILL-6759: Make columns array name for csv data case insensitive
Arina Ielchiieva [Tue, 2 Oct 2018 13:48:53 +0000 (16:48 +0300)] 
DRILL-6759: Make columns array name for csv data case insensitive

closes #1485

2 months agoDRILL-6765: Exclude unused shaded guava classes from drill-jdbc-all jar
Igor Guzenko [Tue, 2 Oct 2018 17:45:11 +0000 (20:45 +0300)] 
DRILL-6765: Exclude unused shaded guava classes from drill-jdbc-all jar

closes #1486