drill.git
4 years agoDRILL-6044: Fixed shutdown button in Web UI when ssl,auth are enabled 1.13.0 drill-1.13.0
dvjyothsna [Wed, 7 Mar 2018 02:10:22 +0000 (18:10 -0800)] 
DRILL-6044: Fixed shutdown button in Web UI when ssl,auth are enabled

This closes #1153

4 years agoDRILL-6187: Exception in RPC communication between DataClient/ControlClient and respe...
Sorabh Hamirwasia [Thu, 1 Mar 2018 23:08:10 +0000 (15:08 -0800)] 
DRILL-6187: Exception in RPC communication between DataClient/ControlClient and respective servers when bit-to-bit security is on

This closes #1145

4 years agoDRILL-4547: Disabling doclint.
dvjyothsna [Thu, 8 Mar 2018 18:14:34 +0000 (10:14 -0800)] 
DRILL-4547: Disabling doclint.

4 years agoDRILL-1491: Support for JDK 8
vladimir tkach [Mon, 5 Mar 2018 10:07:54 +0000 (12:07 +0200)] 
DRILL-1491: Support for JDK 8

Changed jdk version from 7 to 8 in pom.xml travis and drill-config.sh

This closes #1143

4 years agoDRILL-6217: NaN/Inf NestedLoopJoin processes NaN values incorrectly
Vladimir Tkach [Tue, 6 Mar 2018 09:42:02 +0000 (11:42 +0200)] 
DRILL-6217: NaN/Inf NestedLoopJoin processes NaN values incorrectly

- Changed loggic for equality functions to handle NaN values as the biggest ones

closes #1154

4 years agoDRILL-6213: During restart drillbit should be killed forcefully if exceeds allowed...
Arina Ielchiieva [Wed, 7 Mar 2018 10:35:30 +0000 (12:35 +0200)] 
DRILL-6213: During restart drillbit should be killed forcefully if exceeds allowed timeout

closes #1155

4 years agoDRILL-6192: Drill is vulnerable to CVE-2017-12197
vladimir tkach [Tue, 27 Feb 2018 18:25:28 +0000 (20:25 +0200)] 
DRILL-6192: Drill is vulnerable to CVE-2017-12197

Changed libpam4j version from 1.8-rev1 to 1.9-mapr

closes #1136

4 years agoDRILL-6210: Enhanced test schema utilities
Paul Rogers [Mon, 5 Mar 2018 00:14:33 +0000 (16:14 -0800)] 
DRILL-6210: Enhanced test schema utilities

closes #1150

4 years agoDRILL-6216: Metadata mismatch when connecting to a Drill 1.12.0 with a Drill-1.13...
Sorabh Hamirwasia [Wed, 7 Mar 2018 20:06:32 +0000 (12:06 -0800)] 
DRILL-6216: Metadata mismatch when connecting to a Drill 1.12.0 with a Drill-1.13.0-SNAPSHOT driver

closes #1157

4 years agoDRILL-6177: Merge Join - Allocate memory for outgoing value vectors based on sizes...
Padma Penumarthy [Wed, 7 Mar 2018 00:09:43 +0000 (16:09 -0800)] 
DRILL-6177: Merge Join - Allocate memory for outgoing value vectors based on sizes of incoming batches.

closes #1125

4 years agoDRILL-6126: Allocate memory for value vectors upfront in flatten operator
Padma Penumarthy [Wed, 7 Mar 2018 00:09:13 +0000 (16:09 -0800)] 
DRILL-6126: Allocate memory for value vectors upfront in flatten operator

4 years agoDRILL-6162: Enhance record batch sizer to retain nesting information.
Padma Penumarthy [Wed, 7 Mar 2018 00:05:48 +0000 (16:05 -0800)] 
DRILL-6162: Enhance record batch sizer to retain nesting information.
Refactor record batch sizer and add unit tests for sizing and vector allocation.

4 years agoDRILL-6203: Repeated Map Vector does not give correct payload bytecount.
Padma Penumarthy [Tue, 6 Mar 2018 23:57:34 +0000 (15:57 -0800)] 
DRILL-6203: Repeated Map Vector does not give correct payload bytecount.

4 years agoDRILL-6151: Fragment executors may terminate without sending final batch to a downstr...
Vlad Rozov [Tue, 6 Mar 2018 17:12:32 +0000 (09:12 -0800)] 
DRILL-6151: Fragment executors may terminate without sending final batch to a downstream causing query to hang

closes #1151

4 years agoAdded usage for graceful_stop in drillbit.sh
dvjyothsna [Thu, 22 Feb 2018 19:30:50 +0000 (11:30 -0800)] 
Added usage for graceful_stop in drillbit.sh

closes #1135

4 years agoDRILL-6021:Show shutdown button when authentication is not enabled
Jyothsna Donapati [Thu, 22 Feb 2018 19:17:48 +0000 (11:17 -0800)] 
DRILL-6021:Show shutdown button when authentication is not enabled

closes #1127

4 years agoDRILL-4120: Allow implicit columns for Avro storage format
Volodymyr Vysotskyi [Mon, 5 Mar 2018 12:25:33 +0000 (14:25 +0200)] 
DRILL-4120: Allow implicit columns for Avro storage format

closes #1138

4 years agoDRILL-6218: Update release profile to not generate MD5 checksum
Parth Chandra [Wed, 7 Mar 2018 09:53:42 +0000 (15:23 +0530)] 
DRILL-6218: Update release profile to not generate MD5 checksum

This closes #1156

4 years agoDRILL-6208: Fix FunctionInitializerTest#testConcurrentFunctionBodyLoad to use Mockito...
Arina Ielchiieva [Sat, 3 Mar 2018 19:00:50 +0000 (21:00 +0200)] 
DRILL-6208: Fix FunctionInitializerTest#testConcurrentFunctionBodyLoad to use Mockito instead of JMockit

closes #1149

4 years agoDRILL-6189: Security: passwords logging and file permisions
Vladimir Tkach [Wed, 28 Feb 2018 17:13:51 +0000 (19:13 +0200)] 
DRILL-6189: Security: passwords logging and file permisions

1. Overrided serialization methods for instances with passwords
2. Changed file permissions for configuration files

closes #1139

4 years agoDRILL-1170: YARN integration for Drill
Paul Rogers [Thu, 26 Oct 2017 07:24:00 +0000 (00:24 -0700)] 
DRILL-1170: YARN integration for Drill

closes #1011

4 years agoDRILL-6114: Metadata revisions
Paul Rogers [Tue, 6 Feb 2018 04:18:18 +0000 (20:18 -0800)] 
DRILL-6114: Metadata revisions

Support for union vectors, list vectors, repeated list vectors. Refactored metadata classes.

closes #1112

4 years agoDRILL-6153: Operator framework
Paul Rogers [Tue, 13 Feb 2018 06:27:23 +0000 (22:27 -0800)] 
DRILL-6153: Operator framework

closes #1121

4 years agoDRILL-6180: Use System Option "output_batch_size" for External Sort
Padma Penumarthy [Fri, 23 Feb 2018 00:41:47 +0000 (16:41 -0800)] 
DRILL-6180: Use System Option "output_batch_size" for External Sort

closes #1129

4 years agoDRILL-6099: Push limit past flatten(project) without pushdown into scan
Gautam Parai [Thu, 18 Jan 2018 23:46:42 +0000 (15:46 -0800)] 
DRILL-6099: Push limit past flatten(project) without pushdown into scan

closes #1096

4 years agoDRILL-6197: Skip duplicate entry for OperatorStats
Kunal Khatua [Wed, 28 Feb 2018 22:07:27 +0000 (14:07 -0800)] 
DRILL-6197: Skip duplicate entry for OperatorStats

org.apache.drill.exec.ops.FragmentStats should skip injecting the org.apache.drill.exec.ops.OperatorStats instance for these operators:
org.apache.drill.exec.proto.beans.CoreOperatorType.SCREEN
org.apache.drill.exec.proto.beans.CoreOperatorType.SINGLE_SENDER
org.apache.drill.exec.proto.beans.CoreOperatorType.BROADCAST_SENDER
org.apache.drill.exec.proto.beans.CoreOperatorType.HASH_PARTITION_SENDER

closes #1141

4 years agoDRILL-6185: Fixed error while displaying system profiles via the Web-UI
Kunal Khatua [Tue, 27 Feb 2018 20:10:06 +0000 (12:10 -0800)] 
DRILL-6185: Fixed error while displaying system profiles via the Web-UI

The bug lies in the absence of a text plan for profiles related to system queries:
e.g.
{code:sql}
use dfs.tmp;
show tables;
alter session reset all;
{code}
This addresses that by ensuring an empty string is substituted and the tokenization is done correctly.

closes #1137

4 years agoDRILL-6188: Fix C++ client build on Centos7, OS X
Parth Chandra [Tue, 27 Feb 2018 12:01:51 +0000 (17:31 +0530)] 
DRILL-6188: Fix C++ client build on Centos7, OS X

closes #1132

4 years agoDRILL-6191: Add acknowledgement sequence number and flags fields, details for flags
Ted Dunning [Wed, 3 Jan 2018 00:20:35 +0000 (16:20 -0800)] 
DRILL-6191: Add acknowledgement sequence number and flags fields, details for flags

closes #1134

4 years agoDRILL-6198: OpenTSDB unit tests fail when Lilith client is running
Volodymyr Vysotskyi [Thu, 1 Mar 2018 12:52:28 +0000 (14:52 +0200)] 
DRILL-6198: OpenTSDB unit tests fail when Lilith client is running

closes #1142

4 years agoDRILL-6190: Fix handling of packets longer than legally allowed
Ted Dunning [Thu, 11 Jan 2018 00:52:53 +0000 (16:52 -0800)] 
DRILL-6190: Fix handling of packets longer than legally allowed

closes #1133

4 years agoDRILL-6174: Parquet filter pushdown improvements.
Roman Kulyk [Thu, 15 Feb 2018 15:37:43 +0000 (15:37 +0000)] 
DRILL-6174: Parquet filter pushdown improvements.

Added support IS [NOT] NULL/TRUE/FALSE operator for the parquet filter pushdown.
Added timestamp/date/time implicit/explicit casts.

closes #1131

4 years agoDRILL-6204: Pass tables columns without partition columns to empty Hive reader
Arina Ielchiieva [Fri, 2 Mar 2018 11:38:00 +0000 (13:38 +0200)] 
DRILL-6204: Pass tables columns without partition columns to empty Hive reader

closes #1146

4 years agoDRILL-6195: Quering Hive non-partitioned transactional tables via Drill
Vitalii Diravka [Wed, 28 Feb 2018 14:16:17 +0000 (14:16 +0000)] 
DRILL-6195: Quering Hive non-partitioned transactional tables via Drill

closes #1140

4 years agoDRILL-6205: Reduce memory consumption of testFlattenUpperLimit test
Padma Penumarthy [Fri, 2 Mar 2018 19:13:59 +0000 (11:13 -0800)] 
DRILL-6205: Reduce memory consumption of testFlattenUpperLimit test

closes #1147

4 years agoDRILL-5994: Enable configuring number of Jetty acceptors and selectors (default to...
Vlad Rozov [Fri, 2 Mar 2018 18:39:31 +0000 (10:39 -0800)] 
DRILL-5994: Enable configuring number of Jetty acceptors and selectors (default to 1 acceptor and 2 selectors)

closes #1148

4 years agoDRILL-6118: Handle item star columns during project / filter push down and directory... 1080/head
Arina Ielchiieva [Thu, 21 Dec 2017 17:31:00 +0000 (19:31 +0200)] 
DRILL-6118: Handle item star columns during project / filter push down and directory pruning

1. Added DrillFilterItemStarReWriterRule to re-write item star fields to regular field references.
2. Refactored DrillPushProjectIntoScanRule to handle item star fields, factored out helper classes and methods from PreUitl.class.
3. Fixed issue with dynamic star usage (after Calcite upgrade old usage of star was still present, replaced WILDCARD -> DYNAMIC_STAR  for clarity).
4. Added unit tests to check project / filter push down and directory pruning with item star.

4 years agoDRILL-5741: Automatically manage memory allocations during startup
Kunal Khatua [Fri, 12 Jan 2018 01:35:13 +0000 (17:35 -0800)] 
DRILL-5741: Automatically manage memory allocations during startup

**Note:** This commit is rebased on (hence, requires) DRILL-6068 commit

This commit provides a way for distributions and users to automatically define the Drillbit JVM's Heap, Direct and CodeCache allocations. To do this, it leverages the DRILL-6068 commit, which provides for configuration setup.

The only new variable introduced is an environment variable - `DRILLBIT_MAX_PROC_MEM` that defines a Drillbit's max memory parameters. The logic defined by `auto-setup.sh` will check that if any (or all) of the JVM memory parameters have been declared; they, cumulatively, don't exceed the value specified by `DRILLBIT_MAX_PROC_MEM`.

```
NOTE: To disable this, simply unset (or don't define) the environment variable DRILLBIT_MAX_PROC_MEM
```

The variable can be defined in KB, MB, or  GB; similar in syntax to how the JVM MaxHeap is specified. For e.g.
```
DRILLBIT_MAX_PROC_MEM=13G
DRILLBIT_MAX_PROC_MEM=8192m
DRILLBIT_MAX_PROC_MEM=4194304K
```
In addition, you can specify it as a percent of the total system memory prior to the Drillbit starting up:
`DRILLBIT_MAX_PROC_MEM=40%`

For a system with with 48GB free memory, when set to (say) 25% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit fails startup with the following message:
```
2018-01-03 14:27:57  [WARN] 25% of System Memory (47 GB) translates to 12 GB
2018-01-03 14:27:57  [ERROR]    Unable to start Drillbit due to memory constraint violations
  Total Memory Requested : 19 GB
  Check the following settings to possibly modify (or increase the Max Memory Permitted):
        DRILLBIT_MAX_PROC_MEM=25%
        DRILL_HEAP=8G
        DRILL_MAX_DIRECT_MEMORY=10G
        DRILLBIT_CODE_CACHE_SIZE=1024m
        *NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB
```

For all other combinations, the undefined parameters are adjusted to ensure that the total memory allocated is within the value specified by `DRILLBIT_MAX_PROC_MEM`,

For a system with with 48GB free memory, when set to (say) 50% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit startup with the following warning:
```
2018-01-03 14:31:06  [WARN] 50% of System Memory (47 GB) translates to 24 GB
2018-01-03 14:31:06  [WARN] You have an allocation of 4 GB that is currently unused from a total of 24 GB. You can increase your existing memory configuration to use this extra memory
        DRILLBIT_MAX_PROC_MEM=50%
        DRILL_HEAP=8G
        DRILL_MAX_DIRECT_MEMORY=10G
        DRILLBIT_CODE_CACHE_SIZE=1024m
        *NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB
```

In addition, if the available free memory is less than the allocation, an additional warning is provided under the assumption that the OS will reclaim more free memory when required:
```
2018-01-03 14:31:06  [WARN] Total Memory Allocation for Drillbit (19GB) exceeds available free memory (11GB)
2018-01-03 14:31:06  [WARN] Drillbit will start up, but can potentially crash due to oversubscribing of system memory.
```

For more details, refer the attachments in https://issues.apache.org/jira/browse/DRILL-5741

Changes to auto configure messaging

Publishing final values prior to startup

Minor update for printing to console's err stream

close apache/drill#1082

4 years agoDRILL-6068: Support user/distrib-specific config checks during startup
Kunal Khatua [Thu, 11 Jan 2018 23:45:04 +0000 (15:45 -0800)] 
DRILL-6068: Support user/distrib-specific config checks during startup

1. Allows for distrib/user specific checks to be done
2. Place-holder files for distribution and user specific checks
3. Moved JVM Version Check to head of script
Separation of setups into 3 levels

1. Apache Drill (Default)
2. Distribition Specific
3. User-Defined
1 & 2 are mutually exclusive.
3 is additional checks that the user can specify.

4 years agoDRILL-5978: Updating of Apache and MapR Hive libraries to 2.3.2 and 2.1.2-mapr-1710...
Vitalii Diravka [Mon, 13 Nov 2017 16:04:03 +0000 (16:04 +0000)] 
DRILL-5978: Updating of Apache and MapR Hive libraries to 2.3.2 and 2.1.2-mapr-1710 versions respectively

  * Improvements to allow of reading Hive bucketed transactional ORC tables;
  * Updating hive properties for tests and resolving dependencies and API conflicts:
  - Fix for "hive.metastore.schema.verification", MetaException(message: Version information
  not found in metastore) https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool
  METASTORE_SCHEMA_VERIFICATION="false" property is added
  - Added METASTORE_AUTO_CREATE_ALL="true", properties to tests, because some additional
  tables are necessary in Hive metastore
  - Disabling calcite CBO for (Hive's CalcitePlanner) for tests, because it is in conflict
  with Drill's Calcite version for Drill unit tests. HIVE_CBO_ENABLED="false" property
  - jackson and parquet libraries are relocated in hive-exec-shade module
  - org.apache.parquet:parquet-column Drill version is added to "hive-exec" to
  allow of using Parquet empty group on MessageType level (PARQUET-278)
  - Removing of commons-codec exclusion from hive core. This dependency is
  necessary for hive-exec and hive-metastore.
  - Setting Hive internal properties for transactional scan:
  HiveConf.HIVE_TRANSACTIONAL_TABLE_SCAN and for schema evolution: HiveConf.HIVE_SCHEMA_EVOLUTION,
  IOConstants.SCHEMA_EVOLUTION_COLUMNS, IOConstants.SCHEMA_EVOLUTION_COLUMNS_TYPES
  - "io.dropwizard.metrics:metrics-core" with last 4.0.2 version is added to dependencyManagement block in Drill root POM
  - Exclusion of "hive-exec" in "hive-hbase-handler" is already in Drill root dependencyManagement POM
  - Hive Calcite libraries are excluded (Calcite CBO was disabled)
  - "jackson-core" dependency is added to DependencyManagement block in Drill root POM file
  - For MapR Hive 2.1 client older "com.fasterxml.jackson.core:jackson-databind" is included
  - "log4j:log4j" dependency is excluded from "hive-exec", "hive-metastore", "hive-hbase-handler".

close apache/drill#1111

4 years agoDRILL-6172: setValueCount of VariableLengthVectors throws IOB exception when called...
Sorabh Hamirwasia [Wed, 21 Feb 2018 01:14:29 +0000 (17:14 -0800)] 
DRILL-6172: setValueCount of VariableLengthVectors throws IOB exception when called with 0 value after clearing vectors

close apache/drill#1124

4 years agoDRILL-6154: NaN, Infinity issues
vladimir tkach [Fri, 16 Feb 2018 14:36:49 +0000 (16:36 +0200)] 
DRILL-6154: NaN, Infinity issues

- changed comparison rules for NaN, Infinity values. For now NaN is the biggest value, Infinity - second biggest value
- fixed min, max, trunc functions for NaN, Infinity values
- made drill use original sqrt function instead of power(x,0.5) substitution

close apache/drill#1123

4 years agoDRILL-5902: Queries encounter random failure due to RPC connection timed out
Vlad Rozov [Tue, 6 Feb 2018 03:15:56 +0000 (19:15 -0800)] 
DRILL-5902: Queries encounter random failure due to RPC connection timed out

close apache/drill#1113

4 years agoDRILL-6115: SingleMergeExchange is not scaling up when many minor fragments are alloc...
Hanumath Rao Maduri [Tue, 13 Feb 2018 02:19:07 +0000 (18:19 -0800)] 
DRILL-6115: SingleMergeExchange is not scaling up when many minor fragments are allocated for a query.

DRILL-6115: Refactoring the existing code.

close apache/drill#1110

4 years agoDRILL-6090: While connecting to drill-bits using JDBC Driver through Zookeeper, a...
milindt [Wed, 17 Jan 2018 09:05:33 +0000 (14:35 +0530)] 
DRILL-6090: While connecting to drill-bits using JDBC Driver through Zookeeper, a lot of "Curator-Framework-0" threads are created if connection to drill-bit is not successful(no drill-bits are up/reachable)

I am using Drill JDBC driver 1.12.0 to connect to MapR-DB. I am finding the available drill-bits using Zookeepers. When drill-bits are not up or not reachable, the connection is failed with exception: "Failure in connecting to Drill: oadd.org.apache.drill.exec.rpc.RpcException: Failure setting up ZK for client", which is expected, but number of threads created by ZKClusterCoordinator just keeps on increasing.

Steps to reproduce the issue

Setup a connection with a drill-bit using Apache Drill JDBC driver 1.12.0 through Zookeeper hosts(port 5181)
Now stop the drill-bit services or block the drill-bit IPs using iptable rules
Truncate catalina logs
Try to connect to the drill-bit/hit a code path that requires connection to drill-bits.
Take thread dump using kill -QUIT <java process id>
grep -c "Curator-Framework-0" catalina.out
Observe that the curator framework thread just keep on accumulating

RCA:

ZKClusterCoordinator creates curator threads in the constructor
ZKClusterCoordinator is instantiated by DrillClient.connect
DrillClient.connect is called in DrillConnectionImpl constructor

Fix:

Call DrillConnectionImpl .cleanup() from all the catch blocks in the DrillConnectionImpl  constructor.

close apache/drill#1094

4 years agoDRILL-6164: Heap memory leak during parquet scan and OOM
Vlad Rozov [Fri, 16 Feb 2018 03:25:21 +0000 (19:25 -0800)] 
DRILL-6164: Heap memory leak during parquet scan and OOM

closes #1122

4 years agoDRILL-6089: Removed ordering trait from HashJoin in planner and verified the planner...
Timothy Farkas [Fri, 26 Jan 2018 21:46:22 +0000 (13:46 -0800)] 
DRILL-6089: Removed ordering trait from HashJoin in planner and verified the planner does not assume HashJoin preserves ordering.

closes #1117

4 years agoDRILL-6123: Limit batch size for Merge Join based on memory
Padma Penumarthy [Fri, 9 Feb 2018 21:54:38 +0000 (13:54 -0800)] 
DRILL-6123: Limit batch size for Merge Join based on memory

closes #1107

4 years agoDRILL-6130: Fix NPE during physical plan submission for various storage plugins
Arina Ielchiieva [Thu, 1 Feb 2018 17:44:43 +0000 (17:44 +0000)] 
DRILL-6130: Fix NPE during physical plan submission for various storage plugins

1. Fixed ser / de issues for Hive, Kafka, Hbase plugins.
2. Added physical plan submission unit test for all storage plugins in contrib module.
3. Refactoring.

closes #1108

4 years agoDRILL-6140: Correctly list Operators in Profiles Page
Kunal Khatua [Wed, 7 Feb 2018 06:14:58 +0000 (22:14 -0800)] 
DRILL-6140: Correctly list Operators in Profiles Page

Operators listed in Profiles Page don't always correspond with operator specified in Physical Plan.
This commit fixes that by using the PhysicalPlan as a reference, but reverts to the inferred names in the event of an Exchange-based operator

closes #1116

4 years agoDRILL-6143: Made FragmentsRunner's rpc timeout larger to reduce random failures and...
Timothy Farkas [Thu, 8 Feb 2018 23:25:59 +0000 (15:25 -0800)] 
DRILL-6143: Made FragmentsRunner's rpc timeout larger to reduce random failures and made it configurable as a SystemOption.

closes #1119

4 years agoDILL-6148: TestSortSpillWithException is sometimes failing.
Hanumath Rao Maduri [Fri, 9 Feb 2018 22:13:04 +0000 (14:13 -0800)] 
DILL-6148: TestSortSpillWithException is sometimes failing.

closes #1120

4 years agoDRILL-6128: Wrong Result with Nested Loop Join
Sorabh Hamirwasia [Sat, 3 Feb 2018 02:52:38 +0000 (18:52 -0800)] 
DRILL-6128: Wrong Result with Nested Loop Join

This closes #1109

4 years agoDRILL-6144: Make direct memory amount configurable in tests.
Timothy Farkas [Fri, 9 Feb 2018 00:25:23 +0000 (16:25 -0800)] 
DRILL-6144: Make direct memory amount configurable in tests.

This closes #1118

4 years agoDRILL-6138: Move RecordBatchSizer to org.apache.drill.exec.record package
Padma Penumarthy [Tue, 6 Feb 2018 05:41:45 +0000 (21:41 -0800)] 
DRILL-6138: Move RecordBatchSizer to org.apache.drill.exec.record package

This closes #1115

4 years agoDRILL-6129: Fixed query failure due to nested column data type change
Salim Achouche [Thu, 1 Feb 2018 02:59:58 +0000 (18:59 -0800)] 
DRILL-6129: Fixed query failure due to nested column data type change

This closes #1106

4 years agoDRILL-6137: Fixed join error when one file partition is empty
Timothy Farkas [Mon, 5 Feb 2018 20:23:31 +0000 (12:23 -0800)] 
DRILL-6137: Fixed join error when one file partition is empty

4 years agoDRILL-5993: Fixed copyEntry for var length nullable vectors.
Ben-Zvi [Tue, 16 Jan 2018 19:29:38 +0000 (11:29 -0800)] 
DRILL-5993: Fixed copyEntry for var length nullable vectors.

4 years agoDRILL-5993: Used generic copiers in the selection vector remover, and implemented...
Timothy Farkas [Thu, 30 Nov 2017 17:55:32 +0000 (09:55 -0800)] 
DRILL-5993: Used generic copiers in the selection vector remover, and implemented testing improvements for RowSets and codegen.

closes #1057

4 years agoDRILL-5993: Adding generic copiers which do not require codegen
Paul Rogers [Thu, 30 Nov 2017 17:39:16 +0000 (09:39 -0800)] 
DRILL-5993: Adding generic copiers which do not require codegen

4 years agoDRILL-6119: The OpenTSDB storage plugin is not included in the Drill distribution
Vlad Storona [Tue, 30 Jan 2018 12:55:04 +0000 (14:55 +0200)] 
DRILL-6119: The OpenTSDB storage plugin is not included in the Drill distribution

closes #1102

4 years agoDRILL-4185: UNION ALL involving empty directory on any side of union all results...
Vitalii Diravka [Fri, 1 Dec 2017 20:48:05 +0000 (22:48 +0200)] 
DRILL-4185: UNION ALL involving empty directory on any side of union all results in Failed query

closes #1083

4 years agoDRILL-6106: Use valueOf method instead of constructor since valueOf has a higher...
reudismam [Wed, 24 Jan 2018 12:36:04 +0000 (09:36 -0300)] 
DRILL-6106: Use valueOf method instead of constructor since valueOf has a higher performance by caching frequently requested values.

closes #1099

4 years agoDRILL-6080: Sort incorrectly limits batch size to 65535 records
Paul Rogers [Thu, 11 Jan 2018 00:04:53 +0000 (16:04 -0800)] 
DRILL-6080: Sort incorrectly limits batch size to 65535 records

closes #1090

* Sort incorrectly limits batch size to 65535 records rather than 65536.
* This PR also includes a few code cleanup items.
* Fix for overflow in offset vector in row set writer
* Performance tool update
* Replace "unsafe" methods with "set" methods
* Also fixes an indexing issue with nullable writers
* Removed debug & timing code
* Increase strictness for batch size

4 years agoDRILL-6071: Limit batch size for flatten operator
Padma Penumarthy [Wed, 10 Jan 2018 13:06:58 +0000 (05:06 -0800)] 
DRILL-6071: Limit batch size for flatten operator

closes #1091

4 years agoDRILL-3640: Support JDBC Statement.setQueryTimeout(int)
Kunal Khatua [Wed, 29 Nov 2017 20:12:53 +0000 (12:12 -0800)] 
DRILL-3640: Support JDBC Statement.setQueryTimeout(int)

Allow for queries to be cancelled if they don't complete within the stipulated time.
This is done by having Drill[Prepared]StatementImpl create a Stopwatch timer to track elapsed time.
  * DrillCursor uses this to detect timeouts.
  * DrillResultSetImpl uses this to detech timeout from the client side (e.g. a slow client, when all batches have been processed by DrillCursor)
* Tests added to test these and other query timeout scenarios.
* Dependent on DRILL-5973 for enabling server-triggered timeout tests
NOTE: PreparedStatementTest.testServerTriggeredQueryTimeout is disabled

closes #1024

4 years agoDRILL-6102: Fix ConcurrentModificationException in the BaseAllocator's print method
Timothy Farkas [Mon, 22 Jan 2018 20:02:43 +0000 (12:02 -0800)] 
DRILL-6102: Fix ConcurrentModificationException in the BaseAllocator's print method

closes #1100

4 years agoDRILL-5730: Mock testing improvements and interface improvements
Timothy Farkas [Thu, 11 Jan 2018 22:59:41 +0000 (14:59 -0800)] 
DRILL-5730: Mock testing improvements and interface improvements

closes #1045

4 years agoDRILL-6100: Intermittent failure while reading Parquet file footer during planning...
Sorabh Hamirwasia [Mon, 22 Jan 2018 00:30:25 +0000 (16:30 -0800)] 
DRILL-6100: Intermittent failure while reading Parquet file footer during planning phase

close apache/drill#1097

4 years agoDRILL-6049: Misc. hygiene and code cleanup changes
Paul Rogers [Thu, 21 Dec 2017 07:59:06 +0000 (23:59 -0800)] 
DRILL-6049: Misc. hygiene and code cleanup changes

close apache/drill#1085

4 years agoDRILL-6002: Avoid memory copy from direct buffer to heap while spilling to local...
Vlad Rozov [Wed, 22 Nov 2017 22:06:13 +0000 (14:06 -0800)] 
DRILL-6002: Avoid memory copy from direct buffer to heap while spilling to local disk

close apache/drill#1058

4 years agoDRILL-5879: Improved SQL Pattern Contains Performance
Salim Achouche [Wed, 13 Dec 2017 22:24:45 +0000 (14:24 -0800)] 
DRILL-5879: Improved SQL Pattern Contains Performance

close apache/drill#1072

4 years agoDRILL-6079 : Fixed memory leak in Parquet Reader
Salim Achouche [Fri, 22 Dec 2017 19:50:56 +0000 (11:50 -0800)] 
DRILL-6079 : Fixed memory leak in Parquet Reader

close apache/drill#1087

4 years agoDRILL-5868: Support SQL syntax highlighting of queries
Kunal Khatua [Mon, 22 Jan 2018 20:01:32 +0000 (12:01 -0800)] 
DRILL-5868: Support SQL syntax highlighting of queries

Based on the commit for DRILL-5981 (PR #1043), this commit further leverages the Ace JavaScript library with customizations specific to Drill.

This commit introduces the following to the Query Editor (including the Edit Query tab within an existing profile to rerunning the query).
1. Syntax highlighting (This is supported for submitted query profiles
2. Autocomplete supported in editors
3. Specifying Drill specific keywords and functions in visible autocomplete
4. Key snippets (template SQLs) allowing for rapid writing of syntax:
  i. Query System Tables
  ii. CView, CTAS and CTempTAS
  iii. Alter Session
  iv. Explain and Select * queries

NOTE: The lists for #3 and #4 are not exhaustive. As more features are added to Drill, these lists can be expanded.
Updates based on review comments

1. Disabled warning message
2. Extended reserved keyword list
3. Fixed bugs
List:
Update 1: Bug fix for when Impersonation is enabled
Update 2: Remove the duplicate editor that might get injected
Update 3: Removed trailing whitespaces in Javascripts

close apache/drill#1084

4 years agoDRILL-6093 : Account for simple columns in project cpu costing
Gautam Parai [Tue, 16 Jan 2018 23:16:16 +0000 (15:16 -0800)] 
DRILL-6093 : Account for simple columns in project cpu costing

close apache/drill#1093

4 years agoDRILL-5851: Empty table during a join operation with a non empty table produces cast...
Hanumath Maduri [Mon, 9 Oct 2017 20:08:13 +0000 (13:08 -0700)] 
DRILL-5851: Empty table during a join operation with a non empty table produces cast exception.

close apache/drill#1059

4 years agoDRILL-6088: MainLoginPageModel errors out when http.auth.mechanisms is not configured
Sorabh Hamirwasia [Mon, 15 Jan 2018 06:25:51 +0000 (22:25 -0800)] 
DRILL-6088: MainLoginPageModel errors out when http.auth.mechanisms is not configured

DRILL-6088: Update based on review feedback

close apache/drill#1092

4 years agoDRILL-6085: Fixed spontaneous vm exits on Travis.
Timothy Farkas [Tue, 16 Jan 2018 20:11:25 +0000 (12:11 -0800)] 
DRILL-6085: Fixed spontaneous vm exits on Travis.

close apache/drill#1095

4 years agoDRILL-6081: Set query end time before writing final profile
Arina Ielchiieva [Thu, 11 Jan 2018 15:40:04 +0000 (17:40 +0200)] 
DRILL-6081: Set query end time before writing final profile

close apache/drill#1088

4 years agoDRILL-5967: Fixed memory leak in OrderedPartitionSender
Timothy Farkas [Thu, 14 Dec 2017 18:48:27 +0000 (10:48 -0800)] 
DRILL-5967: Fixed memory leak in OrderedPartitionSender

closes apache/drill#1073

4 years agoDRILL-3993: Fix typo in DrillRelBuilder Javadoc 1066/head
Volodymyr Vysotskyi [Wed, 17 Jan 2018 19:50:34 +0000 (21:50 +0200)] 
DRILL-3993: Fix typo in DrillRelBuilder Javadoc

4 years agoDRILL-3993: Changes after review:
Volodymyr Vysotskyi [Tue, 16 Jan 2018 13:38:17 +0000 (15:38 +0200)] 
DRILL-3993: Changes after review:

Moved checks for OOM into spillIfNeeded method to avoid excessive call of chooseAPartitionToFlush method.
Ignored unit tests until DRILL-6018 is fixed.
These failures appears because new Calcite tries to simplify expressions from the query when applying ReduceExpressionsRule and fails with NFE.
Similar problem, but with old version of Calcite was described in DRILL-6018, therefore these unit tests are marked as ignored until DRILL-6018 is fixed.

4 years agoDRILL-5089: Dynamically load schema of storage plugin only when needed for every...
chunhui-shi [Fri, 3 Nov 2017 00:06:25 +0000 (02:06 +0200)] 
DRILL-5089: Dynamically load schema of storage plugin only when needed for every query

For each query, loading all storage plugins and loading all workspaces under file system plugins is not needed.

This patch use DynamicRootSchema as the root schema for Drill. Which loads correspondent storage only when needed.

infoschema to read full schema information and load second level schema accordingly.

for workspaces under the same Filesyetm, no need to create FileSystem for each workspace.

use fs.access API to check permission which is available after HDFS 2.6 except for windows + local file system case.

Add unit tests to test with a broken mock storage: with a storage that will throw Exception in regiterSchema method,
all queries even on good storages shall fail without this fix(Drill still load all schemas from all storages).

(cherry picked from commit a66d1d7)

4 years agoDRILL-3993: Resolve conflicts
Volodymyr Vysotskyi [Tue, 9 Jan 2018 10:44:12 +0000 (12:44 +0200)] 
DRILL-3993: Resolve conflicts

4 years agoDRILL-3993: Add unit tests for DRILL-4469 & DRILL-5768
Volodymyr Vysotskyi [Fri, 5 Jan 2018 13:35:45 +0000 (15:35 +0200)] 
DRILL-3993: Add unit tests for DRILL-4469 & DRILL-5768

4 years agoDRILL-3993: Add check for OOM in HashAgg
Volodymyr Vysotskyi [Fri, 29 Dec 2017 15:36:08 +0000 (17:36 +0200)] 
DRILL-3993: Add check for OOM in HashAgg

4 years agoDRILL-3993: Fix aggregate exchange rules for the cases when aggregate rel node contai...
Volodymyr Vysotskyi [Thu, 4 Jan 2018 14:05:53 +0000 (16:05 +0200)] 
DRILL-3993: Fix aggregate exchange rules for the cases when aggregate rel node contains several calls

4 years agoDRILL-3993: Changes for CALCITE-2018
Volodymyr Vysotskyi [Tue, 26 Dec 2017 13:03:59 +0000 (15:03 +0200)] 
DRILL-3993: Changes for CALCITE-2018

4 years agoDRILL-3993: Changes for CALCITE-2092
Volodymyr Vysotskyi [Tue, 26 Dec 2017 12:03:54 +0000 (14:03 +0200)] 
DRILL-3993: Changes for CALCITE-2092

4 years agoDRILL-3993: Changes after code review.
Volodymyr Vysotskyi [Thu, 21 Dec 2017 13:01:50 +0000 (15:01 +0200)] 
DRILL-3993: Changes after code review.

4 years agoDRILL-3993: Move Drill-specific commits 'CALCITE-628' and 'Drill-specific change...
Volodymyr Vysotskyi [Wed, 13 Dec 2017 12:53:47 +0000 (14:53 +0200)] 
DRILL-3993: Move Drill-specific commits 'CALCITE-628' and 'Drill-specific change: Add back AbstractConverter in RelSet.java' from Calcite into DRILL

4 years agoDRILL-3993: Changes to support Calcite 1.15.
Volodymyr Vysotskyi [Fri, 8 Dec 2017 13:05:03 +0000 (15:05 +0200)] 
DRILL-3993: Changes to support Calcite 1.15.

Fix AssertionError: type mismatch for tests with aggregate functions.
Fix VARIANCE agg function
Remove using deprecated Subtype enum
Fix 'Failure while loading table a in database hbase' error
Fix 'Field ordinal 1 is invalid for  type '(DrillRecordRow[*])'' unit test failures

4 years agoDRILL-3993: Fix failed tests after Calcite update
Roman Kulyk [Thu, 2 Nov 2017 18:22:36 +0000 (18:22 +0000)] 
DRILL-3993: Fix failed tests after Calcite update

- fix temporary table errors according to updated logic;
- fixed errors when we trying to make select from hbase table with schema name in query (example: "SELECT row_key FROM hbase.TestTableNullStr) from hbase schema (did "USE hbase" before). Added test for it;
- added fix for views which were created on Calcite 1.4 and test for it.

4 years agoDRILL-3993: Use custom RelBuilder implementation in rules
Volodymyr Vysotskyi [Wed, 15 Nov 2017 10:37:50 +0000 (12:37 +0200)] 
DRILL-3993: Use custom RelBuilder implementation in rules

After the changes, made in CALCITE-1056 if the filter has a predicate that is always false, RelBuilder.filter() method returns values rel node instead of filter rel node. In order to preserve column types, DrillRelBuilder.empty() method, which is returned by filter method  was overridden, and now it returns filter with a false predicate. (advice to override this method was in its javadoc) The goal of all other changes in this commit is to use our custom RelBuilder for all rules that are used in Drill.

4 years agoDRILL-3993: Fix unit test failures connected with support Calcite 1.13
Volodymyr Vysotskyi [Fri, 3 Nov 2017 12:18:09 +0000 (12:18 +0000)] 
DRILL-3993: Fix unit test failures connected with support Calcite 1.13

- Use root schema as default for describe table statement.
Fix TestOpenTSDBPlugin.testDescribe() and TestInfoSchemaOnHiveStorage.varCharMaxLengthAndDecimalPrecisionInInfoSchema() unit tests.
- Modify expected results for tests:
TestPreparedStatementProvider.invalidQueryValidationError();
TestProjectPushDown.testTPCH1();
TestProjectPushDown.testTPCH3();
TestStorageBasedHiveAuthorization.selectUser1_db_u0_only();
TestStorageBasedHiveAuthorization.selectUser0_db_u1g1_only()
- Fix TestCTAS.whenTableQueryColumnHasStarAndTableFiledListIsSpecified(), TestViewSupport.createViewWhenViewQueryColumnHasStarAndViewFiledListIsSpecified(), TestInbuiltHiveUDFs.testIf(), testDisableUtf8SupportInQueryString unit tests.
- Fix UnsupportedOperationException and NPE for jdbc tests.
- Fix AssertionError: Conversion to relational algebra failed to preserve datatypes

*DrillCompoundIdentifier:
According to the changes, made in [CALCITE-546], star Identifier is replaced by empty string during parsing the query. Since Drill uses its own DrillCompoundIdentifier, it should also replace star by empty string before creating SqlIdentifier instance to avoid further errors connected with star column. see SqlIdentifier.isStar() method.

*SqlConverter:
In [CALCITE-1417] added simplification of expressions which should be projected every time when a new project rel node is created using RelBuilder. It causes assertion errors connected with types nullability. This hook was set to false to avoid project expressions simplification. See usage of this hook and RelBuilder.project() method.

In Drill the type nullability of the function depends on only the nullability of its arguments. In some cases, a function may return null value even if it had non-nullable arguments. When Calice simplifies expressions, it checks that the type of the result is the same as the type of the expression. Otherwise, makeCast() method is called. But when a function returns null literal, this cast does nothing, even when the function has a non-nullable type. So to avoid this issue, method makeCast() was overridden.

*DrillAvgVarianceConvertlet:
Problem with sum0 and specific changes in old Calcite (it is CALCITE-777). (see HistogramShuttle.visitCall method) Changes were made to avoid changes in Calcite.

*SqlConverter, DescribeTableHandler, ShowTablesHandler:
New Calcite tries to combine both default and specified workspaces during the query validation. In some cases, for example, when describe table statement is used, Calcite tries to find INFORMATION_SCHEMA in the schema used as default. When it does not find the schema, it tries to find a table with such name. For some storage plugins, such as opentsdb and hbase, when a table was not found, the error is thrown, and the query fails. To avoid this issue, default schema was changed to root schema for validation stage for describe table and show tables queries.

4 years agoDRILL-3993: Changes to support Calcite 1.13
Roman Kulyk [Tue, 29 Aug 2017 14:10:24 +0000 (14:10 +0000)] 
DRILL-3993: Changes to support Calcite 1.13

- fixed all compiling errors (main changes were: Maven changes, chenges RelNode -> RelRoot, implementing some new methods from updated interfaces, chenges some literals, logger changes);
- fixed unexpected column errors, validation errors and assertion errors after Calcite update;
- fixed describe table/schema statement according to updated logic;
- added fixes with time-intervals;
- changed precision of BINARY to 65536 (was 1048576) according to updated logic (Calcite overrides bigger precision to own maxPrecision);
- ignored some incorrect tests with DRILL-3244;
- changed "Table not found" message to "Object not found within" according to new Calcite changes.

4 years agoREVERTED: DRILL-5089
Roman Kulyk [Thu, 30 Nov 2017 16:19:12 +0000 (16:19 +0000)] 
REVERTED:  DRILL-5089

Dynamically load schema of storage plugin only when needed for every query

4 years agoDRILL-6054: don't try to split the filter when it is not AND
chunhui-shi [Fri, 22 Dec 2017 23:42:27 +0000 (15:42 -0800)] 
DRILL-6054: don't try to split the filter when it is not AND

This closes #1078

4 years agoDRILL-6030: Managed sort should minimize number of batches in a k-way merge
Vlad Rozov [Sun, 17 Dec 2017 17:25:55 +0000 (09:25 -0800)] 
DRILL-6030: Managed sort should minimize number of batches in a k-way merge

This closes #1075

4 years agoDRILL-6025: Display execution time of a query in RUNNING state
Prasad Nagaraj Subramanya [Thu, 11 Jan 2018 15:14:52 +0000 (07:14 -0800)] 
DRILL-6025: Display execution time of a query in RUNNING state

This closes #1074