drill.git
7 years agoCreate 1.0.0 release version 1.0.0 drill-1.0.0
Jacques Nadeau [Wed, 20 May 2015 03:05:24 +0000 (20:05 -0700)] 
Create 1.0.0 release version

7 years agoDRILL-3115: SQLLine colors do not work well with CYGWIN
Aditya Kishore [Fri, 15 May 2015 23:57:33 +0000 (16:57 -0700)] 
DRILL-3115: SQLLine colors do not work well with CYGWIN

7 years agoDRILL-3107: Dynamic partition pruning fails on Windows (TestDirectoryExplorerUDFs)
Aditya Kishore [Fri, 15 May 2015 17:13:04 +0000 (10:13 -0700)] 
DRILL-3107: Dynamic partition pruning fails on Windows (TestDirectoryExplorerUDFs)

7 years agoDRILL-3102: The doc link on Drill Web UI points to the old wiki page
Aditya Kishore [Fri, 15 May 2015 20:13:36 +0000 (13:13 -0700)] 
DRILL-3102: The doc link on Drill Web UI points to the old wiki page

7 years agoAdd bounds check in setByte() method
Steven Phillips [Fri, 15 May 2015 22:12:18 +0000 (15:12 -0700)] 
Add bounds check in setByte() method

7 years agoDRILL-3092: Fix memory leak risk near uses of RecordBatchData classes that need to...
Jason Altekruse [Fri, 15 May 2015 00:32:09 +0000 (17:32 -0700)] 
DRILL-3092: Fix memory leak risk near uses of RecordBatchData classes that need to have their resources freed upon an allocation or other runtime failure.

7 years agoDRILL-3110: Coerce Java's Direct OutOfMemoryErrors to look like reservation failures...
Jacques Nadeau [Fri, 15 May 2015 23:50:01 +0000 (16:50 -0700)] 
DRILL-3110: Coerce Java's Direct OutOfMemoryErrors to look like reservation failures to follow existing Drill behavior paths.

7 years agoUpdate sqlline for small rc0 issues.
Jacques Nadeau [Fri, 15 May 2015 23:19:02 +0000 (16:19 -0700)] 
Update sqlline for small rc0 issues.

7 years agoDRILL-3100: TestImpersonationDisabledWithMiniDFS fails on Windows
Aditya Kishore [Fri, 15 May 2015 05:00:34 +0000 (22:00 -0700)] 
DRILL-3100: TestImpersonationDisabledWithMiniDFS fails on Windows

7 years agoDRILL-3099: FileSelection's selectionRoot does not include the scheme and authority
Aditya Kishore [Fri, 15 May 2015 04:56:13 +0000 (21:56 -0700)] 
DRILL-3099: FileSelection's selectionRoot does not include the scheme and authority

7 years agoDRILL-3098: Set Unix style "line.separator" for tests
Aditya Kishore [Fri, 15 May 2015 04:51:18 +0000 (21:51 -0700)] 
DRILL-3098: Set Unix style "line.separator" for tests

7 years agoDRILL-3089: Revert to 2 forked test and allow override from command line
Aditya Kishore [Thu, 14 May 2015 23:11:08 +0000 (16:11 -0700)] 
DRILL-3089: Revert to 2 forked test and allow override from command line

7 years agoWhen interrupt terminating RPC bus waitForSendComplete, make sure to release buffers.
Jacques Nadeau [Fri, 15 May 2015 04:56:14 +0000 (21:56 -0700)] 
When interrupt terminating  RPC bus waitForSendComplete, make sure to release buffers.

7 years agoFix PartitionSenderRootExec possible memory leak.
Jacques Nadeau [Fri, 15 May 2015 04:55:44 +0000 (21:55 -0700)] 
Fix PartitionSenderRootExec possible memory leak.

7 years agoDRILL-3052, DRILL-3066: Improve fragment state management in face of early cancellation.
Jacques Nadeau [Fri, 15 May 2015 03:07:29 +0000 (20:07 -0700)] 
DRILL-3052, DRILL-3066: Improve fragment state management in face of early cancellation.

7 years agoDRILL-3081: Populate connection name as late as possible so RPC error messages are...
Parth Chandra [Thu, 14 May 2015 19:27:40 +0000 (12:27 -0700)] 
DRILL-3081: Populate connection name as late as possible so RPC error messages are reported correctly.

7 years agoUpdate configuration defaults. Rename bounds system property to drill.enable_unsafe_...
Jacques Nadeau [Fri, 15 May 2015 01:09:11 +0000 (18:09 -0700)] 
Update configuration defaults.  Rename bounds system property to drill.enable_unsafe_memory_access.

7 years agoUpdate sys.options table to only show options available via ALTER SYSTEM or ALTER...
Jacques Nadeau [Thu, 14 May 2015 20:41:26 +0000 (13:41 -0700)] 
Update sys.options table to only show options available via ALTER SYSTEM or ALTER SESSION.  Move BOOT options to sys.boot table.

7 years agoIn the case of extended RPC message handling, write warning to log
Jacques Nadeau [Thu, 14 May 2015 19:19:46 +0000 (12:19 -0700)] 
In the case of extended RPC message handling, write warning to log

7 years agoUpdate Drill to use latest sqlline
Jacques Nadeau [Thu, 14 May 2015 16:16:41 +0000 (09:16 -0700)] 
Update Drill to use latest sqlline

7 years agoAdd convenience drill-* executables for sqlline. Rename drill_dumpcat to dumpcat
Jacques Nadeau [Thu, 14 May 2015 17:13:35 +0000 (10:13 -0700)] 
Add convenience drill-* executables for sqlline.  Rename drill_dumpcat to dumpcat

7 years agoDRILL-3072: Update root fragment to not to modify the Foreman state directly.
vkorukanti [Thu, 14 May 2015 02:42:24 +0000 (19:42 -0700)] 
DRILL-3072: Update root fragment to not to modify the Foreman state directly.

Instead use RPC mechanism to send and receive status updates

7 years agoDRILL-3063: TestQueriesOnLargeFile leaks memory with 16M limit
Chris Westin [Thu, 14 May 2015 22:53:05 +0000 (15:53 -0700)] 
DRILL-3063: TestQueriesOnLargeFile leaks memory with 16M limit
Changed the cleanup handling at the end of ImplCreator.getExec(), and
handle the newly returned null value in FragmentExecutor.run().

7 years agoDRILL-3061: Fix memory leaks in TestDrillbitResilience
adeneche [Tue, 12 May 2015 17:53:32 +0000 (10:53 -0700)] 
DRILL-3061: Fix memory leaks in TestDrillbitResilience

- fixes a race condition in WorkEventBus
-  marking TestDrillbitResilience with @Ignore

7 years agoAdd measurement of time taken to start a thread in timed runnable
Parth Chandra [Wed, 13 May 2015 22:42:06 +0000 (15:42 -0700)] 
Add measurement of time taken to start a thread in timed runnable

7 years agoDRILL-3065: Ensure the selection vectors in mSort to be closed after failure happens
Hsuan-Yi Chu [Thu, 14 May 2015 00:17:53 +0000 (17:17 -0700)] 
DRILL-3065: Ensure the selection vectors in mSort to be closed after failure happens

7 years agoDRILL-3093: Close raw batch buffers in data collectors
Mehant Baid [Fri, 15 May 2015 00:55:48 +0000 (17:55 -0700)] 
DRILL-3093: Close raw batch buffers in data collectors

7 years agoDRILL-2875: Show record number relative to beginning of file in JsonRecordReader
Steven Phillips [Thu, 14 May 2015 03:37:00 +0000 (20:37 -0700)] 
DRILL-2875: Show record number relative to beginning of file in JsonRecordReader

7 years agoDRILL-3088: Kill and cleanup remaing batches in left upstream in NestedLoopJoin
Steven Phillips [Thu, 14 May 2015 22:44:52 +0000 (15:44 -0700)] 
DRILL-3088: Kill and cleanup remaing batches in left upstream in NestedLoopJoin

7 years agoDRILL-3071: fix memory leak in RecordBatchLoader#load
Hanifi Gunes [Wed, 13 May 2015 23:36:41 +0000 (16:36 -0700)] 
DRILL-3071: fix memory leak in RecordBatchLoader#load

7 years agoDRILL-3053: add unchecked exception injection site in ChildAllocator
adeneche [Wed, 13 May 2015 16:45:23 +0000 (09:45 -0700)] 
DRILL-3053: add unchecked exception injection site in ChildAllocator

7 years agoDRILL-3080: Fix queue failure messages.
Jacques Nadeau [Thu, 14 May 2015 02:30:33 +0000 (19:30 -0700)] 
DRILL-3080: Fix queue failure messages.

7 years agoDRILL-3062: Rtrim the fixed char strings in the IN list for comparison with varchar...
Aman Sinha [Thu, 14 May 2015 01:31:33 +0000 (18:31 -0700)] 
DRILL-3062:  Rtrim the fixed char strings in the IN list for comparison with varchar columns.

7 years agoDRILL-3069: Fix StreamingAggBatch to appropriately set current index when it receives...
Mehant Baid [Thu, 14 May 2015 00:17:35 +0000 (17:17 -0700)] 
DRILL-3069: Fix StreamingAggBatch to appropriately set current index when it receives empty batches

7 years agoDRILL-3074: Fix infinite loop issue in ReconnectingConnection
vkorukanti [Thu, 14 May 2015 00:56:14 +0000 (17:56 -0700)] 
DRILL-3074: Fix infinite loop issue in ReconnectingConnection

Also use the remaining timeout incase of interruption

7 years agoDRILL-3079: Move execution fragment json parsing from RPC message to fragment start.
Jacques Nadeau [Wed, 13 May 2015 19:55:01 +0000 (12:55 -0700)] 
DRILL-3079: Move execution fragment json parsing from RPC message to fragment start.

7 years agoDRILL-2936: Use SpoolingRawBatchBuffer for HashToMergeExchange
Steven Phillips [Thu, 7 May 2015 03:01:35 +0000 (20:01 -0700)] 
DRILL-2936: Use SpoolingRawBatchBuffer for HashToMergeExchange
In order to avoid deadlocks

Refactored common code in UnlimitedRawBatchBuffer and SpoolingRawBatchBuffer
 into BaseRawBatchBuffer

Removed reflection-based construction of RawBatchBuffer. Now use choose implementation
 based on plan

Updated SpoolingRawBatchBuffer to use a separate thread for spooling

7 years agoDRILL-3046: Release data buffers when interrupt occurs in DataTunnel.sendRecordBatch
adeneche [Wed, 13 May 2015 18:48:23 +0000 (11:48 -0700)] 
DRILL-3046: Release data buffers when interrupt occurs in DataTunnel.sendRecordBatch

7 years agoDRILL-3037: Fix impersonation issue with HDFS based filesystem plugins.
vkorukanti [Wed, 13 May 2015 05:57:11 +0000 (22:57 -0700)] 
DRILL-3037: Fix impersonation issue with HDFS based filesystem plugins.

We have common code where we impersonate the owner of scan (proxy user) while creating
a filesystem object. When impersonation is enabled the proxy user is either query
user or view owner (if the query involves views). When impersonation is disabled the
proxy user is always the user who started Drillbit (processUser). This causes problems
as "processUser" tries to impersonate "processUser" when impersonation is disabled.
HDFS based filesystems which have impersonation disabled prohibit this (even the self
impersonation).

Fix is don't impersonate "processUser". If the given proxy user name is same as
"processUser", then return the process UserGroupInformation directly instead of
proxy UserGroupInformation.

7 years agoDRILL-3020: Copy cause exception's message to thrown SQLException's message.
dbarclay [Sun, 10 May 2015 05:19:49 +0000 (22:19 -0700)] 
DRILL-3020: Copy cause exception's message to thrown SQLException's message.

Changed SQLException construction to set SQLException's message to the cause
exception's toString().

Also:
- Narrowed one SQLException to SQLNonTransientConnectionException (not other
  two, since unclear whether transient, non-transient, or varied).
- Clarified/simplified exception messages.
- Fixed message typo.

7 years agoDRILL-2993: Unthrottle at cancel() to fix post-cancelation hangs.
dbarclay [Sun, 10 May 2015 05:17:35 +0000 (22:17 -0700)] 
DRILL-2993: Unthrottle at cancel() to fix post-cancelation hangs.

Added unthrottling in close().
Cleaned up throttling logic code:
- Applied AtomicBoolean to eliminate race conditions.
- Extracted methods for starting/stopping throttling.

Made small edits to some message:
- Fixed missed, inconsistent ResultsListener log messages.
- Clarified exception message.

7 years agoDRILL-2998: Implement heartbeat in C++ client
Parth Chandra [Sat, 9 May 2015 00:53:39 +0000 (17:53 -0700)] 
DRILL-2998: Implement heartbeat in C++ client

7 years agoDRILL-2750: Running 1 or more queries against Drillbits having insufficient DirectMem...
Chris Westin [Wed, 13 May 2015 04:00:57 +0000 (21:00 -0700)] 
DRILL-2750: Running 1 or more queries against Drillbits having insufficient DirectMem renders the Drillbits in an unusable state

Fix memory leaks found at operator tree construction time if there is an
allocation failure.

7 years agoDRILL-2229: For SqlParseException, SqlUnsupportedException, throw UserException
Hsuan-Yi Chu [Sun, 3 May 2015 17:29:47 +0000 (10:29 -0700)] 
DRILL-2229: For SqlParseException, SqlUnsupportedException, throw UserException

7 years agoDRILL-3009: Bump up Calcite version to 1.1.0-drill-r7 (For IN list threshold).
Aman Sinha [Wed, 13 May 2015 01:14:43 +0000 (18:14 -0700)] 
DRILL-3009: Bump up Calcite version to 1.1.0-drill-r7 (For IN list threshold).

7 years agoDRILL-2976: Part 2 - disable extended json in the default convert_toJSON method,...
Jason Altekruse [Fri, 8 May 2015 23:45:15 +0000 (16:45 -0700)] 
DRILL-2976: Part 2 - disable extended json in the default convert_toJSON method, the functionality is still accessible in a new convert_toComplexJSON method.

Added unit tests for both versions of the function (in the case of the simple json version I added result verification to an old test)

Address Mehant's review comments.

Fix failing unit test to turn on now disabled option for using extended types in written json files.

7 years agoDRILL-2976: Default extended types option for json writer is false
Sudheesh Katkam [Thu, 7 May 2015 00:09:35 +0000 (17:09 -0700)] 
DRILL-2976: Default extended types option for json writer is false

7 years agoDRILL-3051: Fix integer overflow in TimedRunnable
Steven Phillips [Wed, 13 May 2015 01:19:49 +0000 (18:19 -0700)] 
DRILL-3051: Fix integer overflow in TimedRunnable

also fixes a similar overflow in Foreman

7 years agoDRILL-3050: Increase QueryContext max allocation to 256MB
Steven Phillips [Wed, 13 May 2015 01:05:34 +0000 (18:05 -0700)] 
DRILL-3050: Increase QueryContext max allocation to 256MB

7 years agoDRILL-3049: Increase sort spooling threshold
Steven Phillips [Wed, 13 May 2015 00:55:54 +0000 (17:55 -0700)] 
DRILL-3049: Increase sort spooling threshold

7 years agoDRILL-3048: Disable assertions by default in distribution
Steven Phillips [Wed, 13 May 2015 00:49:12 +0000 (17:49 -0700)] 
DRILL-3048: Disable assertions by default in distribution

7 years agoDRILL-2755: (part2) Use and handle InterruptedException during query processing
vkorukanti [Tue, 12 May 2015 17:23:38 +0000 (10:23 -0700)] 
DRILL-2755: (part2) Use and handle InterruptedException during query processing

7 years agoDRILL-2780: Check for open files when closing OperatorContext
Steven Phillips [Sun, 10 May 2015 09:33:14 +0000 (02:33 -0700)] 
DRILL-2780: Check for open files when closing OperatorContext

7 years agoDRILL-2528 - make sure DrillResultSet is accurately preserved in drill-jdbc-all
Patrick Wong [Wed, 6 May 2015 00:49:20 +0000 (00:49 +0000)] 
DRILL-2528 - make sure DrillResultSet is accurately preserved in drill-jdbc-all

7 years agoDRILL-1662: Decrease drillbit kill -9 timeout in shutdown script to 120 seconds
rinukonda [Fri, 8 May 2015 20:42:38 +0000 (13:42 -0700)] 
DRILL-1662: Decrease drillbit kill -9 timeout in shutdown script to 120 seconds

7 years agoDRILL-3033: Add memory leak fixes found so far in DRILL-1942 to 1.0.0 Fixes some...
Chris Westin [Tue, 12 May 2015 05:06:30 +0000 (22:06 -0700)] 
DRILL-3033: Add memory leak fixes found so far in DRILL-1942 to 1.0.0 Fixes some missing buffer retains() and missing vector clears().

7 years agoDRILL-1980: Add support for CTAS with interval data type.
Mehant Baid [Mon, 11 May 2015 23:31:42 +0000 (16:31 -0700)] 
DRILL-1980: Add support for CTAS with interval data type.
Modify ParquetOutputRecordWriter to be able to write interval day, interval year and interval data type into parquet.
Modify the two parquet readers to be able to read interval data type from parquet file

Add unit tests

7 years agoDRILL-2953: Ensure sort operator be inserted to enforce desired order, when a cast...
Jinfeng Ni [Mon, 11 May 2015 22:39:24 +0000 (15:39 -0700)] 
DRILL-2953: Ensure sort operator be inserted to enforce desired order, when a cast function is involved.

7 years agoDRILL-2977, DRILL-2978: Swap fragment execution method implementations, and cancellat...
Sudheesh Katkam [Mon, 11 May 2015 19:02:35 +0000 (12:02 -0700)] 
DRILL-2977, DRILL-2978: Swap fragment execution method implementations, and cancellation changes

Execution: In WorkManager,
+ swap implementations of startFragmentPendingRemote() and addFragmentRunner()
+ warn if there are running fragments in close()

Cancellation:
+ for fragments waiting on data, delegate cancellations to WorkEventBus (in Foreman and ControlMessageHandler)
+ documentation

7 years agoDRILL-2476: Added BatchState.STOP in buildSchema() so AbstractRecordBatch returns...
Sudheesh Katkam [Wed, 6 May 2015 21:24:26 +0000 (14:24 -0700)] 
DRILL-2476: Added BatchState.STOP in buildSchema() so AbstractRecordBatch returns IterOutcome.STOP

7 years agoDRILL-3010: Convert bad command error messages into UserExceptions in SqlHandlers
vkorukanti [Sat, 18 Apr 2015 00:20:10 +0000 (17:20 -0700)] 
DRILL-3010: Convert bad command error messages into UserExceptions in SqlHandlers

7 years agoDRILL-2343: Add tracing proxy JDBC driver for tracing JDBC method calls.
dbarclay [Mon, 9 Feb 2015 18:52:11 +0000 (10:52 -0800)] 
DRILL-2343: Add tracing proxy JDBC driver for tracing JDBC method calls.

Overview of org.apache.drill.jdbc.proxy classes/interfaces:

Entry point:
- TracingProxyDriver - Tracing proxy JDBC driver class.  Class description
  Javadoc has usage instructions.

Other core types:
- ProxiesManager - creates and tracks java.lang.reflect.Proxy-based proxy
  objects
- TracingInvocationHandler - java.lang.reflect.Proxy invocation handler; maps
  reflective/proxy invocations to InvocationReporter method call/return/throw
  event calls
- InvocationReporter - defines method call/return/throw event calls
- InvocationReporterImpl - implements rendering of method call/return/throw
  event calls (including rendering of parameter/return/exception values)
- ProxySetupSQLException

Unit tests:
- TracingProxyDriverClassLoadingTest - test of loading proxied driver class
- TracingProxyDriverTest - basic test of proxying (pass-through) and tracing
  output

Other:
- exec/jdbc-all/pom.xml - has change to keep TracingProxyDriver and depended-on
  classes in JDBC-all Jar file.

7 years agoDRILL-2776: add extensive tests for empty population coverage
Hanifi Gunes [Mon, 11 May 2015 23:13:52 +0000 (16:13 -0700)] 
DRILL-2776: add extensive tests for empty population coverage

7 years agoDRILL-3027: Add convenience methods to test builder for creating nested baseline...
Hanifi Gunes [Mon, 11 May 2015 22:09:28 +0000 (15:09 -0700)] 
DRILL-3027: Add convenience methods to test builder for creating nested baseline values

7 years agoDRILL-3018: Fix CanNotPlan for nestLoop left join, caused by row count estimation...
Jinfeng Ni [Mon, 11 May 2015 16:02:17 +0000 (09:02 -0700)] 
DRILL-3018: Fix CanNotPlan for nestLoop left join, caused by row count estimation issue.

Add unit testcase.

7 years agoDRILL-2150: Create an abstraction for repeated value vectors.
Hanifi Gunes [Wed, 29 Apr 2015 21:54:24 +0000 (14:54 -0700)] 
DRILL-2150: Create an abstraction for repeated value vectors.

7 years agoDRILL-2961: Part 2--Core: Have set...Timeout methods throw rather than ignore.
dbarclay [Wed, 6 May 2015 06:12:32 +0000 (23:12 -0700)] 
DRILL-2961: Part 2--Core: Have set...Timeout methods throw rather than ignore.

Changed implementation of Statement.setQueryTimeout(...) and
Connection.setNetworkTimeout(...) to throw SQLFeatureNotSupportedException
rather than silently not setting timeout.

Main:
- Added implementations of Statement.{set,get}QueryTimeout(...) and
  Connection.{set,get}NetworkTimeout(...) methods (in classes
  DrillStatementImpl, DrillConnectionImpl).
- Added Drill-specific narrowed declarations and documentation (in
  interfaces DrillStatement and DrillConnection).
- Added unit tests (in new StatementTest, ConnectionTest).

Narrowed other thrown exceptions to allow narrowing in above methods:
- Narrowed exceptions on isClosed() and checkNotClosed() (in
  DrillConnectionImpl, DrillStatementImpl).
- Added InvalidParameterSqlException.

7 years agoDRILL-2961: Part 1--Prep., Hyg.: Split DrillStatement -> DrillStatement{,Impl}.
dbarclay [Wed, 6 May 2015 05:39:25 +0000 (22:39 -0700)] 
DRILL-2961: Part 1--Prep., Hyg.: Split DrillStatement -> DrillStatement{,Impl}.

Preparation for query/network timeout methods:
- Moved class ...jdbc.DrillStatement to ...jdbc.impl.DrillStatementImpl.
- Created interface ...jdbc.DrillStatement interface, extending
  java.sql.Statement (for place for Drill-specific narrowed declarations and
  documentation).  Declared DrillStatementImpl to implement new DrillStatement.
- Updated references.
- Adjusted visibility (temporarily make somethings publich until DRILL-2089 is
  done more).

Misc. code hygiene:
- Purged extraneous imports of ResultSet.
- Purged TODO made obsolete by UserException.
- Misc. whitespace.

7 years agoDRILL-3022: ensure sequential shutdown of drillbits
Hanifi Gunes [Mon, 11 May 2015 06:37:20 +0000 (23:37 -0700)] 
DRILL-3022: ensure sequential shutdown of drillbits

7 years agoDRILL-3017: Safeguard against NPEs in RecordReader.cleanup()s
vkorukanti [Sun, 10 May 2015 22:31:58 +0000 (15:31 -0700)] 
DRILL-3017: Safeguard against NPEs in RecordReader.cleanup()s

7 years agoAdd missing commit fixing queries file path.
Jacques Nadeau [Sun, 10 May 2015 21:48:25 +0000 (14:48 -0700)] 
Add missing commit fixing queries file path.

7 years agoDRILL-1927, DRILL-2966: Issues were resolved in Calcite update. Test cases added
Hsuan-Yi Chu [Mon, 23 Mar 2015 23:14:38 +0000 (16:14 -0700)] 
DRILL-1927, DRILL-2966: Issues were resolved in Calcite update. Test cases added

7 years agoDRILL-2755: Use and handle InterruptedException during query processing.
vkorukanti [Wed, 6 May 2015 16:34:25 +0000 (09:34 -0700)] 
DRILL-2755: Use and handle InterruptedException during query processing.

- Interrupt FragmentExecutor thread as part of FragmentExecutor.cancel()
- Handle InterruptedException in ExternalSortBatch.newSV2(). If the fragment status says
  should not continue, then throw the InterruptedException to caller which returns IterOutcome.STOP
- Add comments reg not handling of InterruptedException in SendingAccountor.waitForSendComplete()
- Handle InterruptedException in OrderedPartitionRecordBatch.getPartitionVectors()
  If interrupted in Thread.sleep calls and fragment status says should not run, then
  return IterOutcome.STOP downstream.
- Interrupt partitioner threads if PartitionerRecordBatch is interrupted while waiting for
  partitioner threads to complete.
- Preserve interrupt status if not handled
- Handle null RecordBatches returned by RawBatchBuffer.getNext() in MergingRecordBatch.buildSchema()
- Change timeout in Foreman to be proportional to the number of intermediate fragments sent instead
  of hard coded limit of 90s.
- Change TimedRunnable to enforce a timeout of 15s per runnable.
  Total timeout is (5s * numOfRunnableTasks) / parallelism.
- Add unit tests
  * Testing cancelling a query interrupts the query fragments which are currently blocked
  * Testing interrupting the partitioner sender which in turn interrupts its helper threads
  * Testing TimedRunanble enforeces timeout for the whole task list.

7 years agoDRILL-2697: Pauses sites wait indefinitely for a resume signal DrillClient sends...
Sudheesh Katkam [Thu, 30 Apr 2015 20:27:08 +0000 (13:27 -0700)] 
DRILL-2697: Pauses sites wait indefinitely for a resume signal DrillClient sends a resume signal to UserServer. UserServer triggers a resume call in the correct Foreman. Foreman resumes all pauses related to the query through the Control layer.

+ Better error messages and more tests in TestDrillbitResilience and TestPauseInjection
+ Added execution controls to operator context
+ Removed ControlMessageHandler interface, renamed ControlHandlerImpl to ControlMessageHandler
+ Added CountDownLatchInjection, useful in cases like ParititionedSender that spawns multiple threads

7 years agoDRILL-2870: Part2 - Fix return type of aggregate functions to be nullable
Mehant Baid [Sat, 9 May 2015 20:36:05 +0000 (13:36 -0700)] 
DRILL-2870: Part2 - Fix return type of aggregate functions to be nullable

7 years agoDRILL-2870: Part1 - Separate count aggregate functions from others
Mehant Baid [Sat, 9 May 2015 19:31:36 +0000 (12:31 -0700)] 
DRILL-2870: Part1 - Separate count aggregate functions from others
Count is the only aggregate function with required type as output, this will reduce some of the conditional logic in the templates

7 years agoDRILL-3012: Fix issue where remote values rel wasn't losing operatorId.
Jacques Nadeau [Sun, 10 May 2015 14:46:54 +0000 (07:46 -0700)] 
DRILL-3012: Fix issue where remote values rel wasn't losing operatorId.

Also enhance Union rule to avoid more than 2 ways inputs

7 years agoDRILL-2841: Fix slow web UI on some cluster nodes due to unneeded transitive dependencies
Jason Altekruse [Sun, 10 May 2015 02:07:16 +0000 (19:07 -0700)] 
DRILL-2841: Fix slow web UI on some cluster nodes due to unneeded transitive dependencies

Fix enforcer rule for jetty servlet-api global exclusion, add explicit exclusions where needed.

7 years agoDRILL-3007: Disable automatic mmap threshold on Linux
Jacques Nadeau [Sun, 10 May 2015 00:17:56 +0000 (17:17 -0700)] 
DRILL-3007: Disable automatic mmap threshold on Linux

7 years agoDRILL-2981: Add queries log. Update profile to store normal and verbose exception...
Jacques Nadeau [Fri, 8 May 2015 15:16:27 +0000 (08:16 -0700)] 
DRILL-2981: Add queries log.  Update profile to store normal and verbose exception as well as node and errorid.

7 years agoDRILL-2723: Add option to change text format row size estimate.
Jacques Nadeau [Sat, 9 May 2015 18:20:47 +0000 (11:20 -0700)] 
DRILL-2723: Add option to change text format row size estimate.

7 years agoDRILL-3005: Ensure that only one outcome can occur for a QueryResult.
Jacques Nadeau [Sat, 9 May 2015 17:44:45 +0000 (10:44 -0700)] 
DRILL-3005: Ensure that only one outcome can occur for a QueryResult.

Includes removing QueryResultHandler close listener which should fix most times.  For exceptional race cases, also added isTerminal condition which ensures only a single transition.

7 years agoDRILL-2878: FragmentExecutor.closeOutResources() is not called if an exception happen...
adeneche [Wed, 6 May 2015 03:59:14 +0000 (20:59 -0700)] 
DRILL-2878: FragmentExecutor.closeOutResources() is not called if an exception happens in the Foreman before the fragment executor starts running

7 years agoDRILL-2757: Verify operators correctly handle low memory conditions and cancellations
adeneche [Wed, 29 Apr 2015 15:29:04 +0000 (08:29 -0700)] 
DRILL-2757: Verify operators correctly handle low memory conditions and cancellations

includes:
DRILL-2816: system error does not display the original Exception message
DRILL-2893: ScanBatch throws a NullPointerException instead of returning OUT_OF_MEMORY
DRILL-2894: FixedValueVectors shouldn't set it's data buffer to null when it fails to allocate it
DRILL-2895: AbstractRecordBatch.buildSchema() should properly handle OUT_OF_MEMORY outcome
DRILL-2905: RootExec implementations should properly handle IterOutcome.OUT_OF_MEMORY
DRILL-2920: properly handle OutOfMemoryException
DRILL-2947: AllocationHelper.allocateNew() doesn't have a consistent behavior when it can't allocate

also:
- added UserException.memoryError() with a pre assigned error message
- injection site in ScanBatch and unit test that runs various tpch queries and injects
  an exception in the ScanBatch that will cause an OUT_OF_MEMORY outcome to be sent

7 years agoAddress review comments. Modify costing for Cartesian Logical Join, to fix CanNotPlan...
Jinfeng Ni [Sat, 9 May 2015 22:29:00 +0000 (15:29 -0700)] 
Address review comments. Modify costing for Cartesian Logical Join, to fix CanNotPlan in some TPCDS queries with view on text format.

Code clean up.

7 years agoDRILL-2958: Move Drill to alternative cost-based planner for join planning.
Jinfeng Ni [Wed, 29 Apr 2015 18:16:04 +0000 (11:16 -0700)] 
DRILL-2958: Move Drill to alternative cost-based planner for join planning.

1. Add a new RelMdDistinctRowCount MetaDataProvider.

2. Resolve issue related to RelMetadataProvider, when querying view (Since view expander will create a different RelOptCluster).

3. Catch CanNotPlanException in Physical Planning as well, since we allow DrillJoinRel to be Cartesian Join, which then pass into LOPT planner.

4. Return none-infinite cost for Cartesian DrillJoinRel.

5. Ruleset change. Only use Join permutation (PushJoinPastThroughJoin) rule in the existing planner. LOPT planner will not them.

Code clean up.

7 years agoDRILL-2532: Fix glob implementation
AdamPD [Tue, 24 Mar 2015 02:33:14 +0000 (12:33 +1000)] 
DRILL-2532: Fix glob implementation

7 years agoDRILL-2036: Make field name in MapWriter case-insensitive
Steven Phillips [Tue, 28 Apr 2015 23:32:58 +0000 (16:32 -0700)] 
DRILL-2036: Make field name in MapWriter case-insensitive

When looking up the vectors in the container vector, we already treat
the field name as case insensitive. But we were not treating the field names
in MapWriter when looking up the FieldWriter as case-insensitive. This
inconsistency resulted in incorrect data. This patch makes the behavior
consistent

7 years ago[DRILL-2914] Update DrillCalcite's version to 1.1.0-drill-r5
Hsuan-Yi Chu [Thu, 7 May 2015 18:57:47 +0000 (11:57 -0700)] 
[DRILL-2914] Update DrillCalcite's version to 1.1.0-drill-r5

7 years agoDRILL-2897: Avoid parallelization when Drill is running limit 0 query.
Jacques Nadeau [Mon, 27 Apr 2015 05:35:57 +0000 (22:35 -0700)] 
DRILL-2897: Avoid parallelization when Drill is running limit 0 query.

7 years agoDRILL-2971, DRILL-2886, DRILL-2778, DRILL-2545: Improve RPC connection detection...
Jacques Nadeau [Fri, 8 May 2015 15:15:36 +0000 (08:15 -0700)] 
DRILL-2971, DRILL-2886, DRILL-2778, DRILL-2545: Improve RPC connection detection failure.  Add RPC timeout.

7 years agoDRILL-2960: Add back hive boostrap-storage-plugin to make it show up in a default...
Jason Altekruse [Thu, 7 May 2015 17:05:30 +0000 (10:05 -0700)] 
DRILL-2960: Add back hive boostrap-storage-plugin to make it show up in a default Drill install as disabled.

7 years agoDRILL-2602: Throw an error on schema change during streaming aggregation
adeneche [Tue, 5 May 2015 20:40:41 +0000 (13:40 -0700)] 
DRILL-2602: Throw an error on schema change during streaming aggregation

7 years agoDRILL-2570 - exclude "META-INF/services" dir from jdbc-all
Patrick Wong [Thu, 9 Apr 2015 22:42:55 +0000 (15:42 -0700)] 
DRILL-2570 - exclude "META-INF/services" dir from jdbc-all

7 years agoDRIL-2994: Incorrect error message when disconnecting from a direct drillbit connection
Parth Chandra [Fri, 8 May 2015 18:38:49 +0000 (11:38 -0700)] 
DRIL-2994: Incorrect error message when disconnecting from a direct drillbit connection

7 years agoDRILL-2848: Provide option to disable decimal data type
Mehant Baid [Fri, 1 May 2015 00:02:18 +0000 (17:02 -0700)] 
DRILL-2848: Provide option to disable decimal data type
Disable casting to decimal
Disable reading decimal from Parquet
Disable reading decimal from Hive
Add unit tests
Modify existing tests to enable decimal data type

7 years agoDRILL-2848: Hygiene - ParquetToDrillConverter cleanup
Mehant Baid [Fri, 1 May 2015 21:12:09 +0000 (14:12 -0700)] 
DRILL-2848: Hygiene - ParquetToDrillConverter cleanup

7 years agoDRILL-2809: Increase the default value of partitioner_sender_threads_factor.
Aman Sinha [Thu, 30 Apr 2015 03:34:04 +0000 (20:34 -0700)] 
DRILL-2809: Increase the default value of partitioner_sender_threads_factor.

7 years agoDRILL-2376: In UnionAllRecordBactch, the mechansim to detect schema change is corrected
Hsuan-Yi Chu [Thu, 23 Apr 2015 20:44:13 +0000 (13:44 -0700)] 
DRILL-2376: In UnionAllRecordBactch, the mechansim to detect schema change is corrected

7 years agoDRILL-2408 (part 2): CTAS should not create empty folders when underlying query retur...
adeneche [Mon, 20 Apr 2015 20:03:07 +0000 (13:03 -0700)] 
DRILL-2408 (part 2): CTAS should not create empty folders when underlying query returns no results

- changed ParquetRecordWriter to avoid creating the parquet file until the first row of data is available
- Moved unit tests in a separate test class that starts 3 drillbits, to test the case where multiple fragments are attempting to write empty parquet files
- changed BaseQueryTest to update the storage plugin in all started bits and not just the first one