drill.git
7 years agoDRILL-2575: FragmentExecutor.cancel() blasts through state transitions regardless... 0.8.0 drill-0.8.0
Chris Westin [Thu, 26 Mar 2015 02:05:25 +0000 (19:05 -0700)] 
DRILL-2575: FragmentExecutor.cancel() blasts through state transitions regardless of current state

FragmentExecutor:
- Changed cancel() to behave asynchronously, and for the cancelation request to
  be checked at an appropriate place in the run() loop.

7 years agoDRILL-2574: SendingAccountor can suffer from lost updates
Chris Westin [Thu, 26 Mar 2015 01:44:41 +0000 (18:44 -0700)] 
DRILL-2574: SendingAccountor can suffer from lost updates

SendingAccountor
- atomically get and set the message count to wait for

7 years agoDRILL-2547: Don't allow Drill to shut down while queries are still executing
Chris Westin [Tue, 24 Mar 2015 23:21:33 +0000 (16:21 -0700)] 
DRILL-2547: Don't allow Drill to shut down while queries are still executing
This will cause Drillbit.close() to block until all currently executing
fragments have completed.

WorkManager
- added waitForExit() and indicateIfSafeToExit(), which use a latch to
  wait to shut down if there are active fragments
  - waitForExit() times out after 5 seconds

Drillbit
- call WorkManager.waitForExit() in close()

7 years agoDRILL-2502: Improve code safety by providing a generic event delivery mechan
Chris Westin [Tue, 24 Mar 2015 23:20:06 +0000 (16:20 -0700)] 
DRILL-2502: Improve code safety by providing a generic event delivery mechan
This fixes some problems with broken state transitions that happen under
queryManager.cancelExecutingFragments(), which cause recursive entry into
Foreman.moveToState().

EventProcessor
- created
Foreman
- altered to use the EventProcessor for moveToState()

7 years agoDRILL-2520: Foreman is being removed from the running query table prematurely
Chris Westin [Mon, 23 Mar 2015 20:31:56 +0000 (13:31 -0700)] 
DRILL-2520: Foreman is being removed from the running query table prematurely
WorkManager:
- reinstate retireForeman() on WorkerBee
- don't use SelfCleaningRunnable to remove Foreman from running query list

Foreman:
- use retireForeman() to remove self from running query list during cleanup

7 years agoDRILL-2567: CONVERT_FROM in where clause cause the query to fail in planning phase
Aditya Kishore [Wed, 25 Mar 2015 22:00:54 +0000 (15:00 -0700)] 
DRILL-2567: CONVERT_FROM in where clause cause the query to fail in planning phase

Set the writeIndex of ByteBuf returned by Unpooled.wrappedBuffer() to 0.

+ Added a unit test to exercise the code path.

7 years agoDRILL-2572: Use PrelUtil to get PlannerSettings for PruneScanRule.
Sudheesh Katkam [Thu, 26 Mar 2015 00:18:57 +0000 (17:18 -0700)] 
DRILL-2572: Use PrelUtil to get PlannerSettings for PruneScanRule.

+ context.getPlannerSettings() returns null sometimes
+ introduced in commit 48c9c01

7 years agoDRILL-2446: Improvement in finding Drill log dir
Patrick Wong [Thu, 12 Mar 2015 21:48:05 +0000 (14:48 -0700)] 
DRILL-2446: Improvement in finding Drill log dir

7 years agoDRILL-2541: Plan fragment logging causes significant delay in query start
Chris Westin [Tue, 24 Mar 2015 20:02:10 +0000 (13:02 -0700)] 
DRILL-2541: Plan fragment logging causes significant delay in query start

Foreman
- change plan fragment logging from INFO level to TRACE level

7 years agoDRILL-2537: Set a maximum source code size of 2mb for scalar replacement.
Jacques Nadeau [Tue, 24 Mar 2015 17:46:43 +0000 (10:46 -0700)] 
DRILL-2537: Set a maximum source code size of 2mb for scalar replacement.

7 years agoDRILL-2510: Fix unclosed allocators detected in Java shutdown hook
Chris Westin [Fri, 20 Mar 2015 19:29:29 +0000 (12:29 -0700)] 
DRILL-2510: Fix unclosed allocators detected in Java shutdown hook

TestWriteToDisk
- fix warnings, including closing unclosed resources with try-with-resources

TestAllocators
- close unclosed resources
  doesn't use try-with-resources because this stresses interleaved allocation
  operations that wouldn't necessarily be nested

StackTrace
- created convenience class for tracking stack traces

Drillbit
- used StackTrace to track location of Drillbit starts for shutdown hook
  error messages so we can more easily identify errant tests from just the logs
  in future
- added an isClosed flag to detect that the Drillbit has already been closed
  to avoid complaints about double closures that happen in many tests that
  explicitly close their drillbits (the shutdown hook then closes them again,
  causing bogus complaints)
- synchronized close()
- made the id generator in the ShutdownThread an AtomicInteger

7 years agoDRILL-2402: Update hash functions to use seed strategy as opposed to xor strategy.
Jacques Nadeau [Thu, 5 Mar 2015 03:47:05 +0000 (19:47 -0800)] 
DRILL-2402: Update hash functions to use seed strategy as opposed to xor strategy.

Also: Simplify and consolidate expression materialization.

7 years agoDRILL-2010: MergeJoin: Store/restore the right batch state when existing join loop...
vkorukanti [Fri, 6 Mar 2015 14:58:07 +0000 (06:58 -0800)] 
DRILL-2010: MergeJoin: Store/restore the right batch state when existing join loop due to output batch full.

7 years agoDRILL-2342: Store the nullability property of column in view persistence store.
vkorukanti [Wed, 18 Mar 2015 00:38:55 +0000 (17:38 -0700)] 
DRILL-2342: Store the nullability property of column in view persistence store.

7 years agoDRILL-1735: Have closing of JDBC connection free embedded-server resources.
dbarclay [Tue, 16 Dec 2014 22:05:11 +0000 (14:05 -0800)] 
DRILL-1735: Have closing of JDBC connection free embedded-server resources.

Hooked up closing of JDBC connection to shut down embedded Drillbit, and then
fixed chain of bugs exposed by that:
 1. Added test org.apache.drill.jdbc.test.Bug1735ConnectionCloseTest.
 2. Hooked up connection handler in Driver to actually close JDBC connection.
 3. Released a QueryResultsBatch in DrillCursor.
 4. Reset DrillMetrics in BootStrapContext.close() (so stopping local DrillBit
    and starting new DrillBit doesn't yield "duplicate metric" error.)
 5. Checked cursor/row state before trying to retrieve value in DrillResultSet's
    column accessor methods.
    - Added org.apache.drill.jdbc.JdbcApiSqlException (for JDBC-level errors).
    - Added org.apache.drill.jdbc.InvalidCursorStateSqlException.
    [SqlAccessor, AvaticaDrillSqlAccessor, DrillConnectionImpl, DrillCursor,
    DrillResultSet, InvalidCursorStateSqlException, JdbcApiSqlException,
    DrillResultSetTest]
 6. Released vectors in DrillResultSet.cleanup().
    Added org.apache.drill.jdbc.test.Bug1735ResultSetCloseReleasesBuffersTest.
 7. Delayed last-chunk batch until COMPLETED batch in QueryResultHandler.
 8. Added nextUntilEnd(...) workarounds for fragment cancelation race condition
    to TestView and other JDBC module/subproject tests.
 9. Tracked open statements in orer to close at connection close (DrillStatementRegistry, etc.)
10. Commented out nextUntilEnd(...) workarounds for fragment cancelation race
    condition.
11 Miscellaneous:
   - Added some toString() methods
   - Adjusted some logging (e.g., "// log.debug(...)" -> "log.trace(...)".
   - Cleaned up a bit.  [DrillCursor, DrillResultSet, QueryResultHandler]
   - Added a few documentation comments.
   - Added various TODO comments.

7 years agoDRILL-2353: Add interpreter based partition pruning.
Aman Sinha [Mon, 16 Feb 2015 07:04:50 +0000 (23:04 -0800)] 
DRILL-2353: Add interpreter based partition pruning.

Integrate Jacques's interpreter based partition pruning with Jason's interpreter refactoring that removed interpreter module and added that functionality in the exec module.

Ensure boolean operators are correctly handled when traversing expression tree to find partition filters.

Resolve merge conflicts after rebasing to master branch.

Additional fixes for handling OR conditions.

7 years agoDRILL-2503: AsmUtil ClassTransformer MergeAdapter - add option to pass through ClassR...
Chris Westin [Fri, 20 Mar 2015 00:13:35 +0000 (17:13 -0700)] 
DRILL-2503: AsmUtil ClassTransformer MergeAdapter - add option to pass through ClassReader.EXPAND_FRAMES to satisfy complaint from ASM - rationalize AsmUtils methods' argument lists

TestBugFixes (in ...drill.jdbc.test)
- created this to hold random bug fix tests

7 years agoDRILL-1967: Add null checks to resources in cleanup of Parquet writer that may not...
Jason Altekruse [Fri, 20 Mar 2015 02:50:01 +0000 (19:50 -0700)] 
DRILL-1967: Add null checks to resources in cleanup of Parquet writer that may not be initialized if no schema has been provided to it by the time it is cleaned up.

7 years agoDRILL-2488: Return DEFAULT as supported encoding for MergeJoin since it does not...
Aman Sinha [Wed, 18 Mar 2015 02:23:45 +0000 (19:23 -0700)] 
DRILL-2488: Return DEFAULT as supported encoding for MergeJoin since it does not currently handle SV2 or SV4.

7 years agoDRILL-2499: Remove orphan test Drill2130InterpreterHamcrestConfigurationTest
Aditya Kishore [Thu, 19 Mar 2015 03:25:16 +0000 (20:25 -0700)] 
DRILL-2499: Remove orphan test Drill2130InterpreterHamcrestConfigurationTest

7 years agoDRILL-2245: Clean up query setup and execution kickoff in Foreman/WorkManager in...
Chris Westin [Wed, 25 Feb 2015 18:59:36 +0000 (10:59 -0800)] 
DRILL-2245: Clean up query setup and execution kickoff in Foreman/WorkManager in order to ensure consistent handling, and avoid hangs and races, with the goal of improving Drillbit robustness.

I did my best to keep these clean when I split them up, but this core commit
may depend on some minor changes in the hygiene commit that is also
associated with this bug, so either both should be applied, or neither.
The core commit should be applied first.

protocol/pom.xml
- updated protocol buffer compiler version to 2.6
- this made slight modifications to the formats of a few committed protobuf
  files

AutoCloseables
- created org.apache.drill.common.AutoCloseables to handle closing these
  quietly

BaseTestQuery, and derivatives
- factored out pieces into QueryTestUtil so they can be reused

DeferredException:
- created this so we can collect exceptions during the shutdown process

Drillbit
- uses AutoCloseables for the WorkManager and for the storeProvider
- allow start() to take a RemoteServiceSet
- private, final, formatting

Foreman
- added new state CANCELLATION_REQUESTED (via UserBitShared.proto) to represent
  the time between request of a cancellation, and acknowledgement from all
  remote endpoints running fragments on a query's behalf
- created ForemanResult to manage interleaving cleanup effects/failure with
  query result state
- does not need to implement Comparable
- does not need to implement Closeable
- thread blocking fixes
- add resultSent flag
- add code to log plan fragments with endpoint assignments
- added finals, cleaned up formatting
- do queue management in acquireQuerySemaphore; local tests pass
- rename getContext() to getQueryContext()
- retain DrillbitContext
- a couple of exception injections for testing
- minor formatting
- TODOs

FragmentContext
- added a DeferredException to collect errors during startup/shutdown sequences

FragmentExecutor
- eliminated CancelableQuery
- use the FragmentContext's DeferredException for errors
- common subexpression elimination
- cleaned up

QueryContext
- removed unnecessary functions (with some outside classes tweaked for this)
- finals, formatting

QueryManager
- merge in QueryStatus
  - affects Foreman, ../batch/ControlHandlerImpl,
    and ../../server/rest/ProfileResources
- made some methods private
- removed unused imports
- add finals and formatting
- variable renaming to improve readability
- formatting
- comments
- TODOs

QueryStatus
- getAsInfo() private
- member renaming
- member access changes
- formatting
- TODOs

QueryTestUtil, BaseTestQuery, TestDrillbitResilience
- make maxWidth a parameter to server startup

SelfCleaningRunnable
- created org.apache.drill.common.SelfCleaningRunnable

SingleRowListener
- created org.apache.drill.SingleRowListener results listener
- use in TestDrillbitResilience

TestComparisonFunctions
- fix not to close the FragmentContext multiple times

TestDrillbitResilience
- created org.apache.drill.exec.server.TestDrillbitResilience to test drillbit
  resilience in the face of exceptions and failures during queries

TestWithZookeeper
- factor out work into ZookeeperHelper so that it can be reused by
  TestDrillbitResilience

UserBitShared
- get rid of unused UNKNOWN_QUERY

WorkEventBus
- rename methods, affects Foreman and ControlHandlerImpl
- remove unused WorkerBee reference
- most members final
- formatting

WorkManager
- Closeable to AutoCloseable
- removed unused incomingFragments Set
- eliminated unnecessary eventThread and pendingTasks by posting Runnables
  directly to executor
- use SelfCleaningRunnable for Foreman management
- FragmentExecutor management uses SelfCleaningRunnable
- runningFragments to be a ConcurrentHashMap; TestTpchDistributed passes
- other improvements due to bee no longer needed in various places
- most members final
- minor formatting
- comments
- TODOs

(*) Created exception injection classes to simulate exceptions for testing
- ExceptionInjection
- ExceptionInjector
- ExceptionInjectionUtil
- TestExceptionInjection

DRILL-2245-hygiene: General code cleanup encountered while working on the rest
of this commit. This includes
- making members final whenever possible
- making members private whenever possible
- making loggers private
- removing unused imports
- removing unused private functions
- removing unused public functions
- removing unused local variables
- removing unused private members
- deleting unused files
- cleaning up formatting
  - adding spaces before braces in conditionals and loop bodies
  - breaking up overly long lines
  - removing extra blank lines

While I tried to keep this clean, this commit may have minor dependencies on
DRILL-2245-core that I missed. The intention is just to break this up for
review purposes. Either both commits should be applied, or neither.

7 years agoDRILL-2415: Export Drill C++ Client symbols so as to provide dynamic linking
alzarei [Thu, 19 Mar 2015 03:07:01 +0000 (20:07 -0700)] 
DRILL-2415: Export Drill C++ Client symbols so as to provide dynamic linking

7 years agoDRILL-2497: Remove reference to drill-interpreter in build
Patrick Wong [Wed, 18 Mar 2015 22:23:43 +0000 (15:23 -0700)] 
DRILL-2497: Remove reference to drill-interpreter in build

7 years agoDRILL-2210 Introducing multithreading capability to PartitonerSender
Yuliya Feldman [Tue, 17 Feb 2015 08:09:07 +0000 (00:09 -0800)] 
DRILL-2210 Introducing multithreading capability to PartitonerSender

7 years agoDRILL-2358: Ensure DrillScanRel differentiates skip-all, scan-all & scan-some in...
Hanifi Gunes [Thu, 5 Mar 2015 21:35:48 +0000 (13:35 -0800)] 
DRILL-2358: Ensure DrillScanRel differentiates skip-all, scan-all & scan-some in a backward compatible fashion

7 years agoDRILL-2309: Fix reduce aggregate rule to create new aggregate calls based on input...
Mehant Baid [Wed, 18 Mar 2015 02:45:38 +0000 (19:45 -0700)] 
DRILL-2309: Fix reduce aggregate rule to create new aggregate calls based on input row type

7 years agoDRILL-2180: Star column is enabled to work along with complex expression
Hsuan-Yi Chu [Mon, 16 Mar 2015 19:32:58 +0000 (12:32 -0700)] 
DRILL-2180: Star column is enabled to work along with complex expression

7 years agoDRILL-2106: Fix SplitUpComplexExpression rule to correctly detect last used column...
Mehant Baid [Sat, 14 Mar 2015 19:20:30 +0000 (12:20 -0700)] 
DRILL-2106: Fix SplitUpComplexExpression rule to correctly detect last used column reference in the project expression

7 years agoDRILL-2389: Remove timestamp with time zone
Mehant Baid [Wed, 18 Mar 2015 01:07:53 +0000 (18:07 -0700)] 
DRILL-2389: Remove timestamp with time zone

7 years agoDRILL-2279: Raise exception if schema change is encountered in hash and streaming...
Mehant Baid [Thu, 19 Feb 2015 02:37:39 +0000 (18:37 -0800)] 
DRILL-2279: Raise exception if schema change is encountered in hash and streaming aggregate

7 years agoDRILL-1757 Support for wildcards in repeated_contains()
vsowrirajan [Thu, 19 Feb 2015 20:04:52 +0000 (12:04 -0800)] 
DRILL-1757 Support for wildcards in repeated_contains()

7 years agoDRILL-2491: Fix use of injectable QueryDateTimeInfo in localtimestamp function
Mehant Baid [Wed, 18 Mar 2015 01:02:03 +0000 (18:02 -0700)] 
DRILL-2491: Fix use of injectable QueryDateTimeInfo in localtimestamp function

7 years agoDRILL-2406: part 2 - Allow interpreted expression evaluation at planning time.
Jason Altekruse [Mon, 9 Mar 2015 18:50:17 +0000 (11:50 -0700)] 
DRILL-2406: part 2 - Allow interpreted expression evaluation at planning time.

Changes needed after rebase to expose function determinism to calcite appropriately.

Address Jacques review comments.

Address chris' review comments.

Make things work now that BufferManager is AutoClosable.

Fixes tests that were creating plan fragments directly to create their own query start time,
as this information is now passed along from QueryContext during standard query initialization
(this enables the query start time and timezone to be available to planning time expression
evaluation).

Fix docs in BufferManger.

Update UDF interface to track determinism rather than randomness.

7 years agoDRILL-2406: part 1 - Remove interpreter generation, add new reflection based expressi...
Jason Altekruse [Tue, 3 Mar 2015 23:58:04 +0000 (15:58 -0800)] 
DRILL-2406: part 1 - Remove interpreter generation, add new reflection based expression interpretation.

Changed interpreted evaluation to run the setup method after the input parameters have been set using reflection
so they are available in the case where the inputs are constant and are used in the setup method.

Changes that were originally committed in later patches for 2060 and 2173, as they are needed for this to run on its own:
    - Change to DrillSimpleFuncHolder that belongs with the interpreter refactoring.
    - ValueHolderHelper changes needed for interpreter refactoring.

Updates after review comments from Jinfeng:
Change the DrillSimpleFunc creation in the new interpreter to use a direct reference to the class type of the Function rather than the class name.

Add test case for 'like' function evaluated in the interpreter (has a meaningful setup method that uses one of the inputs to initialize a pattern matcher)

Update from Aman's review:
Add a test case for using a cast in interpreted expression evaluation.

7 years agoDRILL-2143: Part 2 - fix interpreter and add QueryDateTimeInfo injectable to fill...
Jason Altekruse [Tue, 3 Mar 2015 23:44:32 +0000 (15:44 -0800)] 
DRILL-2143: Part 2 - fix interpreter and add QueryDateTimeInfo injectable to fill in holes from removing RecordBatch from UDFs - Move list of accepted injectable types into the UdfUtilities interface for easier maintenance.

Fix one more new function to remove RecordBatch from setup method.

Add back UdfUtilities interface to FragmentContext lost in rebasing.

Remove unneeded testcase in TestExampleQueries, remove commented out code in InterpreterGenerator
previously used to handle DrillBuf injectables, now being handled by reflection to set the DrillBuf
rather than the previously generated code that would use the direct reference to the incoming
RecordBatch to get a buffer.

Fix docs on UdfUtilities and package docs for drill/exec/expr, belongs with 2143 part 2 patch.

Fix belongs with 2143, change interpreter to output a ValueHolder instead of a ValueVector in the case of a constant expression.

2143 update - Use reflection to remove boilerplate for adding new injectable types for UDFs.

7 years agoDRILL-2143: Part 1 - just remove RecordBatch from UDF setup method.
Jason Altekruse [Fri, 6 Feb 2015 18:43:38 +0000 (10:43 -0800)] 
DRILL-2143: Part 1 - just remove RecordBatch from UDF setup method.

Disable date/time functions as they can no longer access incoming recordbatch to get query start time or timezone.
During rebase DateTypeFunctions.CurrentDate was updated to include a commented out vesion of the fix from DRILL-2372,
this will be fixed in the new patch for part 2 of 2143.

Remove RecordBatch from setup of a new new UDFs.

7 years agoDRILL-2311: In ProjectRecordBatch, even if a column from incoming recordbatch does...
Hsuan-Yi Chu [Sun, 15 Mar 2015 18:43:42 +0000 (11:43 -0700)] 
DRILL-2311: In ProjectRecordBatch, even if a column from incoming recordbatch does not need to be classified, the output name for this column is still ensured to be unique

7 years ago DRILL-2414: Give proper error message if Union-All is applied on schema-less tables
Hsuan-Yi Chu [Tue, 10 Mar 2015 05:50:59 +0000 (22:50 -0700)] 
 DRILL-2414: Give proper error message if Union-All is applied on schema-less tables

7 years agoDRILL-2441: For outer-join, if there is any inequality condition, Cartesian-Join...
Hsuan-Yi Chu [Thu, 12 Mar 2015 19:15:05 +0000 (12:15 -0700)] 
DRILL-2441: For outer-join, if there is any inequality condition, Cartesian-Join exception will be thrown

7 years agoDRILL-1385, along with some cleanup
Chris Westin [Wed, 25 Feb 2015 22:53:41 +0000 (14:53 -0800)] 
DRILL-1385, along with some cleanup

Cleaned up option handling. This includes using finals, making member variables
private whenever possible, and some formatting.
- fixed a bug in the string formatting for the double range validator
- OptionValidator, OptionValue, and their implementations now conspire not to
  allow the creation of malformed options because the OptionType has been added
  to validator calls to handle OptionValues that are created on demand.

Started with updated byte code rewrite from Jacques
Fixed several problems with scalar value replacement:
- use consistent ASM api version throughout
- stop using deprecated ASM methods (actually causes bugs)
  - visitMethodInsn()
- added a couple of missing super.visitEnd()s
- fixed a couple of minor FindBugs issues
- accounted for required stack size increases when replacing holders for
  longs and doubles
- added accounting for frame offsets to cope with long and double local
  variables and value holder members
- fixed a few minor bugs found with FindBugs
- stop using carrotlabs' hash map lget() method on shared constant data
- fixed an incorrect use of DUP2 on objectrefs when copying long or double
  holder members into locals
- fixed a problem with redundant POP instructions left behind after replacement
- fixed a problem with incorrect DUPs in multiple assignment statements
- fixed a problem with DUP_X1 replacement when handling constants in multiple
  assignment statements
- fixed a problem with non-replaced holder member post-decrements
- don't replace holders passed to static functions as "out" parameters
  (common with Accessors on repeated value vectors)
- increased the maximum required stack size when transferring holder members to
  locals
- changed the code generation block type mappings for constants for external
  sorts
- fixed problems handling constant and non-constant member variables in
  operator classes
  - in general, if a holder is assigned to or from an operator member variable,
    it can't be replaced (at least not until we replace those as well)
  - Use a derived ASM Analyzer (MethodAnalyzer) and Frame
    (AssignmentTrackingFrame) in order to establish relationships between
    assignments of holders through chains of local variables. This effectively
    back-propagates non-replaceability attributes so that if a holder variable
    that can't be replaced is assigned to from another holder variable, that
    second one cannot be replaced either, and so on through longer chains of
    assignments.
- code for dumping generated source code
- MergeAdapter dumps before and after results of scalar replacement
  (if it's on)
- fixed some problems in ReplacingBasicValue by replacing HashSet with
  IdentityHashMap
- made loggers private
- added a retry strategy for scalar replacement
  if a scalar replacement code rewriting fails, then this will try to
  regenerate the bytecode again without the scalar replacement.
- bytecode verification is always on now (required for the retry strategy)
- use system option to determine whether scalar replacement should be used
  - default option: if scalar replacement fails, retry without it
  - force replacement on or off
- unit tests for the retry strategy are based on a single known failure case,
  covered by DRILL-2326.
  - add tests TestConvertFunctions to test the three scalar replacement options
    for the failing test case (testVarCharReturnTripConvertLogical)
- made it possible to set a SYSTEM option as a java property in Drillbit
- added a command line argument to force scalar replacement to be on during
  testing in the rootmost pom.xml

In the course of this, added increased checking of intermediate stages of code
rewriting, as well as logging of classes that cause failures.
- work around a bug in ASM's CheckClassAdapter that doesn't allow for checking
  of inner classes
Added comments, tidied up formatting, and added "final" in a number of places.

Signed-off-by: vkorukanti <venki.korukanti@gmail.com>
7 years agoDRILL-2453: Handle the case where incoming has no schema in PartitionSender.
vkorukanti [Sun, 15 Mar 2015 20:23:03 +0000 (13:23 -0700)] 
DRILL-2453: Handle the case where incoming has no schema in PartitionSender.

7 years agoDRILL-2128.2: Fixed DatabaseMetaData.getColumns's DATA_TYPE, TYPE_NAME.
dbarclay [Wed, 11 Mar 2015 02:30:36 +0000 (19:30 -0700)] 
DRILL-2128.2: Fixed DatabaseMetaData.getColumns's DATA_TYPE, TYPE_NAME.

- Created basic test for DATA_TYPE and TYPE_NAME.
- Fixed DATA_TYPE:  Added mapping from type name/descriptor strings from
  INFORMATION_SCHEMA.COLUMNS.DATA_TYPE to java.sql.Types.* integer type codes
  for DatabaseMetaData.getColumns's DATA_TYPE.
- Fixed TYPE_NAME:  Added TYPE_NAME returning type name/descriptor strings from
  INFORMATION_SCHEMA.COLUMNS.DATA_TYPE
- Added FIXMEs for some missing/misnamed/wrong fields.  (See DRILL-2420.)

7 years agoDRILL-2128.1: Preparatory changes: Labeled result cols.; formatted SQL. [MetaImpl]
dbarclay [Wed, 11 Mar 2015 04:10:14 +0000 (21:10 -0700)] 
DRILL-2128.1: Preparatory changes: Labeled result cols.; formatted SQL. [MetaImpl]

7 years agoDRILL-1719 - handle the case in which /var/log/drill exists but the current user...
Patrick Wong [Mon, 17 Nov 2014 23:16:10 +0000 (15:16 -0800)] 
DRILL-1719 - handle the case in which /var/log/drill exists but the current user does not have permission to write to it

7 years agoDRILL-2225: Fix missing PartitionSenderRootExec stats.
vkorukanti [Tue, 10 Mar 2015 17:51:25 +0000 (10:51 -0700)] 
DRILL-2225: Fix missing PartitionSenderRootExec stats.

7 years agoDRILL-2207: Union RelOperator supports different types to be unioned together by...
Hsuan-Yi Chu [Thu, 12 Feb 2015 02:29:09 +0000 (18:29 -0800)] 
DRILL-2207: Union RelOperator supports different types to be unioned together by casting; Disable Union-all on  Select * from schemaless data source

7 years agoDRILL-2209 Insert ProjectOperator with MuxExchange
Yuliya Feldman [Fri, 13 Feb 2015 03:32:11 +0000 (19:32 -0800)] 
DRILL-2209 Insert ProjectOperator with MuxExchange

7 years agoDRILL-133: LocalExchange planning and exec.
vkorukanti [Fri, 23 Jan 2015 02:04:37 +0000 (18:04 -0800)] 
DRILL-133: LocalExchange planning and exec.

7 years agoDRILL-2347: Support parquet metadata read parallelization
Jacques Nadeau [Sun, 15 Feb 2015 21:53:29 +0000 (13:53 -0800)] 
DRILL-2347: Support parquet metadata read parallelization

Use HashMap internally when building endpoint map.

7 years agoDRILL-2187: Single Broadcast Sender
Jacques Nadeau [Sun, 1 Feb 2015 16:43:47 +0000 (08:43 -0800)] 
DRILL-2187: Single Broadcast Sender

Also includes:
1. Fix merge join planning issue (1c5c810 by jinfengni)
2. ExternalSort: Check the memory available for in-memory sorting or not in making decision to spill or not (36f9dd1)
3. Cleanup in ExternalSortBatch and its helper classes (36f9dd1)
4. MergeJoinBatch: Limit the outgoing record batch size to 2^15 (37dfeb8)
5. StreamingAggBatch: Limit outgoing record batch size to 2^15 (7d8a2e4)

7 years agoDRILL-2400: Part 2: Add merge filter rule to logical rule sets. Increase function...
Jacques Nadeau [Sun, 15 Feb 2015 22:17:20 +0000 (14:17 -0800)] 
DRILL-2400: Part 2: Add merge filter rule to logical rule sets.  Increase function cost differentials.

7 years agoDRILL-2400: Part 1: Change cpu cost estimation formula for DrillFilterRelBase. Add...
Jinfeng Ni [Fri, 20 Feb 2015 02:51:01 +0000 (18:51 -0800)] 
DRILL-2400: Part 1: Change cpu cost estimation formula for DrillFilterRelBase. Add testcase for MergeFilter rule.

Modify costing of Filter.

Cost change to Filter.

Change to filter costing.

Move one test utility method to PlanTestBase.

7 years agoDRILL-2226: Plan verification utility.
Jason Altekruse [Thu, 12 Feb 2015 18:52:47 +0000 (10:52 -0800)] 
DRILL-2226: Plan verification utility.

7 years agoDRILL-2262: selecting columns of certain datatypes from a dictionary encoded parquet...
adeneche [Fri, 27 Feb 2015 03:55:08 +0000 (19:55 -0800)] 
DRILL-2262: selecting columns of certain datatypes from a dictionary encoded parquet file created by drill fails

7 years agoDRILL-2266: Complex parquet reader fails on reading timestamp datatype
adeneche [Thu, 26 Feb 2015 16:56:09 +0000 (08:56 -0800)] 
DRILL-2266: Complex parquet reader fails on reading timestamp datatype

7 years agoDRILL-2267: bumping parquet version to 1.5.1-r7
adeneche [Thu, 26 Feb 2015 15:55:03 +0000 (07:55 -0800)] 
DRILL-2267: bumping parquet version to 1.5.1-r7

7 years agoDRILL-2337: disable dictionary encoding by default
adeneche [Fri, 27 Feb 2015 20:14:00 +0000 (20:14 +0000)] 
DRILL-2337: disable dictionary encoding by default

7 years agoDRILL-2359: ClassPathFileSystem.getFileStatus() should throw FileNotFoundException...
adeneche [Tue, 3 Mar 2015 00:03:40 +0000 (16:03 -0800)] 
DRILL-2359: ClassPathFileSystem.getFileStatus() should throw FileNotFoundException when path doesn't exist

7 years ago1908: new window function implementation
adeneche [Tue, 30 Dec 2014 18:15:05 +0000 (10:15 -0800)] 
1908: new window function implementation

7 years agoDRILL-1897: Fix return type of functions to be nullable during planning phase
Mehant Baid [Thu, 5 Mar 2015 03:08:46 +0000 (19:08 -0800)] 
DRILL-1897: Fix return type of functions to be nullable during planning phase

7 years agoDRILL-2372: Fix current_date to convert input to UTC before storing in the vector
Mehant Baid [Wed, 4 Mar 2015 00:58:58 +0000 (16:58 -0800)] 
DRILL-2372: Fix current_date to convert input to UTC before storing in the vector

7 years agoDRILL-1692: Fixing Mongo join issue when * is selected
Kamesh [Tue, 3 Mar 2015 15:52:25 +0000 (21:22 +0530)] 
DRILL-1692: Fixing Mongo join issue when * is selected

7 years agoDRILL-2118: inform user with a user friendly error message if kvgen fails due to...
Hanifi Gunes [Wed, 4 Mar 2015 00:37:24 +0000 (16:37 -0800)] 
DRILL-2118: inform user with a user friendly error message if kvgen fails due to heterogenous types

7 years agoDRILL-1875: getTransferClone should ignore given wrappers
Hanifi Gunes [Mon, 2 Mar 2015 21:38:17 +0000 (13:38 -0800)] 
DRILL-1875: getTransferClone should ignore given wrappers

7 years agoDRILL-2338: Fix Decimal38/Decimal28 vector's get() to copy the scale and precision...
Mehant Baid [Sat, 28 Feb 2015 03:21:51 +0000 (19:21 -0800)] 
DRILL-2338: Fix Decimal38/Decimal28 vector's get() to copy the scale and precision into the holder

7 years agoDRILL-2236: Optimize hash inner join by swapping inputs based on row count comparison...
Jinfeng Ni [Thu, 29 Jan 2015 21:24:28 +0000 (13:24 -0800)] 
DRILL-2236: Optimize hash inner join by swapping inputs based on row count comparison. Add a planner option to enable/disable this feature.

Revise code based on review comments.

7 years agoDRILL-2178: Update outgoing record batch size and allocation in PartitionSender.
vkorukanti [Fri, 6 Feb 2015 21:20:07 +0000 (13:20 -0800)] 
DRILL-2178: Update outgoing record batch size and allocation in PartitionSender.

Also:
 + Add setInitialCapacity() method to ValueVector interface set the initial capacity
   of memory allocated in first allocateNew() call.
 + Send an empty batch for fast schema instead of flushing the OutgoingRecordBatches
   which throw away allocated memory and reallocate again.
 + Remove the v.getValueVector().makeTransferPair(outgoingVector) hack as the complex
   schema child schema population bug is fixed in DRILL-1885.
 + Cleanup/refactor PartitionSender related code.

7 years agoDRILL-2328: For concat function, null input is treated as empty string; for concat...
Hsuan-Yi Chu [Fri, 27 Feb 2015 03:51:04 +0000 (19:51 -0800)] 
DRILL-2328: For concat function, null input is treated as empty string; for concat operator (i.e., ||), if any input is null, the output is null

7 years agoDRILL-1742: Use Hive stats when planning queries on Hive data sources
adeneche [Tue, 20 Jan 2015 20:23:58 +0000 (12:23 -0800)] 
DRILL-1742: Use Hive stats when planning queries on Hive data sources

7 years agoDRILL-2327: Raise the max value allowed for join's cardinality estimate factor to...
Aman Sinha [Fri, 27 Feb 2015 16:38:23 +0000 (08:38 -0800)] 
DRILL-2327: Raise the max value allowed for join's cardinality estimate factor to allow for hugely expanding joins.

7 years agoDRILL-2321: FlattenRecordBatch should transfer vectors honoring output field reference.
Hanifi Gunes [Thu, 26 Feb 2015 19:23:30 +0000 (11:23 -0800)] 
DRILL-2321: FlattenRecordBatch should transfer vectors honoring output field reference.

7 years agoDRILL-1953: alter session set store.json.all_text_mode does not work as documented
adeneche [Wed, 4 Feb 2015 04:45:54 +0000 (20:45 -0800)] 
DRILL-1953: alter session set store.json.all_text_mode does not work as documented

7 years agoDRILL-2294: Prevent collecting intermediate stats before the operator tree was finish...
Jason Altekruse [Wed, 25 Feb 2015 18:29:19 +0000 (10:29 -0800)] 
DRILL-2294: Prevent collecting intermediate stats before the operator tree was finished being constructed.

7 years agoDRILL-1535:Logical plan deserialization error in Join
Yash Sharma [Wed, 12 Nov 2014 18:49:34 +0000 (00:19 +0530)] 
DRILL-1535:Logical plan deserialization error in Join

7 years agoDRILL-1515: For the Web UI users, if exception is thrown, users are redirected to...
Hsuan-Yi Chu [Mon, 5 Jan 2015 18:28:29 +0000 (10:28 -0800)] 
DRILL-1515: For the Web UI users, if exception is thrown, users are redirected to another page, which shows the error message

7 years agoDRILL-1971: Mongo Storage Plugin slowness issue
akumarb2010 [Fri, 9 Jan 2015 15:19:59 +0000 (20:49 +0530)] 
DRILL-1971: Mongo Storage Plugin slowness issue

7 years agoDRILL-2013: Part 2 - Improve test for flatten bug
AdamPD [Mon, 9 Feb 2015 02:12:36 +0000 (12:12 +1000)] 
DRILL-2013: Part 2 - Improve test for flatten bug

7 years agoDRILL-2013: Fix flatten when no rows match filters
AdamPD [Fri, 23 Jan 2015 07:01:36 +0000 (17:01 +1000)] 
DRILL-2013: Fix flatten when no rows match filters

7 years agoDRILL-2031: Parquet bit column reader fix
Jason Altekruse [Mon, 19 Jan 2015 23:38:46 +0000 (15:38 -0800)] 
DRILL-2031: Parquet bit column reader fix

Removes attempted optimized bit reader, left a comment where it was removed as a pointer back to this change set. For now the higher level Parquet interface will be used to ensure reading is correct, a fix of the optimized reader can be pursued at a later date if is is necessary.

Fixed two bugs in the test framework necessary to verify the fix against the complex parquet reader (which uses a similar interface, but it ensures that none of the other columns are read incorrectly).

Fix conflict between parquet bit reader fix and vector reallocation changes.

7 years agoDRILL-1325: Throw UnsupportedRelOperatorException for unequal joins, implicit cross...
Hsuan-Yi Chu [Wed, 25 Feb 2015 03:08:40 +0000 (19:08 -0800)] 
DRILL-1325: Throw UnsupportedRelOperatorException for unequal joins, implicit cross joins

7 years agoDRILL-1378: Ctrl-C to cancel a query that has not returned with the first result...
Parth Chandra [Wed, 25 Feb 2015 17:56:12 +0000 (09:56 -0800)] 
DRILL-1378: Ctrl-C to cancel a query that has not returned with the first result set.

7 years agoDRILL-1690: Issue with using HBase plugin to access row_key only
Aditya Kishore [Wed, 25 Feb 2015 09:10:48 +0000 (01:10 -0800)] 
DRILL-1690: Issue with using HBase plugin to access row_key only

7 years agoDRILL-2130: Fixed JUnit/Hamcrest/Mockito/Paranamer class path problem.
dbarclay [Sun, 22 Feb 2015 08:45:42 +0000 (00:45 -0800)] 
DRILL-2130: Fixed JUnit/Hamcrest/Mockito/Paranamer class path problem.

7 years agoDRILL-1496: Fix serialization of 'similar to' while converting from optiq expression...
Mehant Baid [Fri, 20 Feb 2015 23:45:54 +0000 (15:45 -0800)] 
DRILL-1496: Fix serialization of 'similar to' while converting from optiq expression to drill.

7 years agoDRILL-2283: Fixed Java VARCHAR(1) in INFO._SCHEMA that caused bad comparisons.
dbarclay [Sat, 21 Feb 2015 02:57:36 +0000 (18:57 -0800)] 
DRILL-2283: Fixed Java VARCHAR(1) in INFO._SCHEMA that caused bad comparisons.

7 years agoDRILL-2253: Vectorized Parquet reader fails to read correctly against RLE Dictionary...
adeneche [Wed, 18 Feb 2015 00:58:11 +0000 (16:58 -0800)] 
DRILL-2253: Vectorized Parquet reader fails to read correctly against RLE Dictionary encoded DATE column

7 years agoDRILL-1733 - Include Hadoop winutils in Drill distribution
Patrick Wong [Sat, 21 Feb 2015 00:08:33 +0000 (16:08 -0800)] 
DRILL-1733 - Include Hadoop winutils in Drill distribution

7 years agoDRILL-1062: Implemented null ordering (NULLS FIRST/NULLS LAST).
dbarclay [Tue, 17 Feb 2015 21:50:16 +0000 (13:50 -0800)] 
DRILL-1062: Implemented null ordering (NULLS FIRST/NULLS LAST).

Primary:
- Split "compare_to" function templates (for sorting) into
  "compare_to_nulls_high" and "compare_to_nulls_low" versions.
- Added tests to verify ORDER BY ordering.
- Added tests to verify merge join order correctness.
- Implemented java.sql.DatabaseMetaData.nullsAreSortedHigh(), etc.
Secondary:
- Eliminated DateInterfaceFunctions.java template (merged into other).
- Renamed comparison-related template data objects and file names.
- Eliminated unused template macros, function template classes.
- Overhauled Order.Ordering; added unit test.
- Regularized some generated-class names.
Miscellaneous:
- Added toString() to ExpressionPosition, Order.Ordering, JoinStatus.
- Fixed some typos.
- Fixed some comment syntax.

7 years agoDRILL-2242 : Propagate distribution trait when Project only outputs a subset of child...
Jinfeng Ni [Tue, 17 Feb 2015 07:41:32 +0000 (23:41 -0800)] 
DRILL-2242 : Propagate distribution trait when Project only outputs a subset of child's distribution field.

Minor code style fix.

change to unit test.

7 years agoDRILL-2269: Add default implementation for estimating cost of evaluating an expressio...
Jinfeng Ni [Wed, 18 Feb 2015 21:57:12 +0000 (13:57 -0800)] 
DRILL-2269: Add default implementation for estimating cost of evaluating an expression, in stead of throwing Exception.

Set default cost of evaluating a HiveFuncHolder expression.

7 years agoDRILL-2197: Fix no applicable constructor error in outer join with a map type
Parth Chandra [Thu, 19 Feb 2015 06:13:49 +0000 (22:13 -0800)] 
DRILL-2197: Fix no applicable constructor error in outer join with a map type

7 years agoDRILL-2054: Interpret "||" as string concat operator; Disable `&&`;
Hsuan-Yi Chu [Fri, 30 Jan 2015 17:09:04 +0000 (09:09 -0800)] 
DRILL-2054: Interpret "||" as string concat operator; Disable `&&`;

7 years agoDRILL-2169: disable embedded server for tests when not needed
Hanifi Gunes [Thu, 5 Feb 2015 03:05:47 +0000 (19:05 -0800)] 
DRILL-2169: disable embedded server for tests when not needed

7 years agoDRILL-2254: use parent field while adding a child vector into a repeated list
Hanifi Gunes [Tue, 17 Feb 2015 20:20:01 +0000 (12:20 -0800)] 
DRILL-2254: use parent field while adding a child vector into a repeated list

7 years agoDRILL-1219. C++ Client. Fix timeout for 32-bit windows platform
alzarei [Mon, 16 Feb 2015 22:41:28 +0000 (14:41 -0800)] 
DRILL-1219. C++ Client. Fix timeout for 32-bit windows platform

7 years agoDRILL-2038: Fix handling of error objects. C++ Client syncronous API has a crash...
Parth Chandra [Mon, 19 Jan 2015 20:07:24 +0000 (12:07 -0800)] 
DRILL-2038: Fix handling of error objects. C++ Client syncronous API has a crash with multiple parallel queries.

7 years agoDRILL-2151: VariableLengthVector.copyFromSafe() unnecessary sets the offsetVector
adeneche [Tue, 3 Feb 2015 21:53:16 +0000 (13:53 -0800)] 
DRILL-2151: VariableLengthVector.copyFromSafe() unnecessary sets the offsetVector

7 years agoDRILL-2252: Cleanup resources when fragment is cancelled.
vkorukanti [Sun, 15 Feb 2015 06:18:31 +0000 (22:18 -0800)] 
DRILL-2252: Cleanup resources when fragment is cancelled.

7 years agoDRILL-2251: Fix resource leaking with new DrillFileSystem
vkorukanti [Fri, 13 Feb 2015 01:27:05 +0000 (17:27 -0800)] 
DRILL-2251: Fix resource leaking with new DrillFileSystem

7 years agoDRILL-2168: Fix incorrect query result issue when group by use an expression with...
Jinfeng Ni [Thu, 12 Feb 2015 01:11:06 +0000 (17:11 -0800)] 
DRILL-2168:  Fix incorrect query result issue when group by use an expression with drill build-in functions.  Fix is in Calcite.