incubator-druid.git
6 hours agoAdd fail message (#6720) master
Atul Mohan [Tue, 11 Dec 2018 16:05:50 +0000 (10:05 -0600)] 
Add fail message (#6720)

21 hours agoadd missing contrib extensions to distribution packaging, fix influx package name...
David Lim [Tue, 11 Dec 2018 01:56:32 +0000 (18:56 -0700)] 
add missing contrib extensions to distribution packaging, fix influx package name, fix checkstyle plugin config to support Maven < 3.3.9 (#6717)

3 days agoFix shutdownAllTasks API for non-existing dataSource (#6706)
Jihoon Son [Sat, 8 Dec 2018 17:54:01 +0000 (09:54 -0800)] 
Fix shutdownAllTasks API for non-existing dataSource (#6706)

3 days agoFileUtils: Sync directory entry too on writeAtomically. (#6677)
Gian Merlino [Sat, 8 Dec 2018 16:12:59 +0000 (08:12 -0800)] 
FileUtils: Sync directory entry too on writeAtomically. (#6677)

* FileUtils: Sync directory entry too on writeAtomically.

See the fsync(2) man page for why this is important:
https://linux.die.net/man/2/fsync

This also plumbs CompressionUtils's "zip" function through
writeAtomically, so the code for handling atomic local filesystem
writes is all done in the same place.

* Remove unused import.

* Avoid FileOutputStream.

* Allow non-atomic writes to overwrite.

* Add some comments. And no need to flush an unbuffered stream.

4 days agoDouble-checked locking bugs (#6662)
Furkan KAMACI [Fri, 7 Dec 2018 16:10:29 +0000 (19:10 +0300)] 
Double-checked locking bugs (#6662)

* Double-checked locking bug is fixed.

* @Nullable is removed since there is no need to use along with @MonotonicNonNull.

* Static import is removed.

* Lazy initialization is implemented.

* Local variables used instead of volatile ones.

* Local variables used instead of volatile ones.

4 days agoFix travis timeout in BufferHashGrouperTest (#6713)
Jihoon Son [Fri, 7 Dec 2018 04:05:27 +0000 (20:05 -0800)] 
Fix travis timeout in BufferHashGrouperTest (#6713)

* Fix travis timeout in BufferHashGrouperTest

* adjust buffer size

* adjust bufferSize and loadFactor

* increase memory

* add debug code

* cat error

* after script

* print logs

* print per 2 min

* use direct mem

* clean up

5 days agoCorrect Documentation for lowerStrict/upperStrict (#6707)
Vincent Newkirk [Thu, 6 Dec 2018 18:14:50 +0000 (03:14 +0900)] 
Correct Documentation for lowerStrict/upperStrict (#6707)

The documentation for Bound filter's lowerStrict/upperStrict is incorrect. It is not consistent with the examples provided and actual behaviour of the bound filter. Correct this.

5 days agoadd close method in Cache interface (#6540)
Mingming Qiu [Thu, 6 Dec 2018 09:28:41 +0000 (17:28 +0800)] 
add close method in Cache interface (#6540)

* add close method in Cache interface

* address comments

* address comments and fix travis-ci

* use try-finally

7 days agoAdd TaskCountStatsMonitor to monitor task count stats (#6657)
Mingming Qiu [Tue, 4 Dec 2018 21:37:17 +0000 (05:37 +0800)] 
Add TaskCountStatsMonitor to monitor task count stats (#6657)

* Add TaskCountStatsMonitor to monitor task count stats

* address comments

* add file header

* tweak test

7 days agoAdd default comparison to HavingSpecMetricComparator for custom Aggregator types...
Atul Mohan [Tue, 4 Dec 2018 21:35:13 +0000 (15:35 -0600)] 
Add default comparison to HavingSpecMetricComparator for custom Aggregator types (#6505)

* Add default comparison

* Switch to BigDecimal comparison

* Add comparator from AggFactory

* Fix indent

* Add tests

7 days agoautosize processing buffers based on direct memory sizing by default (#6588)
Clint Wylie [Tue, 4 Dec 2018 01:40:02 +0000 (17:40 -0800)] 
autosize processing buffers based on direct memory sizing by default (#6588)

* autosize processing buffers based on direct memory sizing

* remove oops, more test

* max 1gb autosize buffers, test, start of docs

* fix oops

* revert accidental change

* print buffer size in exception

* change the things

10 days agoremove AbstractResourceFilter.isApplicable because it is not (#6691)
Clint Wylie [Sat, 1 Dec 2018 13:52:31 +0000 (05:52 -0800)] 
remove AbstractResourceFilter.isApplicable because it is not (#6691)

* remove AbstractResourceFilter.isApplicable because it is not, add tests for OverlordResource.doShutdown and OverlordResource.shutdownTasksForDatasource

* cleanup

10 days agofix links to use relative references (#6696)
David Lim [Sat, 1 Dec 2018 00:32:10 +0000 (17:32 -0700)] 
fix links to use relative references (#6696)

10 days agoSimplify DruidNodeDiscoveryProvider; add DruidNodeDiscovery.Listener.nodeViewInitiali...
Roman Leventov [Sat, 1 Dec 2018 00:12:56 +0000 (01:12 +0100)] 
Simplify DruidNodeDiscoveryProvider; add DruidNodeDiscovery.Listener.nodeViewInitialized() (#6606)

* Simplify DruidNodeDiscoveryProvider; add DruidNodeDiscovery.Listener.nodeViewInitialized() method; prohibit and eliminate some suboptimal Java 8 patterns

* Fix style

* Fix HttpEmitterTest.timeoutEmptyQueue()

* Add DruidNodeDiscovery.Listener.nodeViewInitialized() calls in tests

* Clarify code

11 days agoremove extensions from default configs that have configuration/library dependencies...
David Lim [Fri, 30 Nov 2018 20:52:46 +0000 (13:52 -0700)] 
remove extensions from default configs that have configuration/library dependencies and update docs (#6694)

11 days agoremove #658 doc reference for Schema-less design (#6693)
rcgarcia74 [Fri, 30 Nov 2018 19:53:57 +0000 (14:53 -0500)] 
remove #658 doc reference for Schema-less design (#6693)

11 days agoreplace Files.map() with FileUtils.map() (#6692)
Shimi Kiviti [Fri, 30 Nov 2018 15:40:41 +0000 (17:40 +0200)] 
replace Files.map() with FileUtils.map() (#6692)

11 days agoFix overlord api and console (#6686)
Jihoon Son [Fri, 30 Nov 2018 07:45:28 +0000 (23:45 -0800)] 
Fix overlord api and console (#6686)

* Fix overlord APIs and console

* remove getRunningTasksByDataSource

* add missing path to isApplicable

12 days agolazy create descriptor in ProtobufInputRowParser (#6678)
陈春斌 [Thu, 29 Nov 2018 05:59:29 +0000 (13:59 +0800)] 
lazy create descriptor in ProtobufInputRowParser (#6678)

12 days agofix cannot resolve param at OverlordResource#getTasks (#6679)
Mingming Qiu [Thu, 29 Nov 2018 02:07:21 +0000 (10:07 +0800)] 
fix cannot resolve param at OverlordResource#getTasks (#6679)

13 days agoemit maxLag/avgLag in KafkaSupervisor (#6587)
Mingming Qiu [Wed, 28 Nov 2018 10:11:14 +0000 (18:11 +0800)] 
emit maxLag/avgLag in KafkaSupervisor (#6587)

* emit maxLag/totalLag/avgLag in KafkaSupervisor

* modify ingest/kafka/totalLag to ingest/kafka/lag for backwards compatibility

13 days agoadd rule count on log (#6467)
hate13 [Wed, 28 Nov 2018 08:08:38 +0000 (16:08 +0800)] 
add rule count on log (#6467)

* add rule count on log

* add final

13 days agoindexing-service: fix HTML title on overlord console (#6671)
David Glasser [Wed, 28 Nov 2018 06:34:26 +0000 (22:34 -0800)] 
indexing-service: fix HTML title on overlord console (#6671)

This follows a similar fix to the body of the page done in #5627.

13 days agofix missing property in JsonTypeInfo of SegmentWriteOutMediumFactory (#6656)
Mingming Qiu [Tue, 27 Nov 2018 23:59:58 +0000 (07:59 +0800)] 
fix missing property in JsonTypeInfo of SegmentWriteOutMediumFactory (#6656)

13 days agoFix IndexTaskClient to retry on ChannelException (#6649)
Jihoon Son [Tue, 27 Nov 2018 23:54:38 +0000 (15:54 -0800)] 
Fix IndexTaskClient to retry on ChannelException (#6649)

* Fix IndexTaskClient to retry on ChannelException

* fix travis and add javadoc

* address comment

13 days agoEmit query metrics even if the ETags are equal (#6663)
Mingming Qiu [Tue, 27 Nov 2018 23:18:01 +0000 (07:18 +0800)] 
Emit query metrics even if the ETags are equal (#6663)

13 days agoRemove duplicate DataSegmentTest (#6669)
Jihoon Son [Tue, 27 Nov 2018 23:13:39 +0000 (15:13 -0800)] 
Remove duplicate DataSegmentTest (#6669)

2 weeks agofaster flattening for non-existent paths (#6654)
Clint Wylie [Tue, 27 Nov 2018 22:14:11 +0000 (14:14 -0800)] 
faster flattening for non-existent paths (#6654)

* faster flattening for non-existent properties to circumvent upstream json-path issue

* fix json provider

* revert to using null instead of undefined

2 weeks agoFix a race in FileSessionCredentialsProvider (#6664)
Roman Leventov [Tue, 27 Nov 2018 20:02:30 +0000 (21:02 +0100)] 
Fix a race in FileSessionCredentialsProvider (#6664)

`sessionToken`, `accessKey` and `secretKey` must be updated atomically.

Another race is possible between the file updater and the Druid process reading the file. It could be enforced only with mandatory file locking, but file locking is advisory by default in Linux.

2 weeks agobloom filter sql (#6502)
Clint Wylie [Tue, 27 Nov 2018 06:11:18 +0000 (22:11 -0800)] 
bloom filter sql (#6502)

* bloom filter sql support

* docs

* style fix

* style fixes after rebase

* use copied/patched bloomkfilter

* remove context literal lookup function, changes from review

* fix build

* rename LookupOperatorConversion to QueryLookupOperatorConversion

* remove doc

* revert unintended change

* add internal exception to bloom filter deserialization exception

2 weeks agoFind duplicate lines with checkstyle; enable some duplicate inspections in IntelliJ...
Roman Leventov [Mon, 26 Nov 2018 15:55:42 +0000 (16:55 +0100)] 
Find duplicate lines with checkstyle; enable some duplicate inspections in IntelliJ (#6558)

Not putting this to 0.13 milestone because the found bugs are not critical (one is a harmless DI config duplicate, and another is in a benchmark.

Change in `DumpSegment` is just an indentation change.

2 weeks agoDocs: Fix wikipedia links in Ingestion:Rollup (#6659)
Evans Hauser [Sat, 24 Nov 2018 00:28:05 +0000 (16:28 -0800)] 
Docs: Fix wikipedia links in Ingestion:Rollup (#6659)

The rendered site doesn't have automatic link detection, so we need to add these links in explicitly. This also fixes the Measure link, which included an extra `)`

http://druid.io/docs/latest/ingestion/index.html#rollup

2 weeks agoFix issue that tasks tables in metadata storage are not cleared (#6592)
seoeun [Thu, 22 Nov 2018 03:50:31 +0000 (12:50 +0900)] 
Fix issue that tasks tables in metadata storage are not cleared (#6592)

* tasks tables in metadata storage are not cleared

* address comments. remove tasklogs and revert obsolete changes

* address comments. change comment and update doc.

* address comments. update doc more detailed

* address comments. remove redundant log and update doc more detailed.

* address comments. update document

2 weeks agoFix missing default config in some calls to coordinator dynamic configs. (#6652)
Gian Merlino [Thu, 22 Nov 2018 02:25:39 +0000 (21:25 -0500)] 
Fix missing default config in some calls to coordinator dynamic configs. (#6652)

* Fix missing default config in some calls to coordinator dynamic configs.

The lack of a default config meant that if someone called an API
_without_ a default config before one _with_ a default config, then
the default value would get stuck at null instead of the intended
default value. I noticed this in a cluster where calling /druid/coordinator/v1/config
before a coordinator had fully started up would lead to NPEs during
DruidCoordinatorRuleRunner.

This patch makes the default configs consistent across all calls.

* Remove unnecessary null check.

2 weeks agoUse PasswordProvider for basic HTTP escalator (#6650)
Jonathan Wei [Wed, 21 Nov 2018 15:34:15 +0000 (07:34 -0800)] 
Use PasswordProvider for basic HTTP escalator (#6650)

3 weeks agoAdded support for filtering by unused parameter for HeapMemoryTaskStorage (#6510)
Marat [Tue, 20 Nov 2018 21:42:44 +0000 (00:42 +0300)] 
Added support for filtering by unused parameter for HeapMemoryTaskStorage (#6510)

* 1. added support for unused DateTime start parameter in getRecentlyFinishedTaskInfoSince method:
 HeapMemoryTaskStorage.getRecentlyFinishedTaskInfoSince return the finished tasks by comparing TaskStuff.createdDate with the start time
2. added filtering by status complete to TaskStuff list stream in HeapMemoryTaskStorage.getNRecentlyFinishedTaskInfo method.
3. changed names of methods and parameters to present that public API method OverlordResource.getTasks return the list of completed tasks, which createdDate, not date of completion, belongs to the interval parameter.

* 1. added support for unused DateTime start parameter in getRecentlyFinishedTaskInfoSince method:
 HeapMemoryTaskStorage.getRecentlyFinishedTaskInfoSince return the finished tasks by comparing TaskStuff.createdDate with the start time
2. added filtering by status complete to TaskStuff list stream in HeapMemoryTaskStorage.getNRecentlyFinishedTaskInfo method.
3. changed names of methods and parameters to present that public API method OverlordResource.getTasks return the list of completed tasks, which createdDate, not date of completion, belongs to the interval parameter.

* Fixed OverlordResourceTest to Support changed methods names

* Changed methods and parameters names to make them more obvious to understand.

* Changed String.replace() for the StringUtils.replace()(#6607)

* Fixed checkstyle error

3 weeks agoAdd checkstyle rules about imports and empty lines between members (#6543)
Roman Leventov [Tue, 20 Nov 2018 11:42:15 +0000 (12:42 +0100)] 
Add checkstyle rules about imports and empty lines between members (#6543)

* Add checkstyle rules about imports and empty lines between members

* Add suppressions

* Update Eclipse import order

* Add empty line

* Fix StatsDEmitter

3 weeks agofix typo in config page of docs (#6645)
Caroline1000 [Tue, 20 Nov 2018 00:32:58 +0000 (16:32 -0800)] 
fix typo in config page of docs (#6645)

3 weeks agoSupport DogStatsD style tags in statsd-emitter (#6605)
Deiwin Sarjas [Mon, 19 Nov 2018 17:47:57 +0000 (19:47 +0200)] 
Support DogStatsD style tags in statsd-emitter  (#6605)

* Replace StatsD client library

The [Datadog package][1] is a StatsD compatible drop-in replacement for the
client library, but it seems to be [better maintained][2] and has support for
Datadog DogStatsD specific features, which will be made use of in a subsequent
commit.

The `count`, `time`, and `gauge` methods are actually exactly compatible with
the previous library and the modifications shouldn't be required, but EasyMock
seems to have a hard time dealing with the variable arguments added by the
DogStatsD library and causes tests to fail if no arguments are provided for the
last String vararg. Passing an empty array fixes the test failures.

[1]: https://github.com/DataDog/java-dogstatsd-client
[2]: https://github.com/tim-group/java-statsd-client/issues/37#issuecomment-248698856

* Retain dimension key information for StatsD metrics

This doesn't change behavior, but allows separating dimensions from the metric
name in subsequent commits.

There is a possible order change for values from
`dimsBuilder.build().values()`, but from the tests it looks like it doesn't
affect actual behavior and the order of user dimensions is also retained.

* Support DogStatsD style tags in statsd-emitter

Datadog [doesn't support name-encoded dimensions and uses a concept of _tags_
instead.][1] This change allows Datadog users to send the metrics without
having to encode the various dimensions in the metric names. This enables
building graphs and monitors with and without aggregation across various
dimensions from the same data.

As tests in this commit verify, the behavior remains the same for users who
don't enable the `druid.emitter.statsd.dogstatsd` configuration flag.

[1]: https://www.datadoghq.com/blog/the-power-of-tagged-metrics/#tags-decouple-collection-and-reporting

* Disable convertRange behavior for DogStatsD users

DogStatsD, unlike regular StatsD, supports floating-point values, so this
behavior is unnecessary. It would be possible to still support `convertRange`,
even with `dogstatsd` enabled, but that would mean that people using the
default mapping would have some of the gauges unnecessarily converted.

`time` is in milliseconds and doesn't support floating-point values.

3 weeks agoSystemSchema: Fix data types for various fields. (#6642)
Gian Merlino [Mon, 19 Nov 2018 01:24:19 +0000 (17:24 -0800)] 
SystemSchema: Fix data types for various fields. (#6642)

* SystemSchema: Fix data types for various fields.

- segments: start, end, partition_num
- servers: plaintext_port, tls_port
- tasks: plaintext_port, tls_port

The declared and actual types did not match, but they must or
else queries may generate ClassCastExceptions.

Also adjusted some of the code for generating values to be more
robust in the face of nulls or malformed strings.

* Fix style.

3 weeks agoKafka: Add warning to doc for earlyMessageRejectionPeriod. (#6644)
Gian Merlino [Sun, 18 Nov 2018 22:47:38 +0000 (14:47 -0800)] 
Kafka: Add warning to doc for earlyMessageRejectionPeriod. (#6644)

3 weeks agoFix Hadoop Indexing doc to clarify segmentOutputPath only required for CLI indexer...
Benjamin Hopp [Sat, 17 Nov 2018 04:19:20 +0000 (22:19 -0600)] 
Fix Hadoop Indexing doc to clarify segmentOutputPath only required for CLI indexer (#6636)

* Updated hadoop indexing doc to reflect segmentOutputPath is only required when using CLI indexer; otherwise it must be NULL

3 weeks agokafkasupervisor checkpointing bug (#6639)
Joshua Sun [Sat, 17 Nov 2018 00:09:55 +0000 (16:09 -0800)] 
kafkasupervisor checkpointing bug (#6639)

3 weeks agooptimize loading end offsets of all partitions (#6623)
hate13 [Fri, 16 Nov 2018 07:24:01 +0000 (15:24 +0800)] 
optimize loading end offsets of all partitions (#6623)

3 weeks agoExpressions: Fix improper supplier reuse with missing columns. (#6600)
Gian Merlino [Fri, 16 Nov 2018 06:13:32 +0000 (22:13 -0800)] 
Expressions: Fix improper supplier reuse with missing columns. (#6600)

* Expressions: Fix improper supplier reuse with missing columns.

ExpressionSelectors has an optimization that skips building a Map
when there is only one input supplier. However, this optimization
should not be used in the case where the is one input supplier but
more than one input identifier (which can happen when only one
input identifier corresponds to an actual column).

Fixes #6556.

* Add underscores to statics.

3 weeks agooptimize input row parsers (#6590)
Mingming Qiu [Fri, 16 Nov 2018 03:48:32 +0000 (11:48 +0800)] 
optimize input row parsers (#6590)

* optimize input row parsers

* address comments

3 weeks agoremove backpressure time from DefaultQueryMetrics pending on-going discussion (#6631)
David Lim [Fri, 16 Nov 2018 02:29:50 +0000 (19:29 -0700)] 
remove backpressure time from DefaultQueryMetrics pending on-going discussion (#6631)

3 weeks agoEnforce logging when killing a task (#6621)
Jihoon Son [Fri, 16 Nov 2018 02:01:56 +0000 (18:01 -0800)] 
Enforce logging when killing a task (#6621)

* Enforce logging when killing a task

* fix test

* address comment

* address comment

3 weeks agoProhibit String.replace() and String.replaceAll(), fix and prohibit some toString...
Roman Leventov [Thu, 15 Nov 2018 21:21:34 +0000 (22:21 +0100)] 
Prohibit String.replace() and String.replaceAll(), fix and prohibit some toString()-related redundancies (#6607)

* Prohibit String.replace() and String.replaceAll(), fix and prohibit some toString()-related redundancies

* Fix bug

* Replace checkstyle regexp with IntelliJ inspection

3 weeks agoProperly reset total size of segmentsToCompact in NewestSegmentFirstIterator (#6622)
Jihoon Son [Thu, 15 Nov 2018 09:00:51 +0000 (01:00 -0800)] 
Properly reset total size of segmentsToCompact in NewestSegmentFirstIterator (#6622)

* Properly reset total size of segmentsToCompact in NewestSegmentFirstIterator

* add test

3 weeks agoFix bugs in ExprEval (#6617)
Roman Leventov [Wed, 14 Nov 2018 23:20:52 +0000 (00:20 +0100)] 
Fix bugs in ExprEval (#6617)

3 weeks agoFix smile query documentation (#6620)
Niketh Sabbineni [Wed, 14 Nov 2018 00:51:02 +0000 (16:51 -0800)] 
Fix smile query documentation (#6620)

4 weeks agoDeprecate IntervalChunkingQueryRunner (#6591)
Jihoon Son [Tue, 13 Nov 2018 22:33:27 +0000 (14:33 -0800)] 
Deprecate IntervalChunkingQueryRunner (#6591)

* Deprecate IntervalChunkingQueryRunner

* add doc

* deprecate metric

* fix doc

4 weeks agoSQL: Set INFORMATION_SCHEMA catalog name to "druid". (#6595)
Gian Merlino [Tue, 13 Nov 2018 22:32:40 +0000 (14:32 -0800)] 
SQL: Set INFORMATION_SCHEMA catalog name to "druid". (#6595)

* SQL: Set INFORMATION_SCHEMA catalog name to "druid".

Some third party tools ignore catalogs with empty names. So using
the name "druid" for the catalog makes integration easier.

* Update tests.

4 weeks agoSQL: Support AVG on system tables. (#6601)
Gian Merlino [Tue, 13 Nov 2018 22:31:33 +0000 (14:31 -0800)] 
SQL: Support AVG on system tables. (#6601)

4 weeks agoSQL: Add "POSITION" function. (#6596)
Gian Merlino [Tue, 13 Nov 2018 21:39:00 +0000 (13:39 -0800)] 
SQL: Add "POSITION" function. (#6596)

Also add a "fromIndex" argument to the strpos expression function. There
are some -1 and +1 adjustment terms due to the fact that the strpos
expression behaves like Java indexOf (0-indexed), but the POSITION SQL
function is 1-indexed.

4 weeks agofix cannot resolve param at OverlordResource#getTasks (#6593)
QiuMM [Tue, 13 Nov 2018 17:47:11 +0000 (01:47 +0800)] 
fix cannot resolve param at OverlordResource#getTasks (#6593)

4 weeks agoRemove unnecessary path param from auto compaction api (#6594)
Jihoon Son [Tue, 13 Nov 2018 17:46:13 +0000 (09:46 -0800)] 
Remove unnecessary path param from auto compaction api (#6594)

* Remove unnecessary path param from auto compaction api

* fix ci

4 weeks agoadd missing license headers, in particular to MD files; clean up RAT … (#6563)
David Lim [Tue, 13 Nov 2018 17:38:37 +0000 (10:38 -0700)] 
add missing license headers, in particular to MD files; clean up RAT … (#6563)

* add missing license headers, in particular to MD files; clean up RAT exclusions

* revert inadvertent doc changes

* docs

* cr changes

* fix modified druid-production.svg

4 weeks agoOptimization for expressions that hit a single long column. (#6599)
Gian Merlino [Tue, 13 Nov 2018 17:36:32 +0000 (09:36 -0800)] 
Optimization for expressions that hit a single long column. (#6599)

* Optimization for expressions that hit a single long column.

There was previously a single-long-input optimization that applied only
to the time column. These have been combined together. Also adds
type-specific value caching to ExprEval, which allowed simplifying
the SingleLongInputCachingExpressionColumnValueSelector code.

* Add more benchmarks.

* Don't use LRU cache for __time.

* Simplify a bit.

* Let the cache grow.

4 weeks agofix kafka indexing task not processing through end offsets on publish, fixes #6602...
Clint Wylie [Mon, 12 Nov 2018 22:27:32 +0000 (14:27 -0800)] 
fix kafka indexing task not processing through end offsets on publish, fixes #6602 (#6603)

4 weeks agofix druid-bloom-filter thread-safety (#6584)
Clint Wylie [Fri, 9 Nov 2018 18:55:17 +0000 (10:55 -0800)] 
fix druid-bloom-filter thread-safety (#6584)

* use BloomFilter instead of BloomKFilter since the latters test method is not threadsafe

* fix formatting

* style and forbidden api

* remove redundant hive notice entry

* add todo with note to delete copied implementation and link to related hive jira

* better fix for masks than ThreadLocal

5 weeks agoFix various bugs; Enable more IntelliJ inspections and update error-prone (#6490)
Roman Leventov [Tue, 6 Nov 2018 22:38:08 +0000 (23:38 +0100)] 
Fix various bugs; Enable more IntelliJ inspections and update error-prone (#6490)

* Fix various bugs; Enable more IntelliJ inspections and update error-prone

* Fix NPE

* Fix inspections

* Remove unused imports

5 weeks agoUse current coordinator leader instead of cached one (#6551) (#6552)
Surekha [Tue, 6 Nov 2018 21:09:51 +0000 (13:09 -0800)] 
Use current coordinator leader instead of cached one (#6551) (#6552)

* Use current coordinator leader instead of cached one (#6551)

Check the response status and throw exception if not OK

* Modify tests

* PR comment

* Add the correct check for status of BytesAccumulatingResponseHandler

* Move the status check into JsonParserIterator so sql query outputs meaningful message on failure

* Fix tests

5 weeks agooverhaul 'druid-parquet-extensions' module, promoting from 'contrib' to 'core' (...
Clint Wylie [Tue, 6 Nov 2018 05:33:42 +0000 (21:33 -0800)] 
overhaul 'druid-parquet-extensions' module, promoting from 'contrib' to 'core' (#6360)

* move parquet-extensions from contrib to core, adds new hadoop parquet parser that does not convert to avro first and supports flattenSpec and int96 columns, add support for flattenSpec for parquet-avro conversion parser, much test with a bunch of files lifted from spark-sql

* fix avro flattener to support nullable primitives for auto discovery and now only supports primitive arrays instead of all arrays

* remove leftover print

* convert micro timestamp to millis

* checkstyle

* add ignore for .parquet and .parq to rat exclude

* fix legit test failure from avro flattern behavior change

* fix rebase

* add exclusions to pom to cut down on redundant jars

* refactor tests, add support for unwrapping lists for parquet-avro, review comments

* more comment

* fix oops

* tweak parquet-avro list handling

* more docs

* fix style

* grr styles

5 weeks agoHide NullDimensionSelector from public (#6480)
Roman Leventov [Fri, 2 Nov 2018 11:38:21 +0000 (12:38 +0100)] 
Hide NullDimensionSelector from public (#6480)

5 weeks agofixup docs to download from Apache mirror, fixup tarball name and path, change refere...
David Lim [Fri, 2 Nov 2018 04:47:29 +0000 (22:47 -0600)] 
fixup docs to download from Apache mirror, fixup tarball name and path, change references from quickstart/* to quickstart/tutorial/* (#6570)

5 weeks agocorrect default tier name (#6568)
Caroline1000 [Fri, 2 Nov 2018 00:51:13 +0000 (17:51 -0700)] 
correct default tier name (#6568)

5 weeks agocorrect default value for maxTotalRows (#6566)
QiuMM [Thu, 1 Nov 2018 23:53:15 +0000 (07:53 +0800)] 
correct default value for maxTotalRows (#6566)

5 weeks agoMove supervisor APIs to api-reference (#6555)
Jihoon Son [Thu, 1 Nov 2018 20:10:05 +0000 (13:10 -0700)] 
Move supervisor APIs to api-reference (#6555)

* Move supervisor APIs to api-reference

* fix kafka-specific docs

* add ingestion stats report

5 weeks agoPeriod load/drop/broadcast rules should include the future by default (#6414)
QiuMM [Thu, 1 Nov 2018 16:43:34 +0000 (00:43 +0800)] 
Period load/drop/broadcast rules should include the future by default (#6414)

* Period load/drop/broadcast rules should include the future by default

* address comments

* adjust coordinator console and tweak docs

* address comments

* fix travis-ci

5 weeks agoAdd doc for missing balancerComputeThreads configuration (#6561)
Jihoon Son [Thu, 1 Nov 2018 01:43:12 +0000 (18:43 -0700)] 
Add doc for missing balancerComputeThreads configuration (#6561)

* Add doc for missing balancerComputeThreads configuration

* remove duplicate

5 weeks agoAdd RequestLogEventBuilderFactory (#6477)
Roman Leventov [Wed, 31 Oct 2018 21:24:37 +0000 (22:24 +0100)] 
Add RequestLogEventBuilderFactory (#6477)

This PR allows to control the fields in `RequestLogEvent`, emitted in `EmittingRequestLogger`. In our case, we want to get rid of the `intervals` fields of the query objects that are a part of `DefaultRequestLogEvent`. They are enormous (thousands of segments) and not useful.

Related to #5522, FYI @a2l007.

5 weeks agoSet new keys for integration-tests. (#6554)
Gian Merlino [Wed, 31 Oct 2018 16:01:42 +0000 (09:01 -0700)] 
Set new keys for integration-tests. (#6554)

6 weeks agoUpdate mysql.md (#6545)
taiii [Tue, 30 Oct 2018 21:01:33 +0000 (05:01 +0800)] 
Update mysql.md (#6545)

6 weeks agoProhibit some guava collection APIs and use JDK collection APIs directly (#6511)
QiuMM [Mon, 29 Oct 2018 12:02:43 +0000 (20:02 +0800)] 
Prohibit some guava collection APIs and use JDK collection APIs directly (#6511)

* Prohibit some guava collection APIs and use JDK APIs directly

* reset files that changed by accident

* sort codestyle/druid-forbidden-apis.txt alphabetically

6 weeks agoRemove unused code (#6504)
Samarth Jain [Fri, 26 Oct 2018 18:31:10 +0000 (11:31 -0700)] 
Remove unused code (#6504)

* Remove unused code

* Remove usage of list in setDimensions and setAggregatorSpecs

* Fix formatting to adhere to 120 character guideline

6 weeks agoRemove unused bin/init script, conf-quickstart reference (#6520)
Jonathan Wei [Fri, 26 Oct 2018 18:30:01 +0000 (11:30 -0700)] 
Remove unused bin/init script, conf-quickstart reference (#6520)

6 weeks agofix KafkaSupervisor stats report error (#6508)
Joshua Sun [Thu, 25 Oct 2018 22:45:54 +0000 (15:45 -0700)] 
fix KafkaSupervisor stats report error (#6508)

* fix kafkasupervisor stats 500

* added unit test

* throw error if group already exists

6 weeks agofix exception in Supervisor.start causing overlord unable to become leader (#6516)
Clint Wylie [Thu, 25 Oct 2018 22:44:04 +0000 (15:44 -0700)] 
fix exception in Supervisor.start causing overlord unable to become leader (#6516)

* fix exception thrown by Supervisor.start causing overlord unable to become leader

* fix style

6 weeks agoUpdate init (#6514)
Michael Trelinski [Thu, 25 Oct 2018 20:40:23 +0000 (13:40 -0700)] 
Update init (#6514)

Fix bin/init to source from proper directory.

6 weeks agoFix NPE in TaskLockbox that prevents overlord leadership (#6512)
Clint Wylie [Thu, 25 Oct 2018 20:06:11 +0000 (13:06 -0700)] 
Fix NPE in TaskLockbox that prevents overlord leadership (#6512)

* fix NPE that prevents overlord from assuming leadership if extension that provides indexing task type is not loaded

* heh

6 weeks agoAllow custom TLS cert checks (#6432)
Jonathan Wei [Wed, 24 Oct 2018 23:31:52 +0000 (16:31 -0700)] 
Allow custom TLS cert checks (#6432)

* Allow custom TLS cert checks

* PR comment

* Checkstyle, PR comment

6 weeks agoAdd period drop before rule (#6415)
QiuMM [Wed, 24 Oct 2018 19:44:30 +0000 (03:44 +0800)] 
Add period drop before rule (#6415)

* Add period drop before rule

* add license header

* support period drop before rule in coordinator console

* address comments

7 weeks agoupdated to use the latest sketches-core-0.12.0 (#6381)
Alexander Saydakov [Tue, 23 Oct 2018 18:20:19 +0000 (11:20 -0700)] 
updated to use the latest sketches-core-0.12.0 (#6381)

7 weeks agoCatch some incorrect method parameter or call argument formatting patterns with check...
Roman Leventov [Tue, 23 Oct 2018 10:17:38 +0000 (07:17 -0300)] 
Catch some incorrect method parameter or call argument formatting patterns with checkstyle (#6461)

* Catch some incorrect method parameter or call argument formatting patterns with checkstyle

* Fix DiscoveryModule

* Inline parameters_and_arguments.txt

* Fix a bug in PolyBind

* Fix formatting

7 weeks agoupdate insert pending segments logic to synchronous (#6336)
Faxian Zhao [Tue, 23 Oct 2018 02:48:20 +0000 (10:48 +0800)] 
update insert pending segments logic to synchronous (#6336)

* 1. Mysql default transaction isolation is REPEATABLE_READ, treat it as READ_COMMITTED will reduce insert id conflict.
2. Add an index to 'dataSource used end' is work well for the most of scenarios(get recently segments), and it will speed up sync add pending segments in DB.
3. 'select and insert' is not need within transaction.

* Use TaskLockbox.doInCriticalSection instead of synchronized syntax to speed up insert pending segments.

* fix typo for NullPointerException

7 weeks agoImplement force push down for nested group by query (#5471)
Samarth Jain [Mon, 22 Oct 2018 20:43:47 +0000 (13:43 -0700)] 
Implement force push down for nested group by query (#5471)

* Force nested query push down

* Code review changes

7 weeks agoRemove consumer.listTopics() method in case when too many topics in kafka causes...
elloooooo [Mon, 22 Oct 2018 17:46:31 +0000 (01:46 +0800)] 
Remove consumer.listTopics() method in case when too many topics in kafka causes the FullGC in Overlord (#6455)

* remove consumer.listTopics() method

* add consumerLock and exception handling for consumer.partitionFor() and remove some useless checks

* add check in case consumer.partitionsFor() returns null

* fix CI failure

* fix failed UT

* Revert "fix CI failure"

This reverts commit f839d09e1ed6a3c73bb5ba524b26ee83c0e79c41.

* revert unless commit and re-commit the useful part to fix failed UT

7 weeks agouse a sha512 hash of bloom filter for cache key instead of filter bytes (#6488)
Clint Wylie [Mon, 22 Oct 2018 14:57:21 +0000 (07:57 -0700)] 
use a sha512 hash of bloom filter for cache key instead of filter bytes (#6488)

* use a sha512 hash of bloom filter for cache key instead of filter bytes

* make serde private, BloomDimFilter.toString and BloomDimFilter.equals use hash instead of bloomKFilter which has no tostring or equals of its own

* keep and use HashCode object instead of converting to bytes up front

* uneeded imports oops

* tweaks from review

* refactor dupe code

* refactor

7 weeks agoinclude mysql-metadata-storage extension in distribution, but without… (#6497)
David Lim [Sun, 21 Oct 2018 01:18:58 +0000 (19:18 -0600)] 
include mysql-metadata-storage extension in distribution, but without… (#6497)

* include mysql-metadata-storage extension in distribution, but without the GPL-licensed connector library

* Install mysql connector package

* use symlinks to avoid versioning issues

* add documentation for fetching the mysql connector

7 weeks agoFix integration test service logging (#6479)
Jonathan Wei [Sat, 20 Oct 2018 21:55:14 +0000 (14:55 -0700)] 
Fix integration test service logging (#6479)

7 weeks agofix shallow clone git-commit-id plugin unable to find commits until tag issue (#6495)
Joshua Sun [Sat, 20 Oct 2018 21:45:09 +0000 (14:45 -0700)] 
fix shallow clone git-commit-id plugin unable to find commits until tag issue (#6495)

7 weeks agofix distribution to not include contrib extensions by default, don't pull the entire...
David Lim [Fri, 19 Oct 2018 20:50:05 +0000 (14:50 -0600)] 
fix distribution to not include contrib extensions by default, don't pull the entire AWS SDK bundle (#6494)

7 weeks agoQueryCountStatsMonitor: emit query/count (#6473)
QiuMM [Fri, 19 Oct 2018 13:15:02 +0000 (21:15 +0800)] 
QueryCountStatsMonitor: emit query/count (#6473)

Let `QueryCountStatsMonitor` emit `query/count`, then I can monitor QPS of my services, or I have to count it by myself.

7 weeks agofix TaskRunnerUtils String formatting issue (#6492)
Joshua Sun [Fri, 19 Oct 2018 01:16:46 +0000 (18:16 -0700)] 
fix TaskRunnerUtils String formatting issue (#6492)

* fix TaskRunnerUtils String formatting issue

* additional fixes

7 weeks agoApache-ize POM (#6482)
David Lim [Thu, 18 Oct 2018 01:37:01 +0000 (19:37 -0600)] 
Apache-ize POM (#6482)

* Apache-ize POM

* put revision information into MANIFEST.MF for binary release

* remove nightly profile

* fix flaky travis by overriding maven-remote-resources-plugin execution from the parent POM

7 weeks agoUpgrade lz4-java to 1.5.0 (#6478)
patelh [Thu, 18 Oct 2018 01:34:20 +0000 (18:34 -0700)] 
Upgrade lz4-java to 1.5.0 (#6478)

7 weeks agofix a couple of missed changes that should been in #6443 (#6487)
Clint Wylie [Wed, 17 Oct 2018 23:22:17 +0000 (16:22 -0700)] 
fix a couple of missed changes that should been in #6443 (#6487)

7 weeks agoadd DISCLAIMER file and modify NOTICE file for ASF (#6475)
David Lim [Wed, 17 Oct 2018 15:36:13 +0000 (09:36 -0600)] 
add DISCLAIMER file and modify NOTICE file for ASF (#6475)

* add DISCLAIMER file and modify NOTICE file for ASF

* add line breaks