asterixdb-hyracks.git
6 years agoAdd Compatibility for FileRemoveOperatorDescriptor master
Abdullah Alamoudi [Mon, 4 Apr 2016 20:59:55 +0000 (23:59 +0300)] 
Add Compatibility for FileRemoveOperatorDescriptor

Change-Id: Ic08ba645db6936df3c4c59be0e104dbb18088370
Reviewed-on: https://asterix-gerrit.ics.uci.edu/784
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoASTERIXDB-1377: Prevent Operations from Exiting Components Multiple Times
Murtadha Hubail [Fri, 1 Apr 2016 12:23:25 +0000 (05:23 -0700)] 
ASTERIXDB-1377: Prevent Operations from Exiting Components Multiple Times

Change-Id: I5907b1b4c76ce48366f5447f2940f3561c474cfe
Reviewed-on: https://asterix-gerrit.ics.uci.edu/769
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoEnable Remove Quietly for Remove Directory Jobs
Abdullah Alamoudi [Mon, 4 Apr 2016 06:45:29 +0000 (09:45 +0300)] 
Enable Remove Quietly for Remove Directory Jobs

Change-Id: I00a60bf4ac6d64295224c2cebc28fb22f92e77fd
Reviewed-on: https://asterix-gerrit.ics.uci.edu/775
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoExclude Temporary Indexes From Replication
Murtadha Hubail [Sun, 3 Apr 2016 03:55:44 +0000 (20:55 -0700)] 
Exclude Temporary Indexes From Replication

- Exclude non-durable LSM indexes from replication.
- Stop heartbeat task after NC has stopped to avoid false failures detection.

Change-Id: Icce91a203e04cb068a7a5aa541720bbd0289eacb
Reviewed-on: https://asterix-gerrit.ics.uci.edu/777
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>
6 years agoASTERIXDB-1226: support SQL++ core group-by semantics.
Yingyi Bu [Sun, 27 Mar 2016 00:14:36 +0000 (17:14 -0700)] 
ASTERIXDB-1226: support SQL++ core group-by semantics.

-Fixed introduce group-by combiner rule;
-Fixed deep copy (without new variables) visitor.

Change-Id: I9602681a698925bde4eca33d85df66d06b9a33dd
Reviewed-on: https://asterix-gerrit.ics.uci.edu/751
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoProvide a Way to Create UTF8 Serde with Reader and Writer
Abdullah Alamoudi [Sat, 26 Mar 2016 09:06:52 +0000 (12:06 +0300)] 
Provide a Way to Create UTF8 Serde with Reader and Writer

Change-Id: Ia1a0a2f240ab780e21e7170e03767d8e9981899c
Reviewed-on: https://asterix-gerrit.ics.uci.edu/757
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoAdd Another Signature to Send Messages to CC
Abdullah Alamoudi [Thu, 17 Mar 2016 21:33:16 +0000 (00:33 +0300)] 
Add Another Signature to Send Messages to CC

The existing interface to send messages from NC
to CC expects a byte[]. The new method takes
a Serializable and serialize it.

Change-Id: I0ec4e0898c341723f29af32043cec8c042377ed9
Reviewed-on: https://asterix-gerrit.ics.uci.edu/727
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoFixed Bugs Caused by Disabling Installer Tests
Abdullah Alamoudi [Tue, 22 Mar 2016 11:20:33 +0000 (14:20 +0300)] 
Fixed Bugs Caused by Disabling Installer Tests

Change-Id: Ic77f4ee76a77f4b49cac6835bbb976eb9dc92d3a
Reviewed-on: https://asterix-gerrit.ics.uci.edu/736
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years agoASTERIXDB-1233: Fixed the bulk-loading with an inverted index on an open-type field
Taewoo Kim [Tue, 22 Mar 2016 03:00:56 +0000 (20:00 -0700)] 
ASTERIXDB-1233: Fixed the bulk-loading with an inverted index on an open-type field

Change-Id: I70e90101053d5aa5f5fc2d883135e82c4b7b677a
Reviewed-on: https://asterix-gerrit.ics.uci.edu/739
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ildar Absalyamov <ildar.absalyamov@gmail.com>
6 years agoFix lifecycleMappingMetadata in the pom (for Eclipse).
Yingyi Bu [Fri, 18 Mar 2016 23:41:46 +0000 (16:41 -0700)] 
Fix lifecycleMappingMetadata in the pom (for Eclipse).

Change-Id: I77796920b02f98678083e3107ec8ef1ad6e98fa1
Reviewed-on: https://asterix-gerrit.ics.uci.edu/732
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years agoMerge branch 'release-0.2.17' into master
Ian Maxon [Fri, 18 Mar 2016 01:21:57 +0000 (18:21 -0700)] 
Merge branch 'release-0.2.17' into master

Change-Id: I8fcf41c009f9faf51bc8bccc0c3c7217b7769147

6 years agoASTERIXDB-865: fix for if-else expression.
Yingyi Bu [Wed, 16 Mar 2016 07:38:29 +0000 (00:38 -0700)] 
ASTERIXDB-865: fix for if-else expression.

Change-Id: I17978d2f694e2a5082903002b8388c5bd42811a5
Reviewed-on: https://asterix-gerrit.ics.uci.edu/702
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoASTERIXDB-1343: support heterogeneity of computation nodes and storage nodes.
Yingyi Bu [Wed, 16 Mar 2016 00:49:07 +0000 (17:49 -0700)] 
ASTERIXDB-1343: support heterogeneity of computation nodes and storage nodes.

Change-Id: Ic21d8da2cd457aa17cc9861c0b92ac5960978e03
Reviewed-on: https://asterix-gerrit.ics.uci.edu/718
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoMerge changes necessary for asterix-experiments pkg
Ian Maxon [Thu, 18 Feb 2016 21:51:55 +0000 (13:51 -0800)] 
Merge changes necessary for asterix-experiments pkg

Change-Id: I1256608bbb97d747747325c2050d5a2dc89beeb0
Reviewed-on: https://asterix-gerrit.ics.uci.edu/641
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years agoSupport Change Feeds and Ingestion of Records with MetaData
Abdullah Alamoudi [Tue, 15 Mar 2016 22:58:46 +0000 (01:58 +0300)] 
Support Change Feeds and Ingestion of Records with MetaData

This change allows feeds to perform upserts and deletes
in order to perform replication of an external data source.
The change does so by performing the following:
1. The adapter produces [PK][Record]. (Record == null --> delete)
2. The insert is replaced by an upsert operator.
Change-Id: I3749349e2b9f1b03c8b310eb99d3f44d08be77df
Reviewed-on: https://asterix-gerrit.ics.uci.edu/620
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
6 years agoFix the ExternalGroupby for fuzzy join
Jianfeng Jia [Sat, 12 Mar 2016 08:31:48 +0000 (00:31 -0800)] 
Fix the ExternalGroupby for fuzzy join

Change-Id: I5a3852d08f9a3e2738ea2952f05aa082a2e6dbbb
Reviewed-on: https://asterix-gerrit.ics.uci.edu/713
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoASTERIXDB-1337: Dataset Memory Management on Multi-Partition NC
Michael Blow [Fri, 11 Mar 2016 21:18:53 +0000 (16:18 -0500)] 
ASTERIXDB-1337: Dataset Memory Management on Multi-Partition NC

As sugggested in ASTERIXDB-1337, this change is to maintain a per-
partition MultitenantVirtualBufferCache budget, as opposed to sharing
the budget across the dataset.

Change-Id: Ie554ab6d9b0d01ed149ab0729d20cdac3c969ec1
Reviewed-on: https://asterix-gerrit.ics.uci.edu/711
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoASTERIXDB-1168: fix queries with duplicates in a subplan's input.
Yingyi Bu [Thu, 10 Mar 2016 00:45:49 +0000 (16:45 -0800)] 
ASTERIXDB-1168: fix queries with duplicates in a subplan's input.

Fixed PushSelectDown to consider stateful functions;
Added a primary key visitor to generate/progate primary key information.

Change-Id: I83907c29699a76540abd1a246776f55576eeced8
Reviewed-on: https://asterix-gerrit.ics.uci.edu/695
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoAdded LeftOuterUnnestMap operator.
Taewoo Kim [Tue, 8 Mar 2016 02:43:29 +0000 (18:43 -0800)] 
Added LeftOuterUnnestMap operator.

 - Added LeftOuterUnnestMap operator to represent the left-outer-join semantics properly.

Change-Id: I4525899cf8e5e43551aa2ac2a78806ef6cc85e58
Reviewed-on: https://asterix-gerrit.ics.uci.edu/638
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoASTERIXDB-1127: fix ExtractCommonOperatorsRule.
Yingyi Bu [Mon, 7 Mar 2016 17:33:18 +0000 (09:33 -0800)] 
ASTERIXDB-1127: fix ExtractCommonOperatorsRule.

Change-Id: I16933a4b72432b5fbd523ca80ce6426f6b6743a9
Reviewed-on: https://asterix-gerrit.ics.uci.edu/691
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Pouria Pirzadeh <pouria.pirzadeh@gmail.com>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoMisc. Coverity Issues
Michael Blow [Fri, 4 Mar 2016 00:43:08 +0000 (19:43 -0500)] 
Misc. Coverity Issues

Clean up a few issues encountered in Coverity scan

Change-Id: Ie90a91682ae0ee2443fce1677fca0e00fa45c5f0
Reviewed-on: https://asterix-gerrit.ics.uci.edu/686
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoASTERIXDB-1330: fix NPE in ExternalGroupByOperator.
Yingyi Bu [Fri, 4 Mar 2016 01:16:31 +0000 (17:16 -0800)] 
ASTERIXDB-1330: fix NPE in ExternalGroupByOperator.

Change-Id: I2279221abbef1440179a31df180a24f6a642c641
Reviewed-on: https://asterix-gerrit.ics.uci.edu/687
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years agoFix Double-Checked Locking (Coverity)
Michael Blow [Mon, 29 Feb 2016 19:58:29 +0000 (14:58 -0500)] 
Fix Double-Checked Locking (Coverity)

Fix CIDs 68208,68209,68210 - eliminate problematic double-checked locking

Change-Id: I3383035b5e8e0aa3cfd54f803f12d56090f006cf
Reviewed-on: https://asterix-gerrit.ics.uci.edu/667
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years agoASTERIXDB-1322: fix variable visitors for SubplanOperator.
Yingyi Bu [Wed, 24 Feb 2016 18:33:54 +0000 (10:33 -0800)] 
ASTERIXDB-1322: fix variable visitors for SubplanOperator.

- Simplified the implementation of OperatorPropertiesUtil;
- Fixed PushProjectDownRule to not introduce project operator
  into a subplan to project outer variables.

Change-Id: I53ee1e27b41c9c80d51a7e1baf058d97338c18a9
Reviewed-on: https://asterix-gerrit.ics.uci.edu/662
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoImplemented the memory-bounded HashGroupby and HashJoin for BigObject
Jianfeng Jia [Fri, 26 Feb 2016 02:28:20 +0000 (18:28 -0800)] 
Implemented the memory-bounded HashGroupby and HashJoin for BigObject

It contains both hash grouby and hash join changes.

The main change is
1. update the ExternalGroupby to Hash-based groupby
2. update the Join operators to use the Buffermanager.

The buffer manager part is moved from the Sort package to upper
level so that it can be shared by all the operators.

Change-Id: I248f3a374fdacad7d57e49cf18d8233745e55460
Reviewed-on: https://asterix-gerrit.ics.uci.edu/398
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoFix Remove File Operator
Abdullah Alamoudi [Wed, 24 Feb 2016 15:18:18 +0000 (18:18 +0300)] 
Fix Remove File Operator

The operator didn't get the correct path of the folder to be
deleted. After the fix, the operator gets the correct path
from the IO manager.

Change-Id: I5910409d79e9c1f8e7192a1050b2be0f823e912f
Reviewed-on: https://asterix-gerrit.ics.uci.edu/659
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoRemove Unneeded Suppress Warnings
Abdullah Alamoudi [Tue, 23 Feb 2016 14:14:25 +0000 (17:14 +0300)] 
Remove Unneeded Suppress Warnings

When the classes in this change were created, we were using
deprecated Hadoop library. Since we updated the library,
there is no need to suppress deprecation warnings.

Change-Id: I710c1efbe006ee6de22299f8e6e833a566fb884d
Reviewed-on: https://asterix-gerrit.ics.uci.edu/656
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
Reviewed-by: Till Westmann <tillw@apache.org>
Reviewed-by: Michael Blow <michael.blow@couchbase.com>
6 years agoMore Coverity Fixes
Michael Blow [Wed, 24 Feb 2016 19:32:52 +0000 (14:32 -0500)] 
More Coverity Fixes

- CID 68210 (#1 of 1): Check of thread-shared field evades lock
acquisition (LOCK_EVASION)
6. thread1_overwrites_value_in_field: Thread1 sets index to a new
value. Now the two threads have an inconsistent view of index and
updates to fields of index or fields correlated with index may be
lost.

- CID 68209 (#1 of 1): Check of thread-shared field evades lock
acquisition (LOCK_EVASION)
6. thread1_overwrites_value_in_field: Thread1 sets index to a new
value. Now the two threads have an inconsistent view of index and
updates to fields of index or fields correlated with index may be
lost.

- CID 68208 (#1 of 1): Check of thread-shared field evades lock
acquisition (LOCK_EVASION)
6. thread1_overwrites_value_in_field: Thread1 sets index to a new
value. Now the two threads have an inconsistent view of index and
updates to fields of index or fields correlated with index may be
lost.

- CID 68435 (#1 of 1): UR: Uninitialized read of field in constructor
(FB.UR_UNINIT_READ)
1. defect: Uninitialized read of btreeCmp.

- CID 68416 (#1 of 1): UL: Lock not released on all paths
(FB.UL_UNRELEASED_LOCK_EXCEPTION_PATH)
1. defect:
org.apache.hyracks.storage.am.lsm.invertedindex.inmemory.PartitionedIn
MemoryInvertedIndex.updatePartitionIndexes(short) does not release
lock on all exception paths.

- CID 68415 (#1 of 1): UL: Lock not released on all paths
(FB.UL_UNRELEASED_LOCK)
1. defect:
org.apache.hyracks.storage.common.buffercache.BufferCache.sweepAndFlus
h(int, boolean) does not release lock on all paths.

- CID 68414 (#1 of 1): UL: Lock not released on all paths
(FB.UL_UNRELEASED_LOCK)
1. defect:
org.apache.hyracks.storage.common.buffercache.BufferCache.dumpState()
does not release lock on all paths.

- CID 68412 (#1 of 1): UL: Lock not released on all paths
(FB.UL_UNRELEASED_LOCK)
1. defect:
org.apache.hyracks.storage.am.lsm.common.impls.VirtualBufferCache.dele
teFile(int, boolean) does not release lock on all paths.

- CID 68337 (#1 of 1): RV: Bad use of return value
(FB.RV_ABSOLUTE_VALUE_OF_RANDOM_INT)
1. defect: Bad attempt to compute absolute value of signed random
integer.

- CID 68336 (#1 of 1): RV: Bad use of return value
(FB.RV_ABSOLUTE_VALUE_OF_RANDOM_INT)
1. defect: Bad attempt to compute absolute value of signed random
integer.

- CID 68335 (#1 of 1): RV: Bad use of return value
(FB.RV_ABSOLUTE_VALUE_OF_RANDOM_INT)
1. defect: Bad attempt to compute absolute value of signed random
integer.

- CID 68334 (#1 of 1): RV: Bad use of return value
(FB.RV_ABSOLUTE_VALUE_OF_RANDOM_INT)
1. defect: Bad attempt to compute absolute value of signed random
integer.

- CID 68259 (#1 of 1): Result is not floating-point
(UNINTENDED_INTEGER_DIVISION)
integer_division: Dividing integer expressions 2147483647 and 2, and
then converting the integer quotient to type double. Any remainder,
or fractional part of the quotient, is ignored.

Change-Id: I0f63f7fb75b1ac8ffc1fc1d70e66bb19b788fe8e
Reviewed-on: https://asterix-gerrit.ics.uci.edu/661
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years agoFixes for Coverity Issues
Michael Blow [Wed, 24 Feb 2016 00:58:28 +0000 (19:58 -0500)] 
Fixes for Coverity Issues

(https://scan.coverity.com/projects/hyracks)

- CID 68257 (#1 of 1): Resource leak (RESOURCE_LEAK)
5. leaked_resource: Variable outputStream going out of scope leaks the
 resource it refers to.
- CID 68260 (#1 of 1): Useless call (USELESS_CALL)
side_effect_free: Calling equivalentVars.get(0) is only useful for its
 return value, which is ignored.
- CID 68261 (#1 of 1): BC: Bad casts of object references
 (FB.BC_EQUALS_METHOD_SHOULD_WORK_FOR_ALL_OBJECTS)
1. defect: Equals method for org.apache.hyracks.algebricks.core.
algebra.properties.LocalOrderProperty assumes the argument is of type
 LocalOrderProperty.
- CID 68262 (#1 of 1): BC: Bad casts of object references
(FB.BC_EQUALS_METHOD_SHOULD_WORK_FOR_ALL_OBJECTS)
1. defect: Equals method for org.apache.hyracks.storage.am.rtree.
RTreeCheckTuple assumes the argument is of type RTreeCheckTuple.

Change-Id: I8ba6e578f173b1081a8f55b228704744f08588df
Reviewed-on: https://asterix-gerrit.ics.uci.edu/658
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
Reviewed-by: Till Westmann <tillw@apache.org>
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
6 years agoFix ASTERIXDB-1205: cleanup union related rules.
Yingyi Bu [Tue, 23 Feb 2016 00:04:51 +0000 (16:04 -0800)] 
Fix ASTERIXDB-1205: cleanup union related rules.

-fixed EliminateSubplanWithInputCardinalityOneRule for general cases;
-fixed the type inference invocation in PushAssignBelowUnionAllRule;
-factored out ITypingContext and IVariableContext so that the operator deep copy
 visitor can also be used in the language translator without types;
-added a rule to remove Cartesian product with an ETS input.

Change-Id: I7ad982108a4dc119249222c8ebb9e5897d93783c
Reviewed-on: https://asterix-gerrit.ics.uci.edu/628
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoFix Potential IllegalArgumentException
Michael Blow [Sat, 20 Feb 2016 21:29:14 +0000 (16:29 -0500)] 
Fix Potential IllegalArgumentException

In the event multiple URLs were used for jar deployment,
an IllegalArgumentException would have been thrown.
Replaced problematic reflection call in favor of a sub-
class.

Change-Id: I7c9e1ec2c1541e7593dbc09dac52bfdc9e1cbec6
Reviewed-on: https://asterix-gerrit.ics.uci.edu/649
Reviewed-by: Till Westmann <tillw@apache.org>
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
6 years agoRemove Trailing White Spaces
Abdullah Alamoudi [Fri, 19 Feb 2016 20:38:07 +0000 (23:38 +0300)] 
Remove Trailing White Spaces

This change removes all trailing white spaces in
java files in hyracks

Change-Id: I1c2ea4ff1190a965a857c0c3760cbebdf21860b3
Reviewed-on: https://asterix-gerrit.ics.uci.edu/645
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Michael Blow <michael.blow@couchbase.com>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoPass resource partition id to IModificationOperationCallback
Murtadha Hubail [Sun, 14 Feb 2016 10:19:03 +0000 (02:19 -0800)] 
Pass resource partition id to IModificationOperationCallback

- Pass resource partition id to IModificationOperationCallback
  to include it in txn logs.
- New cluster event NODE_SHUTTING_DOWN for graceful shutdown.
- Remove unnecessary replication STOP operation.

Change-Id: Iddd566ea97512c1dbd2217befdc2bb8822e77763
Reviewed-on: https://asterix-gerrit.ics.uci.edu/589
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>
6 years agoSupport Sending Messages Alongside Frame Data
Abdullah Alamoudi [Wed, 17 Feb 2016 12:23:01 +0000 (15:23 +0300)] 
Support Sending Messages Alongside Frame Data

This change support sending messages with records. The tuple Appender
reserves 100 bytes for a message. Before sending the frame, it appends
The message in the last tuple position. The message is read from the
task context as the shared object between different operators in the
pipeline. The first use of this feature will be within feeds to request
acks for at least once semantics.

Change-Id: I56ae8124052c13a52ca42965b8d00e18ecf35a28
Reviewed-on: https://asterix-gerrit.ics.uci.edu/604
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Michael Blow <michael.blow@couchbase.com>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoASTERIXDB-1272 Remove ExternalLookupOperator
Abdullah Alamoudi [Tue, 16 Feb 2016 12:22:43 +0000 (15:22 +0300)] 
ASTERIXDB-1272 Remove ExternalLookupOperator

ExternalLookupOperator has been removed in this change
and UnnestMapOperator was used instead. different visitors
and methods in the UnnestMapOperator were changed to
support both external and internal data

Change-Id: I86489a96932006bf689ba7c0c7b2f2dd8b0f8be0
Reviewed-on: https://asterix-gerrit.ics.uci.edu/591
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoAllow Project Runtime to Pass Through an Empty Frame
Abdullah Alamoudi [Sun, 14 Feb 2016 07:59:55 +0000 (10:59 +0300)] 
Allow Project Runtime to Pass Through an Empty Frame

Before this change, project runtime expects at least a single record.
Now it can also process an empty frame.

Change-Id: I87dc6eb83a748f7f91610e7d11ebaec9be914e29
Reviewed-on: https://asterix-gerrit.ics.uci.edu/634
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>
6 years agoASTERIXDB-1204: fixed LIMIT pushdown into join
Taewoo Kim [Sun, 14 Feb 2016 18:04:36 +0000 (10:04 -0800)] 
ASTERIXDB-1204: fixed LIMIT pushdown into join

 - Fixed PushMapOperatorDownThroughProductRule not to pushdown LIMIT into a JOIN operator

Change-Id: I19e73c8d444ac0c8ecfcdf3ad3ebe744d6c8d0df
Reviewed-on: https://asterix-gerrit.ics.uci.edu/632
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoIntersect the 2ndary indexes before primary search
Jianfeng Jia [Sat, 13 Feb 2016 18:06:24 +0000 (10:06 -0800)] 
Intersect the 2ndary indexes before primary search

The following commits from your working branch will be included:

Change-Id: Ic16c67c529ca19d8b1a5439ddef22760945fd0d7
Reviewed-on: https://asterix-gerrit.ics.uci.edu/577
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoMove to non-copy-based evaluator interfaces for scalar functions, aggregate functions...
Yingyi Bu [Fri, 12 Feb 2016 17:47:50 +0000 (09:47 -0800)] 
Move to non-copy-based evaluator interfaces for scalar functions, aggregate functions, running aggregate functions and unnest functions.

Change-Id: I92a630550f3d45a7a5f00cfbc93e7b049b06330d
Reviewed-on: https://asterix-gerrit.ics.uci.edu/614
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoASTERIXDB-1157, ASTERIXDB-1051: Pushdown limit
Taewoo Kim [Wed, 10 Feb 2016 17:39:44 +0000 (09:39 -0800)] 
ASTERIXDB-1157, ASTERIXDB-1051: Pushdown limit

- Limit Pushdown into an Order (ExternalSort) operator.
- CopyLimitDownRule doesn't copy LIMIT through UNNESTMAP operator.

Change-Id: I49fb5f38fe8eb4b4419e596a03e2187939d9fd2e
Reviewed-on: https://asterix-gerrit.ics.uci.edu/616
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
6 years agoFixed a Bug in the Register Node Work
Abdullah Alamoudi [Wed, 10 Feb 2016 16:12:23 +0000 (19:12 +0300)] 
Fixed a Bug in the Register Node Work

When registering a node, in order to allow multiple nodes to
share an ip address, we store them in a map <ipaddress,List<nodeNames>>
However, the way we check for the existence of an entry had a bug in it
which causes the entry to never be found.

Change-Id: I39273c2d72cee0f85187febc9a2b1a6086181e66
Reviewed-on: https://asterix-gerrit.ics.uci.edu/624
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years ago[maven-release-plugin] prepare for next development iteration release-0.2.17
Ian Maxon [Wed, 10 Feb 2016 22:55:42 +0000 (14:55 -0800)] 
[maven-release-plugin] prepare for next development iteration

6 years ago[maven-release-plugin] prepare release apache-asterixdb-hyracks-0.2.17-incubating-rc1 apache-asterixdb-hyracks-0.2.17-incubating apache-asterixdb-hyracks-0.2.17-incubating-rc1
Ian Maxon [Wed, 10 Feb 2016 22:55:31 +0000 (14:55 -0800)] 
[maven-release-plugin] prepare release apache-asterixdb-hyracks-0.2.17-incubating-rc1

6 years agoAddress RC0 Comments
Ian Maxon [Wed, 3 Feb 2016 02:16:30 +0000 (18:16 -0800)] 
Address RC0 Comments

Change-Id: I2a6c4fb809b6fcad578007b8f48bf8bce179cd84
Reviewed-on: https://asterix-gerrit.ics.uci.edu/607
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoIPrinters and IAWriters throw HyracksDataException
Till Westmann [Wed, 10 Feb 2016 00:04:27 +0000 (16:04 -0800)] 
IPrinters and IAWriters throw HyracksDataException

instead of AlgebricksException (which should be used at compile-time)

Change-Id: I642ff22a4cc30f1fbf0b61f7b5908a7a0c66da6c
Reviewed-on: https://asterix-gerrit.ics.uci.edu/611
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
6 years agoremove end-of-line whitespace
Till Westmann [Wed, 10 Feb 2016 00:03:14 +0000 (16:03 -0800)] 
remove end-of-line whitespace

Change-Id: If3542874d7f3a536a6fe524aba2fffaf6447a614
Reviewed-on: https://asterix-gerrit.ics.uci.edu/600
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoFix for ASTERIXDB-1291: deals with the case that a NTS' data source operator has...
Yingyi Bu [Tue, 9 Feb 2016 19:42:23 +0000 (11:42 -0800)] 
Fix for ASTERIXDB-1291: deals with the case that a NTS' data source operator has been deep copied.

Change-Id: Ie3efa74e38bbfa9eee4a20e4a9cf6fc2a5840973
Reviewed-on: https://asterix-gerrit.ics.uci.edu/622
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Wenhai Li <lwhaymail@yahoo.com>
Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>
6 years agoRevert "[maven-release-plugin] prepare release apache-asterixdb-hyracks-0.2.17-incuba...
Ian Maxon [Tue, 2 Feb 2016 18:38:56 +0000 (10:38 -0800)] 
Revert "[maven-release-plugin] prepare release apache-asterixdb-hyracks-0.2.17-incubating-rc0"

This reverts commit 4112bf370fac4479b404ca59ef83b3bb9485a4c7.

6 years agoRevert "[maven-release-plugin] prepare for next development iteration"
Ian Maxon [Tue, 2 Feb 2016 18:38:56 +0000 (10:38 -0800)] 
Revert "[maven-release-plugin] prepare for next development iteration"

This reverts commit ba7ed18443fb0789526f9d03af7aa34e953aaa6d.

6 years agoAdd Support for Upsert Operation
Abdullah Alamoudi [Sat, 30 Jan 2016 08:13:47 +0000 (11:13 +0300)] 
Add Support for Upsert Operation

This change adds support for upsert operations. it includes
creating a primary and secondary upsert operators in addition
to adding a new function "before" to the index operation call
back to correctly perform locking for the upsert operation.

Change-Id: I2705f43b6e6d187ee29b9ba5a7946d422990022a
Reviewed-on: https://asterix-gerrit.ics.uci.edu/476
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoAdd flush() to IFrameWriter
Abdullah Alamoudi [Tue, 26 Jan 2016 17:57:52 +0000 (21:57 +0400)] 
Add flush() to IFrameWriter

This change introduces flush() method in frame writers. It is
intended to be used for long running jobs (Feeds) to flush contents
of frames all the way to storage.

Change-Id: I85424bab7965b71aac709280af066e1655457aa3
Reviewed-on: https://asterix-gerrit.ics.uci.edu/584
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoASTERIXDB-1005, ASTERIXDB-1263: clean up subplan flattening, including:
Yingyi Bu [Thu, 28 Jan 2016 03:22:37 +0000 (19:22 -0800)] 
ASTERIXDB-1005, ASTERIXDB-1263: clean up subplan flattening, including:
1. Fixed the data property progation in HashJoin, NestedLoopJoin, PreClusteredGroupBy, and BroadcastExchange;
2. Fixed race conditions in SplitOperatorDescriptor;
3. Added a top-down pass for JobBuilder to set location constraints;
4. Fixed AbstractIntroduceGroupByCombinerRule for general cases.

Change-Id: I0197dc879cf983577e63ea5c047144966c0f7a3c
Reviewed-on: https://asterix-gerrit.ics.uci.edu/572
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoASTERIXDB-1273: Get file references from components to be merged
Murtadha Hubail [Mon, 25 Jan 2016 03:49:56 +0000 (19:49 -0800)] 
ASTERIXDB-1273: Get file references from components to be merged

This change includes the following:
- Get file references from components to be merged instead of file map manager.
- Fix an issue with passing inverted index files to replication job.

Change-Id: I7d0f06c002db86a2a34e6393d06c49fb528c22b7
Reviewed-on: https://asterix-gerrit.ics.uci.edu/596
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: abdullah alamoudi <bamousaa@gmail.com>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years ago[maven-release-plugin] prepare for next development iteration
Ian Maxon [Mon, 25 Jan 2016 10:59:58 +0000 (02:59 -0800)] 
[maven-release-plugin] prepare for next development iteration

6 years ago[maven-release-plugin] prepare release apache-asterixdb-hyracks-0.2.17-incubating-rc0 apache-asterixdb-hyracks-0.2.17-incubating-rc0
Ian Maxon [Mon, 25 Jan 2016 10:59:47 +0000 (02:59 -0800)] 
[maven-release-plugin] prepare release apache-asterixdb-hyracks-0.2.17-incubating-rc0

6 years agoAdd per-binary assembly LICENSE and NOTICE
Ian Maxon [Wed, 20 Jan 2016 23:14:15 +0000 (15:14 -0800)] 
Add per-binary assembly LICENSE and NOTICE

Change-Id: I2a6c4fb809b6fcad578007b8f48bf8bce179cd82
Reviewed-on: https://asterix-gerrit.ics.uci.edu/590
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Chris Hillery <ceej@lambda.nu>
6 years agoFix for ASTERIXDB-1200
Ian Maxon [Fri, 15 Jan 2016 00:23:52 +0000 (16:23 -0800)] 
Fix for ASTERIXDB-1200

Fixes an issue where in the Hyracks integration tests,
if the result size is 0, a NPE is thrown instead of
an assert passing or failing.

Change-Id: Ib519882b9cbca941addcd66232c176a2eaeecc4b
Reviewed-on: https://asterix-gerrit.ics.uci.edu/524
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoFix Error in Aborting Task in Super Activity
Abdullah Alamoudi [Mon, 11 Jan 2016 13:33:15 +0000 (16:33 +0300)] 
Fix Error in Aborting Task in Super Activity

When aborting a task, its thread gets interrupted. This creates
a problem when interrupting

Change-Id: I603d3c101e0a4de4816eb5a6a7fd4320df317ce4
Reviewed-on: https://asterix-gerrit.ics.uci.edu/582
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoStop Network Output Channel Sending Extra Bytes
Abdullah Alamoudi [Mon, 11 Jan 2016 05:57:47 +0000 (08:57 +0300)] 
Stop Network Output Channel Sending Extra Bytes

This change re-introduce an optimization to the output channel
which ensures sending of data bytes only. This optimization
was broken when the big object change was introduced.

Change-Id: I896daf80deb23bcae5d5e934565cf5493ed828ba
Reviewed-on: https://asterix-gerrit.ics.uci.edu/581
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
6 years agoFix for ASTERIXDB-1247
Ian Maxon [Fri, 8 Jan 2016 09:03:31 +0000 (01:03 -0800)] 
Fix for ASTERIXDB-1247

It seems like the root of this is the testing harness closing the index/deleting the file before it has
had a chance to flush all of its pages. There are also some changes to cover potential
corner cases where confiscated pages could be lost, but this doesn't seem to directly
affect the bug.

Change-Id: Ia580242b3f7753fc2f793f879332de3270ee3fee
Reviewed-on: https://asterix-gerrit.ics.uci.edu/575
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoFix licenses in Hyracks binary assemblies
Ian Maxon [Tue, 5 Jan 2016 01:28:27 +0000 (17:28 -0800)] 
Fix licenses in Hyracks binary assemblies

Change-Id: I2a6c4fb809b6fcad578007b8f48bf8bce179cd81
Reviewed-on: https://asterix-gerrit.ics.uci.edu/569
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Chris Hillery <ceej@lambda.nu>
6 years agoReduce Object Creation in Index Operations
Abdullah Alamoudi [Mon, 4 Jan 2016 06:26:39 +0000 (09:26 +0300)] 
Reduce Object Creation in Index Operations

Removed proportioanl object creation in btrees and rtrees during insert and
search operations. Instead, we re-use objects in OpContexts.

Change-Id: I7fab280372951522db02f8c2ff1d7d5b15529cd7
Reviewed-on: https://asterix-gerrit.ics.uci.edu/245
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoImprovment on Cursor for Delimited Data
Abdullah Alamoudi [Sat, 2 Jan 2016 09:55:12 +0000 (12:55 +0300)] 
Improvment on Cursor for Delimited Data

This change allows the parser to parse records in addition to streams.

Change-Id: I84ff40db664633c633277e9cc0ffa534cda9f26a
Reviewed-on: https://asterix-gerrit.ics.uci.edu/567
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoMake Index Path Relative to NC IO Devices
Murtadha Hubail [Thu, 31 Dec 2015 16:30:25 +0000 (08:30 -0800)] 
Make Index Path Relative to NC IO Devices

This change includes the following:
- Store relative index path in persisted local resource as resource name.
- Fetch index absolute path based on FileSplit IO device and index relative path.
This change will enable a node to perform operations on an index that was created
on another node by finding the absolve path based on the node IO deviec and
the index relative path. This is needed in case a node leaves the cluster and
a replica node is assigned to take over and recover the index partitions
that belonged to the failed node.

Change-Id: I09fabd097b8a995ca959fe60286acdb43edfcfa1
Reviewed-on: https://asterix-gerrit.ics.uci.edu/563
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoFix for ASTERIXDB-1018, ASTERIXDB-1017, ASTERIXDB-1019,
Yingyi Bu [Thu, 31 Dec 2015 00:05:14 +0000 (16:05 -0800)] 
Fix for ASTERIXDB-1018, ASTERIXDB-1017, ASTERIXDB-1019,
ASTERIXDB-1020, ASTERIXDB-1029, ASTERIXDB-1030, ASTERIXDB-1034:

1. Let the keys of introduced nested group-bys (group-bys in a subplan) be only the
variables that are produced in the subplan;

2. In PushSelectIntoJoinRule, push independent operators (e.g., a current-datetime() call)
into the first branch from which the join condition refers some variables.

3. In SimpleUnnestToJoinRule, move the boundary between the two join branches of a added join
which results from pipelined datascans to be below operators that doesn't use any variables
(e.g., a current-datetime() call), therefore potentially, the upper (left) join branch
can be rewritten to index lookups.

Change-Id: I18cfa3875d676f71b26e91433ff101a7e725c890
Reviewed-on: https://asterix-gerrit.ics.uci.edu/488
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoASTERIXDB-54: s/IHyracksCommonContext/IHyracksTaskContext/
Till Westmann [Mon, 28 Dec 2015 14:37:36 +0000 (15:37 +0100)] 
ASTERIXDB-54: s/IHyracksCommonContext/IHyracksTaskContext/

Change-Id: I5f2daf1d8f933d86e0254be443f3ebcef17ecbcf
Reviewed-on: https://asterix-gerrit.ics.uci.edu/533
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoAdd Method to Get LSN Byte Offset in LSM Disk Components
Murtadha Hubail [Wed, 23 Dec 2015 04:59:29 +0000 (20:59 -0800)] 
Add Method to Get LSN Byte Offset in LSM Disk Components

The LSN byte offset is needed to support updating LSNs in remote replicas.

Change-Id: I2eb407e607157fe9c71a9cf22b04accc832be31b
Reviewed-on: https://asterix-gerrit.ics.uci.edu/560
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years agoChanges required to support MessageBroker implementation
Murtadha Hubail [Mon, 21 Dec 2015 20:11:20 +0000 (12:11 -0800)] 
Changes required to support MessageBroker implementation

This change includes the following:

- API changes to support MessageBroker implementation.
- IResourceIdFactory interface to support application dependent implementation.

Change-Id: Ib9f49234eebe912c48e7f71980433a9b42595741
Reviewed-on: https://asterix-gerrit.ics.uci.edu/485
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoASTERIXDB-1105 - LICENSE per submodule
Ian Maxon [Wed, 25 Nov 2015 15:38:37 +0000 (07:38 -0800)] 
ASTERIXDB-1105 - LICENSE per submodule

This is a good part of what we need to have a specific set of
dependencies per binary assembly. The main key here is using the
maven remote resources plugin, and supplemental resources to
fill in info from the POM of dependencies where it is lacking.
For things that aren't described in Maven dependencies,
like Javascript files, they are added as LICENSE fragments
that get included in binary assemblies.

Change-Id: I2a6c4fb809b6fcad578007b8f48bf8bce179cd83
Reviewed-on: https://asterix-gerrit.ics.uci.edu/520
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years agoChanged the IFrameWriter Contract
Abdullah Alamoudi [Wed, 16 Dec 2015 06:04:10 +0000 (09:04 +0300)] 
Changed the IFrameWriter Contract

Updated existing operators and added a test case for BTreeSearchOperatorNodePushable.
With this change, calling the open method itself moves it to the open state and
hence, close must be called.

Change-Id: I03da090002f79f4db7b5b31454ce3ac2b9e40c7f
Reviewed-on: https://asterix-gerrit.ics.uci.edu/551
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoFix ASTERIXDB-1201
Ian Maxon [Mon, 14 Dec 2015 22:56:43 +0000 (14:56 -0800)] 
Fix ASTERIXDB-1201

The reason the nulls were hitting the bulk loader was because
the filter was always set to null. It seems to work fine when
it is passed in just as it is in IndexInsertDeletePOperator

Change-Id: I0fbfaeb8a98316f4d148285832ad31d991481e4b
Reviewed-on: https://asterix-gerrit.ics.uci.edu/553
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>
6 years agoDisable non-JDK 8 build profiles.
Ian Maxon [Wed, 9 Dec 2015 20:35:33 +0000 (12:35 -0800)] 
Disable non-JDK 8 build profiles.

We use Java 8 specific features now, so we shouldn't bother
with any profiles for compiler settings older than that.

Change-Id: I97b03f76bd7afa66e9be5a20ff64477383525868
Reviewed-on: https://asterix-gerrit.ics.uci.edu/545
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
Reviewed-by: Chris Hillery <ceej@lambda.nu>
6 years agoASTERIXDB-1198: make ISerializerDeserializer implementations in Hyracks stateless.
Yingyi Bu [Thu, 26 Nov 2015 00:14:32 +0000 (16:14 -0800)] 
ASTERIXDB-1198: make ISerializerDeserializer implementations in Hyracks stateless.

Change-Id: I1ec86d0a93d8f15d88d68fab24dbe858c5ba8842
Reviewed-on: https://asterix-gerrit.ics.uci.edu/521
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>
6 years agoAdd a test case for Hyracks error reporting.
Yingyi Bu [Wed, 2 Dec 2015 20:05:35 +0000 (12:05 -0800)] 
Add a test case for Hyracks error reporting.

Change-Id: Ic3ac4c7b0a07eacbc448ce5724085eb8d1e6bc4d
Reviewed-on: https://asterix-gerrit.ics.uci.edu/527
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoASTERIXDB-1206: call initialize()/deinitialize() in parallel for OperatorNodePushable...
Yingyi Bu [Tue, 1 Dec 2015 23:34:17 +0000 (15:34 -0800)] 
ASTERIXDB-1206: call initialize()/deinitialize() in parallel for OperatorNodePushables in SuperActivityOperatorNodePushable.

Change-Id: I8700d5258d658ebbf711b4233bb0def1e8cf7c39
Reviewed-on: https://asterix-gerrit.ics.uci.edu/526
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>
6 years agoFix ASTERIXDB-1108
Ian Maxon [Tue, 24 Nov 2015 00:38:03 +0000 (16:38 -0800)] 
Fix ASTERIXDB-1108

Just adding some missing licenses, for jQuery plugins and text examples in
the Hyracks textserver examples.

Change-Id: I298cf9a5204a25e8556fb00c4577d1f15675ea46
Reviewed-on: https://asterix-gerrit.ics.uci.edu/513
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoAdd a constructor for FileSplit: FileSplit(String nodeName, String path, int ioDeviceId).
Yingyi Bu [Wed, 25 Nov 2015 01:01:55 +0000 (17:01 -0800)] 
Add a constructor for FileSplit: FileSplit(String nodeName, String path, int ioDeviceId).

change-Id: I1c7cc7752dd7b6d5261ff02d591f25d76d442a1c
Reviewed-on: https://asterix-gerrit.ics.uci.edu/324
Reviewed-by: Till Westmann <tillw@apache.org>
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
6 years agoFix ASTERIXDB-1104
Ian Maxon [Tue, 24 Nov 2015 21:04:19 +0000 (13:04 -0800)] 
Fix ASTERIXDB-1104

This changes the wrapper hyracks-ecosystem POM to a name that makes
more sense for distributing as a source tarball. Nothing depends on
this artifact so changing its name should not be a big deal.

Change-Id: Id3a57fe0a26957e42ced6be4ecb8a39a392f16bf
Reviewed-on: https://asterix-gerrit.ics.uci.edu/516
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoFix ASTERIXDB-1192
Ian Maxon [Mon, 23 Nov 2015 21:50:16 +0000 (13:50 -0800)] 
Fix ASTERIXDB-1192

Change-Id: I3fcdd584d761d0d06a30aad41ea2a14438bb43eb
Reviewed-on: https://asterix-gerrit.ics.uci.edu/507
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoAdd method to check state of buffer cache
Ian Maxon [Fri, 13 Nov 2015 21:58:04 +0000 (13:58 -0800)] 
Add method to check state of buffer cache

Change-Id: I80fb891b5310252143854a336b591bf3f8cd4ba8
Reviewed-on: https://asterix-gerrit.ics.uci.edu/497
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoMake LSM bulkload append-only and write-once.
Ian Maxon [Fri, 20 Nov 2015 00:07:22 +0000 (16:07 -0800)] 
Make LSM bulkload append-only and write-once.

Allows for usage of LSM indexes with underlying storage that is append-only.
This also results in a small improvement for LSM component bulk load speed.

- Tree metadata (filters, etc) now lie at the back of the tree file in
  append-only mode.
-- Note that you should *not* ever give the append-only flag on bulk-load,
   if the tree is ever to be modified in place.
- Append-only operations bypass the buffer cache for writes, but utilize
  the buffer cache for memory allocation and reads.
- Addresses ASTERIXDB-1059

Change-Id: I80fb891b5310252143854a336b591bf3f8cd4ba7
Reviewed-on: https://asterix-gerrit.ics.uci.edu/255
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Young-Seok Kim <kisskys@gmail.com>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
6 years agoASTERIXDB-1109: Fixed deletion of records from open secondary index
Ildar Absalyamov [Thu, 12 Nov 2015 01:27:38 +0000 (17:27 -0800)] 
ASTERIXDB-1109: Fixed deletion of records from open secondary index
 - Changed printers for idx\dataset insert\delete operators
 - Fixed visiting policy for idx insert\delete operator

Change-Id: Ib2036d2eac4b0a0c6ac2c2e7e1bac383b11106bd
Reviewed-on: https://asterix-gerrit.ics.uci.edu/462
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoOverride the reset method in ByteArrayAccessibleOutputStream.
Yingyi Bu [Wed, 18 Nov 2015 06:34:41 +0000 (22:34 -0800)] 
Override the reset method in ByteArrayAccessibleOutputStream.

Change-Id: Ib43f0b0fd9291b4a7e08963abb850d368ae2c288
Reviewed-on: https://asterix-gerrit.ics.uci.edu/500
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoASTERIXDB-1137: Prevent file from being deleted while a page is being flushed.
Murtadha Hubail [Tue, 17 Nov 2015 07:09:47 +0000 (23:09 -0800)] 
ASTERIXDB-1137: Prevent file from being deleted while a page is being flushed.

Change-Id: I256a84c416bf9d7251b1ee788d028d6b2928e747
Reviewed-on: https://asterix-gerrit.ics.uci.edu/499
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years agoSQL++ support in Algebricks:
Yingyi Bu [Mon, 16 Nov 2015 06:28:05 +0000 (22:28 -0800)] 
SQL++ support in Algebricks:
1. added the OuterUnnestOperator;
2. fixed several rewriting rules.

Change-Id: I7dcf57f75ebc0a741b6ec9597525e226b6014fc0
Reviewed-on: https://asterix-gerrit.ics.uci.edu/314
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoASTERIXDB-1145: Fix error propagating in operators/connectors:
Yingyi Bu [Fri, 13 Nov 2015 05:16:26 +0000 (21:16 -0800)] 
ASTERIXDB-1145: Fix error propagating in operators/connectors:
1. When an AbstractUnarySourceOperator instance runs into an exception, it should call writer.fail() first and then throw the exception.
2. An IFrameWriter.fail() implementation should not throw yet-another exception, instead, it should just propgate
   the failure to its downstream operators and optionally set a "failed" state so that in the close()/nextFrame() method
   it can potentially behave differently from usual close()/nextFrame().

Change-Id: Ifb538155423687c4aa01a0485adeaab87f291547
Reviewed-on: https://asterix-gerrit.ics.uci.edu/491
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoASTERIXDB-1145: Fix error reporting in AlgebricksMetaOperatorDescriptor to avoid...
Yingyi Bu [Thu, 12 Nov 2015 19:29:48 +0000 (11:29 -0800)] 
ASTERIXDB-1145: Fix error reporting in AlgebricksMetaOperatorDescriptor to avoid undeterministic behaviors.

Change-Id: I07b5f0ab95f3b8feda14b7f6b0f591c237c8b4d3
Reviewed-on: https://asterix-gerrit.ics.uci.edu/487
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoASTERIXDB-1058: ensure memory availablity before allocation
Murtadha Hubail [Wed, 11 Nov 2015 07:59:58 +0000 (23:59 -0800)] 
ASTERIXDB-1058: ensure memory availablity before allocation

This change includes the following:
- IResourceMemeoryManager responsible for resources memory budget calculation.
- ResourceHeapBufferAllocator which checks with IResourceManager before allocation.

Change-Id: I846ff45402410835f5ed0afd2f701509abc95222
Reviewed-on: https://asterix-gerrit.ics.uci.edu/480
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years agoASTERIXDB-1058: Lazy LSM memory components allocation
Murtadha Hubail [Sat, 31 Oct 2015 06:54:11 +0000 (23:54 -0700)] 
ASTERIXDB-1058: Lazy LSM memory components allocation

Change-Id: I476e756f8d71260ea614c8c072fc9503053866c9
Reviewed-on: https://asterix-gerrit.ics.uci.edu/405
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
Reviewed-by: Young-Seok Kim <kisskys@gmail.com>
6 years agoThis change fixes the sporadic connection refused errors after managix starts up...
Abdullah Alamoudi [Fri, 6 Nov 2015 12:30:18 +0000 (15:30 +0300)] 
This change fixes the sporadic connection refused errors after managix starts up an Asterix instance. This can mostly be seen during asterix-installer tests.
The fix basically lets managix wait and listens through zookeeper for a signal from the cluster controller. Once the cluster controller sends the signal, Managix can proceed.

Change-Id: I0975308e1dc2650d8c2e50d58ff2c696b22eaaa4
Reviewed-on: https://asterix-gerrit.ics.uci.edu/365
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoASTERIXDB-1102: VarSize Encoding to store length of String and ByteArray
JavierJia [Fri, 23 Oct 2015 20:49:32 +0000 (13:49 -0700)] 
ASTERIXDB-1102: VarSize Encoding to store length of String and ByteArray

This patch is to change the encoding format that stores the length value of
the variable length type (e.g. String, ByteArray) from fix-size encoding
(2bytes) to variable-size encoding ( 1 to 5bytes)

It will solve the issue 1102 to enable us to store a String that longer
than 64K. Also for the common case of storing the short string ( <=
127), it will save one byte per string.

Some important changes include:
1. Add one hyracks-util package to consolidate all the hyracks
independent utility functions. It will reduce the chances of having
duplicate utils in different packages.
2. Move parts of Asterix string functions down to Hyracks
UTF8StringPointable object, which will benefit the other dependencies,
such as VXQuery.

Change-Id: I7e95df0f06984b784ebac2c84b97e56a50207d27
Reviewed-on: https://asterix-gerrit.ics.uci.edu/449
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Taewoo Kim <wangsaeu@gmail.com>
Reviewed-by: Jianfeng Jia <jianfeng.jia@gmail.com>
6 years agoAdapted Inline Variable rules to allow functions that are treated as constant at...
Steven Jacobs [Mon, 19 Oct 2015 22:38:07 +0000 (15:38 -0700)] 
Adapted Inline Variable rules to allow functions that are treated as constant at runtime to be inlineable

Change-Id: Ib990773ec36a3f51abef72ce6ceb7715aa1d5e37
Reviewed-on: https://asterix-gerrit.ics.uci.edu/368
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoFixed m2e lifecycle mapping pom issues
Ildar Absalyamov [Thu, 15 Oct 2015 06:11:06 +0000 (23:11 -0700)] 
Fixed m2e lifecycle mapping pom issues

Change-Id: Ifc8927a7fe90577d787c3d30c371ea95dd4aa24e
Reviewed-on: https://asterix-gerrit.ics.uci.edu/453
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years agoASTERIXDB-1136: Make Java compiler target configurable via profile
Ian Maxon [Wed, 14 Oct 2015 17:45:19 +0000 (10:45 -0700)] 
ASTERIXDB-1136: Make Java compiler target configurable via profile

Change-Id: I10378f9d3fdec9b4951f53c9b370b7741668e537
Reviewed-on: https://asterix-gerrit.ics.uci.edu/273
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
6 years agoMoved LSMOperationType to LSM API package
Murtadha Hubail [Sat, 10 Oct 2015 04:52:33 +0000 (21:52 -0700)] 
Moved LSMOperationType to LSM API package

Change-Id: Ib6f0b7373388fc88605188e5a8089bd183d23af1
Reviewed-on: https://asterix-gerrit.ics.uci.edu/447
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoASTERIXDB-139: Add temp workspace files deletion to IOManager
Murtadha Hubail [Tue, 6 Oct 2015 06:31:42 +0000 (23:31 -0700)] 
ASTERIXDB-139: Add temp workspace files deletion to IOManager

This change includes the following:
- Add a method to delete temp workspace files (WAF)
- Expose LSMComponents files suffixes to Asterix

Change-Id: I760074764755e7aee100ff33c14b13bf4b29ec2e
Reviewed-on: https://asterix-gerrit.ics.uci.edu/337
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoASTERIXDB-1053: change IndexLifecycleManager API to use resource name
Murtadha Hubail [Thu, 8 Oct 2015 23:14:24 +0000 (16:14 -0700)] 
ASTERIXDB-1053: change IndexLifecycleManager API to use resource name

Change-Id: I2e8e974fc2f746959639ce94351f8e419a7f9093
Reviewed-on: https://asterix-gerrit.ics.uci.edu/343
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoASTERIXDB-1002: Fix exception handling in EmptyTupleSourceRuntimeFactory
Ildar Absalyamov [Fri, 9 Oct 2015 07:06:31 +0000 (00:06 -0700)] 
ASTERIXDB-1002: Fix exception handling in EmptyTupleSourceRuntimeFactory

Revisiting the previous fix by calling fail() on pipeline

Change-Id: I19f8c8485e483e4d4efeff939e6bd82c7a04a101
Reviewed-on: https://asterix-gerrit.ics.uci.edu/443
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <hubailmor@gmail.com>
Reviewed-by: Ian Maxon <imaxon@apache.org>
6 years agoASTERIXDB-221: reduce unneceesary partitioning for hash joins.
Yingyi Bu [Fri, 25 Sep 2015 04:06:28 +0000 (21:06 -0700)] 
ASTERIXDB-221: reduce unneceesary partitioning for hash joins.

For a hash join, start top-down data property optimization from
a partitioning-compatiable child, and hence the other child's
partitioning requirement could be updated.

Change-Id: I835ea712c2f427149d45464fcb3841b8d33f6507
Reviewed-on: https://asterix-gerrit.ics.uci.edu/395
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Wenhai Li <lwhaymail@yahoo.com>
Reviewed-by: Yingyi Bu <buyingyi@gmail.com>
6 years agoASTERIXDB-1077: Fixed - Inverted index tests are slow compared to others
Taewoo Kim [Thu, 8 Oct 2015 18:59:49 +0000 (11:59 -0700)] 
ASTERIXDB-1077: Fixed - Inverted index tests are slow compared to others

- Reduced the number of searches during the inverted index searches
- Reduced the number of Jaccard search modifiers

Change-Id: I1d8fc1d30055798fee7f3f23d08a57af867e3a3c
Reviewed-on: https://asterix-gerrit.ics.uci.edu/442
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Young-Seok Kim <kisskys@gmail.com>
Reviewed-by: Ian Maxon <imaxon@apache.org>