kudu.git
9 hours agoKUDU-2636: LBM supports deleting dead containers master
helifu [Sat, 22 Dec 2018 14:03:20 +0000 (22:03 +0800)] 
KUDU-2636: LBM supports deleting dead containers

this patch intends to support deleting the full containers
that are dead at runtime. It can help to ease disk pressure,
save tserver startup time and etc. It uses the container
size and live blocks to determine whether the container
is dead or live.

Step1: wrapping up containers in scoped_refptr. [V]
Step2: supporting dead containers removal.      [V]

Change-Id: I7d3672ae3c3dd9acd489120d653c44a641537f10
Reviewed-on: http://gerrit.cloudera.org:8080/12075
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Hao Hao <hao.hao@cloudera.com>
38 hours agodocs: Add Gerrit HTTP endpoint instructions
Mike Percy [Fri, 11 Jan 2019 16:44:40 +0000 (08:44 -0800)] 
docs: Add Gerrit HTTP endpoint instructions

Some users cannot use SSH to connect to Gerrit. We should provide
explicit instructions for these users on how to submit a patch via
Gerrit so they don't have to figure it out themselves from the Gerrit
docs.

Change-Id: I70eb2abf6d4c11500d50be27e398bdf0cb3f85c2
Reviewed-on: http://gerrit.cloudera.org:8080/12218
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
3 days agoclient: avoid accessing state after scheduling retry
Andrew Wong [Sun, 6 Jan 2019 06:22:19 +0000 (22:22 -0800)] 
client: avoid accessing state after scheduling retry

I saw a failure of MasterMigrationTest.TestEndToEndMigration in which
the stack watchdog logged the following stacktrace:

User stack:
    @     0x7fa3e575c330  (unknown) at ??:0
    @           0x52dc09  __sanitizer::internal_read() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_syscall_linux_x86_64.inc:46 (discriminator 7)
    @           0x52f79f  __sanitizer::ReadFromFile() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_posix.cc:176
    @           0x53acb9  __sanitizer::SymbolizerProcess::ReadFromSymbolizer() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cc:468
    @           0x53ba76  __sanitizer::SymbolizerProcess::SendCommand() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cc:445
    @           0x53c3c5  __sanitizer::Symbolizer::SymbolizePC() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cc:356
    @           0x539673  __sanitizer::StackTrace::Print() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cc:36
    @           0x541c44  MaybePrintStackTrace() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/ubsan/ubsan_diag.cc:48
    @           0x5422ee  __ubsan::ScopedReport::~ScopedReport() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/ubsan/ubsan_diag.cc:73
    @           0x549608  HandleDynamicTypeCacheMiss() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/ubsan/ubsan_handlers_cxx.cc:81
    @           0x549a43  __ubsan_handle_dynamic_type_cache_miss_abort at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/ubsan/ubsan_handlers_cxx.cc:93
    @     0x7fa3f0086643  _ZZN4kudu6client8internal20AsyncLeaderMasterRpcINS_6master23GetTableSchemaRequestPBENS3_24GetTableSchemaResponsePBEE27RetryOrReconnectIfNecessaryEPNS_6StatusEENKUlvE_clEv at ??:0
    @     0x7fa3f0059b8c  _ZN4kudu13ScopedCleanupIZNS_6client8internal20AsyncLeaderMasterRpcINS_6master23GetTableSchemaRequestPBENS4_24GetTableSchemaResponsePBEE27RetryOrReconnectIfNecessaryEPNS_6StatusEEUlvE_ED2Ev at ??:0
    @     0x7fa3f005945c  kudu::client::internal::AsyncLeaderMasterRpc<>::RetryOrReconnectIfNecessary() at ??:0
    @     0x7fa3f0057d0a  kudu::client::internal::AsyncLeaderMasterRpc<>::SendRpcCb() at ??:0
    @     0x7fa3f0085a85  _ZNSt5_BindIFSt7_Mem_fnIMN4kudu6client8internal20AsyncLeaderMasterRpcINS1_6master23GetTableSchemaRequestPBENS5_24GetTableSchemaResponsePBEEEFvRKNS1_6StatusEEEPS8_S9_EE6__callIvJEJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE at ??:0
    #19 0x7fa3ddc84ffc in clone sysdeps/unix/sysv/linux/x86_64/clone.S:111

Upon looking into this and similar failures, it seems like the following
series of events is possible:

T1: allocate an AsyncLeaderMasterRpc on the stack with
    Synchronizer.Wait() as the user-specified callback
T2: the RPC call results in an error that makes it retry via
    RetryOrReconnectIfNecessary(); a retry is scheduled
T3: the retry completes and calls the user-specified callback to begin
    execution on T1
T1: with the RPC call completed, destroys RPC object
T2: the KLOG messages attempt to access state from the destroyed RPC
    object and hit a crash, race, undefined behavior, etc.

The fix is to make a copy of the RPC's state that was to be logged so
there's no chance that T1's destruction of the RPC will affect T2. I
tested this by looping a test[1] that repeatedly called
Client::IsCreateTableInProgress() on a multi-master cluster configured
to have election storms, and observing 5/1000 that yielded TSAN errors
around the logging calls in RetryOrReconnectIfNecessary. With this
patch, 1000/1000 pass.

[1] https://gist.github.com/andrwng/5d552f75a2e0d671b7ed54dd01892c66

Change-Id: I8cefd9613018247a1a25d17adedc021e8be166f6
Reviewed-on: http://gerrit.cloudera.org:8080/12170
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
6 days ago[docs] Add docs for rack/location-awareness
Will Berkeley [Fri, 11 Jan 2019 20:01:06 +0000 (12:01 -0800)] 
[docs] Add docs for rack/location-awareness

I stuck to the name "rack-awareness" throughout, preferring it to the
term we have used internally, "location-awareness", because even though
the latter term is more technically correct, the former is the more
typical and colloquial name for this sort of thing.

These docs borrow heavily from the WIP blog post written by Alexey.

Change-Id: I505b7fd545a4253170a5b7cac63c33e7d9669490
Reviewed-on: http://gerrit.cloudera.org:8080/12219
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <awong@cloudera.com>
7 days agoDo not fallthrough to next case upon parsing "\n" in JSON
Mike Parker [Fri, 11 Dec 2015 08:24:36 +0000 (00:24 -0800)] 
Do not fallthrough to next case upon parsing "\n" in JSON

Consequently an escaped "\r" would also be written out.

Change-Id: I4827aa68f384bb30b8a03011b989ee775a317668
Reviewed-on: http://gerrit.cloudera.org:8080/1623
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy <mpercy@apache.org>
7 days agoKUDU-2640: Add Spark Structured Streaming Sink
Grant Henke [Fri, 14 Dec 2018 17:03:31 +0000 (11:03 -0600)] 
KUDU-2640: Add Spark Structured Streaming Sink

This patch adds a KuduSink and implements the
Spark StreamSinkProvider interface to support
structured streaming writes to Kudu.

These writes behave the same as writes performed
via the InsertableRelation interface.

Note: The StreamSinkProvider interface is
marked as experimental and unstable, but it has existed
since Spark 2.0.0 and is used by the Kafka integration.
Additionally, per SPARK-26415, there will be no
more Spark 2.x minor releases and Spark would
not break this API in a maintenance release. This
means the interface is effectively stable for all
of Spark 2.

Change-Id: I731e35f82c8cca7d911e4d879aa6853112132b17
Reviewed-on: http://gerrit.cloudera.org:8080/12087
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <awong@cloudera.com>
7 days ago[spark-tools] Fix DistributedDataGenerator num-tasks
Grant Henke [Thu, 10 Jan 2019 02:48:28 +0000 (20:48 -0600)] 
[spark-tools] Fix DistributedDataGenerator num-tasks

This patch fixes the num-tasks argument. Previously
Spark was not creating the correct number of tasks
to match the passed argument.

Change-Id: I7fa560e71b2f84a75002e9e776011e1a11c5a1ff
Reviewed-on: http://gerrit.cloudera.org:8080/12211
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
7 days ago[backup] Make the scanRequestTimeout unit clear
Grant Henke [Wed, 9 Jan 2019 21:38:57 +0000 (15:38 -0600)] 
[backup] Make the scanRequestTimeout unit clear

Changes the scanRequestTimeout parameter to be
scanRequestTimeoutMs to make sure the time unit is
clear.

Change-Id: Id44832a06ae5fb4762a402faff82012729afb5c4
Reviewed-on: http://gerrit.cloudera.org:8080/12210
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
7 days ago[backup] Workaround KUDU-1868 errors
Grant Henke [Wed, 9 Jan 2019 21:27:17 +0000 (15:27 -0600)] 
[backup] Workaround  KUDU-1868 errors

As a workaround for KUDU-1868 the socketReadTimeout is
matched to the scanRequestTimeout. Without this
"Invalid call sequence ID" errors can occur under heavy load.

Change-Id: I9fb16cb5b2d4518534e3752526fa830161fe13b6
Reviewed-on: http://gerrit.cloudera.org:8080/12209
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
7 days ago[examples] fix name of the class for spark-submit
Alexey Serbin [Thu, 10 Jan 2019 01:01:01 +0000 (17:01 -0800)] 
[examples] fix name of the class for spark-submit

I also updated the description of master RPC endpoints and
removed the redundant default port number for masters' RPC
endpoints in the example.

Change-Id: Ic36a6fb62bd55667ea9f11c0124b5388f0e5bc7b
Reviewed-on: http://gerrit.cloudera.org:8080/12206
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <granthenke@apache.org>
Reviewed-by: Hao Hao <hao.hao@cloudera.com>
8 days agoFix DOS line endings in TestServerInfo.java
Will Berkeley [Tue, 8 Jan 2019 18:32:18 +0000 (10:32 -0800)] 
Fix DOS line endings in TestServerInfo.java

Change-Id: Id0c470ed24238e0461e96bd443f3d649e1e11acb
Reviewed-on: http://gerrit.cloudera.org:8080/12180
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <granthenke@apache.org>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
8 days agoSupport location awareness in READ_CLOSEST for the Java client
Will Berkeley [Wed, 26 Dec 2018 21:16:53 +0000 (16:16 -0500)] 
Support location awareness in READ_CLOSEST for the Java client

Change-Id: Ief0f07058cefd0037f4b0f7c60c8b7809dc8313f
Reviewed-on: http://gerrit.cloudera.org:8080/12175
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <granthenke@apache.org>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
8 days agoEliminate redundant VLOG_IS_ON calls
Will Berkeley [Tue, 8 Jan 2019 22:20:42 +0000 (14:20 -0800)] 
Eliminate redundant VLOG_IS_ON calls

In an expression like

  VLOG(3) << Substitute("Your $0 is $1", expensive_foo(), costly_bar());

The expression on the righthand side of the << operator is evaluated
only when VLOG_IS_ON(3) is true. However, there were some cases of
defensive programming like

  if (VLOG_IS_ON(3)) {
    VLOG(3) << Substitute("Your $0 is $1", expensive_foo(), costly_bar());
  }

with a redundant call to VLOG_IS_ON. This patch removes such cases as I
found from a simple review of all VLOG_IS_ON callsites. I also sometimes
transformed VLOG messages from a chain of <<'s to calls to Substitute.

This patch contains no functional changes.

Change-Id: I0781e64f95e33fe067f9f3e65e77aec8654f4ba3
Reviewed-on: http://gerrit.cloudera.org:8080/12194
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
8 days agoAssign locations to tablet servers and the client in Java
Will Berkeley [Wed, 26 Dec 2018 21:16:10 +0000 (16:16 -0500)] 
Assign locations to tablet servers and the client in Java

Change-Id: I9e2c74ab12f7da187bf6e75d42a3089bc20235db
Reviewed-on: http://gerrit.cloudera.org:8080/12174
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <granthenke@apache.org>
8 days agoReduce election-related logging
Will Berkeley [Wed, 9 Jan 2019 00:03:14 +0000 (16:03 -0800)] 
Reduce election-related logging

Frequent simultaneous pre-elections on lots of tablets are a common
symptom of underlying problems on Kudu clusters. This results in a ton
of log spam because a [pre-]election results in multiple log messages.

This patch makes a few conservative changes to reduce the amount of
election-related logging, especially on candidates.

1. No log message per successful VoteRequest response. The candidate
   previously logged a message for every granted or denied vote from a
   voter, e.g.

     I1221 20:16:10.479441  1852 leader_election.cc:387] T cdd0e560bc5c451c978e38c0d7620e52 P dd658299c5194f4b88ae408cd3130103 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1a228ec03dbc4bb79ef66cd036a99bca (127.1.92.67:38265)

   This seems like overkill. Instead, these messages have been moved to
   VLOG(1) and the candidate will print a more detailed summary of the
   election once it is decided:

      I0109 12:39:42.203627 158707712 leader_election.cc:304] T 00000000000000000000000000000000 P 99b33389ebdc47b1ae18947dea49a5b7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 465a338e21d84ad1a4e03867eb60862399b33389ebdc47b1ae18947dea49a5b7; no voters:

   Compared to the current state:

     I1221 20:16:10.479472  1852 leader_election.cc:278] T cdd0e560bc5c451c978e38c0d7620e52 P dd658299c5194f4b88ae408cd3130103 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won.

Note that it is still possible to find a record of each vote in each
voter's INFO logs, and error condition logging has not been changed.

2. No log message per vote request. The candidate previously logged a
   message for each voter it was requesting a vote from. e.g.

     I1221 20:16:10.474704  1993 leader_election.cc:251] T cdd0e560bc5c451c978e38c0d7620e52 P 1a228ec03dbc4bb79ef66cd036a99bca [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer dd658299c5194f4b88ae408cd3130103 (127.1.92.66:39093)
     I1221 20:16:10.475266  1993 leader_election.cc:251] T cdd0e560bc5c451c978e38c0d7620e52 P 1a228ec03dbc4bb79ef66cd036a99bca [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c624ece2977e486fbab958afb000f122 (127.1.92.65:45651)

   Instead, the candidate will summarize all the voters it sent requests
   to:

     I0109 12:39:42.206408 156561408 leader_election.cc:291] T 00000000000000000000000000000000 P 99b33389ebdc47b1ae18947dea49a5b7 [CANDIDATE]: Term 2 election: Requested vote from peers 465a338e21d84ad1a4e03867eb608623 (127.0.0.1:7053), 08b8de9c7a7c4ec5b72c9eb275d29cfd (127.0.0.1:7052)

3. No logging of failure detector sleeps. It seems like it might be
   useful to log when the FD sleeps, and why, e.g.

     I1221 20:16:12.399258  2803 raft_consensus.cc:2834] T cdd0e560bc5c451c978e38c0d7620e52 P c624ece2977e486fbab958afb000f122: Snoozing failure detection for 0.224s (vote granted)

   But frankly in practice it's never been relevant or useful. Since a
   message like this is produced multiple times on the candidate and
   possibly on the voters during an election, I moved it to VLOG(1)
   level to reduce the total amount of logging from elections.

Change-Id: Ie0ab1a37c2b2e9a3f37baf74772b694b907c6adf
Reviewed-on: http://gerrit.cloudera.org:8080/12200
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <awong@cloudera.com>
8 days agoSupport location awareness in READ_CLOSEST for the C++ client
Will Berkeley [Fri, 4 Jan 2019 21:43:15 +0000 (13:43 -0800)] 
Support location awareness in READ_CLOSEST for the C++ client

Previously, in READ_CLOSEST, the C++ client would choose to read from a
local tablet server if possible and otherwise pick a random replica.
This changes and enhances READ_CLOSEST mode to work as follows:

1. If there is a local server, use it. If there are multiple local
   servers, choose one at random.
2. If there is a server in the same location as the client, use it. If
   there are multiple servers in the same location, choose one at
   random.
3. Otherwise, choose a server at random.

This is not a breaking change, as in the absence of locations the
behavior is consistent with what was documented before, which was only
that a local server would be chosen if possible, else a random server.

Change-Id: I2c6bcc7479c5cf2e17cb6e368ca89a1eb7f21713
Reviewed-on: http://gerrit.cloudera.org:8080/12138
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <granthenke@apache.org>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
8 days ago[build] Update repo URL
Attila Bukor [Thu, 3 Jan 2019 13:58:54 +0000 (14:58 +0100)] 
[build] Update repo URL

According to Apache Infra Team the git repo needs to be moved[1] from
git-wip-us.a.o to gitbox.a.o. This commit changes the
build-support/push_to_asf.py script and the releasing docs to use the
new repo URL.

[1] https://lists.apache.org/thread.html/d6db77cb3ba1c03e25e3c3ecd04577ac067b3a2d8678b26203d89b14@%3Cdev.kudu.apache.org%3E

Change-Id: I80132a8ccfa3ca4f7647ec18bf558940399db684
Reviewed-on: http://gerrit.cloudera.org:8080/12150
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
8 days agoKUDU-2195. Add additional gflag to force sync of consensus metadata
Mike Percy [Wed, 9 Jan 2019 01:01:52 +0000 (17:01 -0800)] 
KUDU-2195. Add additional gflag to force sync of consensus metadata

This patch adds an override gflag for consensus metadata fsync so that
XFS users are less likely to lose their consensus metadata files while
voting right before a power outage.

Change-Id: I73212d1670a33479cce7d9ef9ee61cfe9b00cdd3
Reviewed-on: http://gerrit.cloudera.org:8080/12186
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
8 days agoReduce severity of "Error trying to read ahead of the log" log message
Will Berkeley [Tue, 8 Jan 2019 23:37:57 +0000 (15:37 -0800)] 
Reduce severity of "Error trying to read ahead of the log" log message

In clusters under load, it is typical to see the ERROR log dominated by
messages like the following:

E1221 09:09:13.869918 143384 consensus_queue.cc:440] T 1d030514317942ec9d7796a8a029dace P a4eea0affa4545879c8988b24d8cb2bb [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 6620 is ahead of the local log (next sequential op: 6620). Destination peer: Peer: c0a2e34b708845efb8f090459815272c, Is new: false, Last received: 2995.6619, Next index: 6621, Last known committed idx: 6620, Last exchange result: ERROR, Needs tablet copy: false

This message is logged at the error level, and looks a little bit scary,
but it is in fact harmless. Here's what happens:

1. A leader neglects its duties and the followers elect a new leader.
2. The new leader manages to replicate more ops (usually just the NO_OP
   asserting leadership).
3. The leader of the previous term tries to replicate an op to a peer in
   the new term.
4. From the response, it founds out that
   a. It is in an earlier term, so it should step down and increment its
      term.
   b. The last op the peer saw is (leader's index + k) for some k > 0
      (usually k = 1). So the leader will attempt to send up ops of
      index up through (leader's index + k).
5. The term change is asynchronous, and before it happens the leader
   tries to prepare a new request to the peer whose log is ahead of the
   local log. This causes the ERROR message.
6. The term change happens. The leader steps down, and life goes on.

Note that the leader should also have received a VoteRequest with the
new term and an UpdateConsensus with the new term from the leader. In
general, this message appears only when the leader is under enough
stress to lose its leadership and be unable to service some consensus
RPCs in a timely manner. It is not in an of itself a problem, and it's a
decent indicator of stress on the leader, so I am leaving the message
but downgrading it to INFO level.

See KUDU-1078 for more information about this situation, especially its
previous causes (which were at one time actually harmful).

Change-Id: Ib841a52354e5c71450f3e364ab2fdee51ce2adb4
Reviewed-on: http://gerrit.cloudera.org:8080/12185
Reviewed-by: Adar Dembo <adar@cloudera.com>
Reviewed-by: Mike Percy <mpercy@apache.org>
Tested-by: Will Berkeley <wdberkeley@gmail.com>
8 days agoKUDU-2656: pass IOContext to ValidateDeltaOrder
Andrew Wong [Tue, 8 Jan 2019 08:14:33 +0000 (00:14 -0800)] 
KUDU-2656: pass IOContext to ValidateDeltaOrder

Previously, checksum errors that occurred during calls to the debug-only
function ValidateDeltaOrder() would lead to a DCHECK failure. This patch
plumbs an IOContext to these calls to avoid this and adds a regression
test for such cases.

Change-Id: I3364ad95a5b3608db6538151007c4b6d16500f2b
Reviewed-on: http://gerrit.cloudera.org:8080/12178
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
9 days agotablet: clean up MergeIterState API
Mike Percy [Tue, 8 Jan 2019 00:46:42 +0000 (16:46 -0800)] 
tablet: clean up MergeIterState API

This patch simply adds documentation and some readability cleanup to
MergeIterState, which is a helper class for the MergeIterator.

Non-API changes:
 - Add API docs to all public methods.
 - Extract the implementation of Advance() and PullNextBlock() out of
   the class declaration to make the API contract easier to quickly read
   and understand.
 - Rename a couple of private variables to clarify row-oriented
 semantics.

API cleanup:
 - Add a public Init() method that delegates to PullNextBlock().
 - Make PullNextBlock() and IsBlockExhausted() private methods.
 - Replace exposure of the underlying iterator via iter() with a public
   AddStats() method, since stats collection was the only use of iter().

There are no functional changes in this patch.

Change-Id: Ie3f821dc06ddbe3f7ef018eec15b1993cde7e7e0
Reviewed-on: http://gerrit.cloudera.org:8080/12176
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
9 days ago[fs]: wrapping up containers in scoped_refptr
helifu [Sat, 22 Dec 2018 14:03:20 +0000 (22:03 +0800)] 
[fs]: wrapping up containers in scoped_refptr

It's necessary to wrap up containers in scoped_refptr
to support deleting the full containers that are dead
at runtime. Based on this, the KUDU-2636 can be fixed.

Change-Id: I3c5c620014782b3d57dcbe047d0df73c949590c7
Reviewed-on: http://gerrit.cloudera.org:8080/12121
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
9 days agoKUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks
Andrew Wong [Sat, 5 Jan 2019 01:16:55 +0000 (17:16 -0800)] 
KUDU-2652: deflake TsRecoveryITest.TestNoBlockIDReuseIfMissingBlocks

The test waits for a write workload to generate some orphaned blocks.
With the default write pattern, this could sometimes take a while and
cause the test to hit an error. This patch makes orphaned block
generation faster by making the workload update a single row.

Before this, I looped the test in debug mode with 32 stress threads and
it failed 15/100 times. With the fix, this passed 1000/1000.

Change-Id: I24a689702d29a744be8c113fedafec0307a90b1c
Reviewed-on: http://gerrit.cloudera.org:8080/12166
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
Reviewed-by: Mike Percy <mpercy@apache.org>
9 days ago[gutil] suppress -Wdeprecated-declarations warning on macOS
Alexey Serbin [Tue, 8 Jan 2019 22:54:00 +0000 (14:54 -0800)] 
[gutil] suppress -Wdeprecated-declarations warning on macOS

As it turned out, most of the atomic functions used in
atomicops-internals-macosx.h have been declared deprecated since
macOS 10.12, most of the functions in the SASL API are deprecated
since macOS 10.11, and the krb5 API is deprecated in favor
of GSS.framework.

To avoid multiple compilation warnings on macOS while compiling
with clang from the Xcode development environment, this changelist adds
'pragma GCC diagnostic push/pop' for the '-Wdeprecated-declarations'
flag accordingly.  It's confirmed that this patch suppresses
deprecation warnings at least on macOS 10.14.

This is a follow-up to 678dbac6fb05d0370e40e6645d4b1ec530fa0180.

Change-Id: I82b0ebd9917a567d1a8b72c80b47dc102304d860
Reviewed-on: http://gerrit.cloudera.org:8080/12183
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
9 days ago[test] dos2unix for rowset_tree-test
Alexey Serbin [Tue, 8 Jan 2019 22:24:54 +0000 (14:24 -0800)] 
[test] dos2unix for rowset_tree-test

This patch converts DOS-style line breaks into Unix-like ones in
src/kudu/tablet/rowset_tree-test.cc

This changelist does not contain any functional changes.

Change-Id: I212a5523f48bf305d53e471300d20b7eed932deb
Reviewed-on: http://gerrit.cloudera.org:8080/12182
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
10 days agoSupport CXXFLAGS
Tim Armstrong [Fri, 4 Jan 2019 07:56:57 +0000 (23:56 -0800)] 
Support CXXFLAGS

I ran into a case when building Kudu in native-toolchain
where I want to build Kudu with flags matching those used
by other libraries. Specifically, "-D_GLIBCXX_USE_CXX11_ABI=0"
is needed to allow linking with other libraries built with
that flag. I'm doing this to make it easier to stage a GCC
upgrade in Impala.

By default CMake puts CXXFLAGS into CMAKE_CXX_FLAGS but
this is then clobbered by Kudu's CMake script.
See https://cmake.org/cmake/help/latest/envvar/CXXFLAGS.html

The fix is to append to CMAKE_CXX_FLAGS instead of overwriting it.

Change-Id: Id9dcad5ccb4fc4c081d96eaa40957175731c7a9e
Reviewed-on: http://gerrit.cloudera.org:8080/12162
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
10 days agoRename ts_location_assignment-itest to location_assignment-itest
Will Berkeley [Fri, 4 Jan 2019 21:44:49 +0000 (13:44 -0800)] 
Rename ts_location_assignment-itest to location_assignment-itest

The following patch will add client location assignment tests to this
file, and renaming it now makes it easier to review.

Change-Id: I50e3bed6e8cc8aa08877037b1f9a0f7230009e58
Reviewed-on: http://gerrit.cloudera.org:8080/12161
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
13 days ago[c++] remove -Wno-deprecated compiler's flag
Alexey Serbin [Sat, 29 Dec 2018 02:16:17 +0000 (18:16 -0800)] 
[c++] remove -Wno-deprecated compiler's flag

Removed -Wno-deprecated compiler's flag, replaced some of the
deprecated Kudu API calls with their contemporary counterparts and
added 'pragma GCC diagnostic' blocks elsewhere.

The motivation for this change was seeing too many warnings
while building the python Kudu client.

Change-Id: I513bd49642755dbcb7ed405391fb13096ee49759
Reviewed-on: http://gerrit.cloudera.org:8080/12140
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
13 days agoKUDU-2411. Add scripts to build binaries for testing use
Mike Percy [Tue, 14 Aug 2018 14:34:18 +0000 (16:34 +0200)] 
KUDU-2411. Add scripts to build binaries for testing use

This script will build a jar file containing "relocatable" binaries for
use with a minicluster. The idea is that the resulting jar will be
uploaded to Maven Central whenever there is a release. There should be
one artifact built on Linux (preferably an old OS, such as RHEL 6) and
one on macOS, so that people can run integration tests against Kudu on
either platform.

The current approach is to dynamically link the dependencies instead of
statically linking them to optimize for the total size of the resulting
test artifact. With kudu-tserver, kudu-master, and kudu admin tool
binaries all linking against the same shared objects, the size of the
jar file to download and unpack is significantly reduced. There are a
bunch of shared objects we have to ship regardless, such as dependencies
from thirdparty and security libraries from the system.

Other pieces not included in this patch are code to depend on, locate,
and extract the resulting JAR file, as well as a refactor of the
MiniCluster code to use the resulting bits to start up a Kudu cluster as
part of a Java integration test. Some of that work is underway or has
already been merged as part of KUDU-2411.

Very basic testing of this patch has been done on both Linux and macOS.
We will need to do more testing and validation as the integration work
progresses.

Change-Id: I8b8f90cfe80f6830177bf6ea9e0711eb0d034f75
Reviewed-on: http://gerrit.cloudera.org:8080/11377
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
13 days ago[build-and-test] add short-circuiting for RUN_FLAKY_ONLY
Alexey Serbin [Thu, 3 Jan 2019 23:01:14 +0000 (15:01 -0800)] 
[build-and-test] add short-circuiting for RUN_FLAKY_ONLY

Added a short circuit into build-and-test.sh: in case of
RUN_FLAKY_ONLY=1 and no flaky tests are reported, don't build
anything and report success right away.

Change-Id: I69278c08c496749cd5f30e2d042b9291efde202c
Reviewed-on: http://gerrit.cloudera.org:8080/12154
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
13 days agogeneric_iterators-test: fix TestMerge when FLAGS_num_iters > 1
Adar Dembo [Tue, 27 Nov 2018 23:03:40 +0000 (15:03 -0800)] 
generic_iterators-test: fix TestMerge when FLAGS_num_iters > 1

And some assorted cleanup.

Change-Id: Ieb6c5a26e0274772c85c0202c78f906aed154915
Reviewed-on: http://gerrit.cloudera.org:8080/12155
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy <mpercy@apache.org>
13 days agobuild: Factor dependency extraction code from dist_test into a python library
Mike Percy [Sun, 4 Nov 2018 02:33:11 +0000 (21:33 -0500)] 
build: Factor dependency extraction code from dist_test into a python library

This will allow us to reuse this dependency extraction logic when
creating minicluster test binary artifacts that ship their dependencies.

There are no functional changes in this patch.

I ran dist-test a few times and everything seems to work fine:

 - http://dist-test.cloudera.org/job?job_id=mpercy.1546559376.69253 (run)
 - http://dist-test.cloudera.org/job?job_id=mpercy.1546559905.73493 (loop)

Change-Id: I0b4cbfceb053c61dbb1f1d16716acc8926987af2
Reviewed-on: http://gerrit.cloudera.org:8080/12153
Tested-by: Mike Percy <mpercy@apache.org>
Reviewed-by: Adar Dembo <adar@cloudera.com>
13 days ago[tools] Allow ksck output to be filtered by section
acelyc111 [Thu, 6 Dec 2018 00:42:19 +0000 (08:42 +0800)] 
[tools] Allow ksck output to be filtered by section

When there are lots of tables or tservers in the cluster, the `kudu ksck`
output may be very large and hard for administrators to read.
This commit adds a parameter `--sections` to filter output sections.

Change-Id: Ic31dcb7795b5c510b40fa2024960306b9da40e05
Reviewed-on: http://gerrit.cloudera.org:8080/12038
Tested-by: Kudu Jenkins
Reviewed-by: Will Berkeley <wdberkeley@gmail.com>
2 weeks agopython: fix tests run in Python 3.4 environments
Adar Dembo [Thu, 3 Jan 2019 01:19:46 +0000 (17:19 -0800)] 
python: fix tests run in Python 3.4 environments

Our Python build installs pandas, which is used in some tests. If we first
install numpy, pandas will use it for its build. This was the case in the
Python 2 variant of the Kudu build, which specified numpy 1.14, the last
version of numpy that was Python 2.6 compatible.

The Python 3 variant didn't explicitly install numpy, and thus was using
numpy 1.15.4 for quite some time. A few weeks ago numpy 1.16.0rc1 was
released, with one notable anti-feature: loss of Python 3.4 compatibility.
This is the version of Python 3 in Ubuntu 14.04; Python 3 tests run in
such environments would fail [1] like so:

  ../py_env/lib/python3.4/site-packages/py/_path/local.py:668: in pyimport
    __import__(modname)
  kudu/__init__.py:18: in <module>
    from kudu.client import (Client, Table, Scanner, Session,  # noqa
  kudu/client.pyx:50: in init kudu.client
    import pandas
  ../py_env/lib/python3.4/site-packages/pandas/__init__.py:26: in <module>
    from pandas._libs import (hashtable as _hashtable,
  ../py_env/lib/python3.4/site-packages/pandas/_libs/__init__.py:4: in <module>
    from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
  pandas/_libs/src/numpy.pxd:865: in init pandas._libs.tslib
    ???
  E   ValueError: numpy.ufunc has the wrong size, try recompiling. Expected 192, got 216

The fix is simple: since neither we nor pandas care much about the version
of numpy used, let's explicitly install the last Python 3.4 compatible numpy
in all Python 3 builds.

1. The breakage was not universal, likely due to wheel caching on pypi as
   well as in the local pip cache.

Change-Id: I8683c6d24b3b808384a7e4d63142a78f3a7ee5f0
Reviewed-on: http://gerrit.cloudera.org:8080/12148
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <awong@cloudera.com>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
2 weeks agojava: KuduBinaryExtractor should use the thread context classloader
Mike Percy [Thu, 3 Jan 2019 00:35:19 +0000 (16:35 -0800)] 
java: KuduBinaryExtractor should use the thread context classloader

Using the thread context classloader makes it straightforward to write
end-to-end tests, one of which is included in this patch.

This patch makes the following changes:

1. The KuduBinaryExtractor will now search for the test binary jars
   via the thread context classloader, if available.
2. Remove the work-in-progress OS-detection implementation from
   KuduBinaryExtractor (to be replaced in a future commit) because it
   was failing tests.
3. KuduBinaryExtractor will no longer cache its search results to make
   it more straightforward to use a thread local context classloader.
4. KuduBinaryExtractor.extractKuduBinary() now throws
   FileNotFoundException instead of IllegalStateException when the Kudu
   binary test jar cannot be found. Since it is a checked exception and
   a subclass of IOException it will be less likely to go uncaught.
5. Update the API docs to be more specific about the semantics of the
   public methods of KuduBinaryExtractor, including the use of the
   thread context classloader.
6. Add a simple test binary locator test using a child classloader
   plumbed into the KuduBinaryExtractor code by setting it as the thread
   context classloader.

Change-Id: I5e1cf188bb557eeaea0b2867243855f3f2d121f1
Reviewed-on: http://gerrit.cloudera.org:8080/12147
Tested-by: Mike Percy <mpercy@apache.org>
Reviewed-by: Brian McDevitt <brian@phdata.io>
Reviewed-by: Grant Henke <granthenke@apache.org>
2 weeks agodocs: update guidance for binding functions
Andrew Wong [Wed, 2 Jan 2019 23:45:11 +0000 (15:45 -0800)] 
docs: update guidance for binding functions

Tweaked the docs to call out equivalent STL functions and their
relationship with std::shared_ptr.

A rendered version can be found here:
https://github.com/andrwng/kudu/blob/docs_cbs/docs/contributing.adoc#function-binding-and-callbacks

Change-Id: I326931a096b7184d403aea116248cffcde5c9775
Reviewed-on: http://gerrit.cloudera.org:8080/12146
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
2 weeks agoKUDU-683: unify C++ client-to-master retry logic
Andrew Wong [Wed, 28 Nov 2018 03:10:41 +0000 (19:10 -0800)] 
KUDU-683: unify C++ client-to-master retry logic

In preparation for implementing an asynchronous RPC that will need to
connect to the leader master, I've been looking through some of the
retry logic we have scattered around the C++ client. I noticed
near-identical code in a couple places (LookupRpc and
SyncLeaderMasterRpc), so I've ripped it out and put it in its own
reusable class.

A few things are bundled into this patch. The big pieces are:
- A new AsyncLeaderMasterRpc template class is introduced that sends
  RPCs to the leader master and handles reconnection and retries upon
  failure. LookupRpcs will now use this template to perform their RPCs,
  and SyncLeaderMasterRpcs has been replaced entirely to use a
  synchronized AsyncLeaderMasterRpcs.
- In implementing the template, I unified the retry logic to be used for
  the above existing RPCs; the bulk of it was overlapped, but some logic
  existed specifically for LookupRpc. See the comment above
  RetryOrReconnectIfNecessary in master_proxy_rpc.h and the impl for
  LookupRpc::SendRpcCb in meta_cache.cc for more details. For the most
  part, there aren't functional changes.
- SyncLeaderMasterRpcs used exponential backoff when retrying RPCs, but
  the existing RpcRetrier implemented its own backoff. To maintain
  this, I've piped down an enum from Rpc to RpcRetrier to determine
  which backoff strategy to use.
- The new template and most other existing Rpc implementations
  duplicated code from Rpc::HandleResponse() to handle certain remote
  errors. As such, I've removed HandleResponse entirely and added a TODO
  to perhaps merge existing logic even further (see retriable_rpc.h).

Change-Id: I2450676da1c723a247c84deb1b895f116173670e
Reviewed-on: http://gerrit.cloudera.org:8080/12002
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
2 weeks ago[util] use lighter locking scheme for ThreadMgr
Alexey Serbin [Wed, 19 Dec 2018 21:23:03 +0000 (13:23 -0800)] 
[util] use lighter locking scheme for ThreadMgr

Use the rw_spinlock primitive to guard the ThreadMgr's registry
threads instead of Mutex.  Also, use the shared lock pattern to deal
with write and read access to the guarded entities of the ThreadMgr.
In addition, do not hold the lock for the whole duration of generating
the /threadz page for the embedded webserver.

With this patch, ThreadMgr now uses std::unordered_map as a container
for category-specific thread information and enforces stricter
consistency rules while removing thread information from its registry.
The process of adding an information about a thread into the registry
is not guarded by stricter consistency checks (i.e. it stays as it was
before this patch); see below for explanation.

NOTE:
  The stricter consistency around adding a thread into the ThreadMgr's
  registry caused issues with multiprocessing in Python tests,
  and I spent some time trying to work around that.  However, that was
  not fruitful.  I think the proper solution would be keeping the thread
  registry bound to some top-level object (like Kudu client or
  ServerBase object) and cleaning it up in accordance with the object's
  life cycle.

  In essence, the problem happens due to the combination of the
  following:
    * The way how workers are spawned by the multiprocessing.Pool,
      i.e. calling fork() at the point where Pool is instantiated.
    * The fact that pthread_t handles might be the same for threads
      in different processes.

  In detail, if multiprocessing.Pool is spawning worker processes after
  an instance of Kudu client has been created in the main test process,
  every worker gets a copy of the thread registry in its address space.
  The unexpected copy of the registry in a worker process is populated
  with the information on threads spawned due to the activity of the
  Kudu client in the main test process.  Later on, if a worker
  instantiates a Kudu client on its own, the newly spawned threads by
  the worker's Kudu client might have the same pthread_t handles
  as the threads whose information records are inadvertently inherited
  from the main test process.

  Also, it turned out it's impossible to handle worker's crash in a
  multiprocessing.Pool, and that's by design: for details see:
    https://stackoverflow.com/questions/24894682/

  BTW, in Python 3 the problem with the duplicated address space
  in worker processes has been resolved by using context and additional
  worker spawn modes (e.g., 'forkserver').

Change-Id: I4d49c1c39392e01c45019844430a4fe3d116c277
Reviewed-on: http://gerrit.cloudera.org:8080/12112
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
2 weeks ago[python] handle DisableOpenSSLInitialization() result status
Alexey Serbin [Thu, 27 Dec 2018 18:32:28 +0000 (10:32 -0800)] 
[python] handle DisableOpenSSLInitialization() result status

This patch adds verification for DisableOpenSSLInitialization().

The motivation for this change is two-fold:
  a) avoid glitches while initializing the OpenSSL library
     by python Kudu client if DisableOpenSSLInitialization()
     de facto failed
  b) get rid of compilation warnings on unhandled return value
     of the DisableOpenSSLInitialization() function

Change-Id: I1963b6d87d731fbfa87a09b986595aa8ea00da60
Reviewed-on: http://gerrit.cloudera.org:8080/12135
Tested-by: Alexey Serbin <aserbin@cloudera.com>
Reviewed-by: Adar Dembo <adar@cloudera.com>
3 weeks ago[location_awareness] Assign locations to clients
fwang29 [Tue, 13 Nov 2018 07:52:42 +0000 (23:52 -0800)] 
[location_awareness] Assign locations to clients

This patch makes it so the client is assigned a location by each master
as part of the ConnectToMaster RPC (which is part of the
ConnectToCluster process). The client will store the location it is
assigned by the leader master and potentially update it every time it
reconnects to the cluster.

This assignment will be used in a follow-up to implement location-aware
semantics for CLOSEST_REPLICA scans.

Change-Id: I0efb327293d86168a30b05305f69d011ad15587a
Reviewed-on: http://gerrit.cloudera.org:8080/11923
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
3 weeks agotablet: add support for diff scan iterator
Mike Percy [Wed, 12 Dec 2018 02:49:26 +0000 (18:49 -0800)] 
tablet: add support for diff scan iterator

This commit adds plumbing to support creating an iterator at the Tablet
level that supports snap_to_exclude and include_deleted_rows.

Also added a test that validates that ghost rows are returned. This test
should be updated once KUDU-2645 has been implemented, which should
allow the de-duplication of such ghost rows.

Change-Id: I9aa0ce2276bdd37688e7a91a2efcf49a8f802eb5
Reviewed-on: http://gerrit.cloudera.org:8080/12109
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
3 weeks ago[build] Fix bulding codegen on MacOS Mojave
Attila Bukor [Sat, 8 Dec 2018 09:40:49 +0000 (10:40 +0100)] 
[build] Fix bulding codegen on MacOS Mojave

The codegen target uses clang-6.0 from thirdparty even on macOS, after
upgrading to macOS Mojave and Xcode 10 the build failed due to changes
in where the SDK is located.

Xcode 10's release notes claim the macOS headers are no longer installed
in the base system under /usr/include[1].

This commit makes CMake locate the headers using "xcrun --show-sdk-path"
and pass it to clang's sysroot.

[1] https://developer.apple.com/documentation/xcode_release_notes/xcode_10_release_notes

Change-Id: I33f7510c7bf0cc2a7708191653b23c04e6df8a29
Reviewed-on: http://gerrit.cloudera.org:8080/12054
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Attila Bukor <abukor@apache.org>
4 weeks agoKUDU-2411: Initial draft of kudu binary jar extractor.
Brian McDevitt [Tue, 18 Dec 2018 03:13:12 +0000 (21:13 -0600)] 
KUDU-2411: Initial draft of kudu binary jar extractor.

The binary jar extractor enables application developers to include a
Kudu binary jar on their test classpath. The Kudu binaries can be used
for integration testing scenarios with the KuduMiniCluster.

Change-Id: I5029ac5279df3e5d8321706fda3e511c59406c41
Reviewed-on: http://gerrit.cloudera.org:8080/12104
Reviewed-by: Mike Percy <mpercy@apache.org>
Tested-by: Kudu Jenkins
4 weeks agojava: add log4j.properties file to kudu-test-utils module
Mike Percy [Tue, 4 Dec 2018 02:37:36 +0000 (18:37 -0800)] 
java: add log4j.properties file to kudu-test-utils module

This allows logging to function properly for tests in this module.

It also squelches the following warning:

log4j:WARN No appenders could be found for logger (org.apache.kudu.test.cluster.KuduBinaryLocator).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Change-Id: Ic57941d593c32d17c9619f51d18430a75e962635
Reviewed-on: http://gerrit.cloudera.org:8080/12040
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <granthenke@apache.org>
Reviewed-by: Adar Dembo <adar@cloudera.com>
4 weeks agoKUDU-2387: retry exportAuthenticationCredentials and getHiveMetastoreConfig
Adar Dembo [Sun, 16 Dec 2018 01:29:48 +0000 (17:29 -0800)] 
KUDU-2387: retry exportAuthenticationCredentials and getHiveMetastoreConfig

These client calls were not retried when a leader master could not be found.

The fix cribs from RetryRpcErrback and delayedSendRpcToTablet. It's not
possible to reuse those methods directly, as the RPCs involved in these
calls are squirreled away in ConnectToCluster.

Change-Id: Ia534efe776c4b10f45c961a3b279e729dc015ea3
Reviewed-on: http://gerrit.cloudera.org:8080/12099
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <granthenke@apache.org>
4 weeks agoCreate parallelized loader Spark job
Grant Henke [Mon, 17 Dec 2018 20:06:11 +0000 (14:06 -0600)] 
Create parallelized loader Spark job

This patch adds a new DistributedDataGenerator tool
that can load random or sequential data into an existing
Kudu table.

This tool was written to help test the backup and restore
tools. It is currently marked private but could be made
public in the future.

Change-Id: Ibdfd41a21a7f80d22125c7f4e5ca4ed62c31709d
Reviewed-on: http://gerrit.cloudera.org:8080/12101
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
4 weeks ago[tests] add /threadz pages to PeriodicWebUIChecker
Alexey Serbin [Wed, 19 Dec 2018 23:38:22 +0000 (15:38 -0800)] 
[tests] add /threadz pages to PeriodicWebUIChecker

With this patch, the PeriodicWebUIChecker periodically fetches
the /threadz and /threadz?group=all pages of the embedded Web server
along with other diagnostic pages.  Also, shuffle the URLs that
are being check sequentially -- that adds more randomization in
the pattern of the requests sent by the checker.

In addition, this patch updates the webserver-stress-itest to
run more readers/writer threads for TestWorkload -- that allows
to expose more race conditions in TSAN/ASAN builds (and that
is used in a follow-up changelist).

Also, this patch contains other minor/non-functional updates.

Change-Id: Iad6cbfe5108ac99231b82f662cee56ffcd25f026
Reviewed-on: http://gerrit.cloudera.org:8080/12114
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
4 weeks ago[spark] Add a format alias of “kudu”
Grant Henke [Thu, 13 Dec 2018 23:14:22 +0000 (17:14 -0600)] 
[spark] Add a format alias of “kudu”

Implements the spark DataSourceRegister on the
kudu-spark DefaultSource to allow the use of
the alias “kudu” when using the Kudu format.

This patch also updates the documentation examples
to use the new aliased syntax given that is the Spark
defacto approach.

Additionally the old package object that was
effectively an alias for the longer format syntax
is marked as deprecated. This syntax doesn’t match
other spark examples and sources and could be
confusing.

Change-Id: I6bb2c7314652456b2f0fa3c4a110ef156503ef92
Reviewed-on: http://gerrit.cloudera.org:8080/12083
Reviewed-by: Mike Percy <mpercy@apache.org>
Tested-by: Kudu Jenkins
4 weeks agodelta_store: add nulls to fuzz tests
Adar Dembo [Tue, 27 Nov 2018 05:11:33 +0000 (21:11 -0800)] 
delta_store: add nulls to fuzz tests

It was easy enough and it provides some additional test coverage.

Change-Id: I310ebc47e95898e50c3b2052a88a336b7d3f54f1
Reviewed-on: http://gerrit.cloudera.org:8080/11994
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy <mpercy@apache.org>
4 weeks agoFix typo in ASAN cmake command
Will Berkeley [Sat, 15 Dec 2018 06:00:51 +0000 (22:00 -0800)] 
Fix typo in ASAN cmake command

Change-Id: I049679199d9fd3398a857f89a0e919e7bac1c628
Reviewed-on: http://gerrit.cloudera.org:8080/12098
Tested-by: Will Berkeley <wdberkeley@gmail.com>
Reviewed-by: Andrew Wong <awong@cloudera.com>
4 weeks ago[client] clang: explicit override in C++11 mode only
Alexey Serbin [Fri, 14 Dec 2018 21:00:20 +0000 (13:00 -0800)] 
[client] clang: explicit override in C++11 mode only

While playing with the Kudu C++ example application, I noticed that
clang emits a lot of compilation warnings about using the explicit
override for virtual methods.  This patch addresses that, updating
the OVERRIDE macro to take into account the version of the C++ standard
the compiler is using while compiling a Kudu client application.

Change-Id: I2e11c6ce830e0b525a1b92af126f10de745c986a
Reviewed-on: http://gerrit.cloudera.org:8080/12093
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
4 weeks ago[client] add WARN_UNUSED_RESULT for a few static functions
Alexey Serbin [Fri, 14 Dec 2018 18:09:50 +0000 (10:09 -0800)] 
[client] add WARN_UNUSED_RESULT for a few static functions

This patch adds the WARN_UNUSED_RESULT attribute for the following
static functions in the kudu::client namespace:

  * SetInternalSignalNumber(int signum)
  * DisableSaslInitialization()
  * DisableOpenSSLInitialization()

This change does not affect the exported symbols of the above mentioned
functions in the kudu_client library, so this change is
backward-compatible.

Change-Id: Id66cad549ebf57e94da2d8a7cbc85b5476fa168d
Reviewed-on: http://gerrit.cloudera.org:8080/12090
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
4 weeks ago[security] enhance warning on conflicting OpenSSL initialization
Alexey Serbin [Fri, 14 Dec 2018 03:25:51 +0000 (19:25 -0800)] 
[security] enhance warning on conflicting OpenSSL initialization

This patch enhances the warning message on detected conflicting
initialization of the OpenSSL library, making it more actionable.

This patch does not contain any functional changes.

Change-Id: I72e989dce82586ba43d2cb4c00e2ef6948c2bd2a
Reviewed-on: http://gerrit.cloudera.org:8080/12089
Tested-by: Alexey Serbin <aserbin@cloudera.com>
Reviewed-by: Adar Dembo <adar@cloudera.com>
4 weeks ago[webui] drop 'Last status' column for tombstoned tablets
Alexey Serbin [Wed, 21 Nov 2018 03:35:32 +0000 (19:35 -0800)] 
[webui] drop 'Last status' column for tombstoned tablets

This is a follow-up to 9a01205ce10b468698c76aedbf51016a90045620.

Change-Id: I5c21b9fbf7c6992c8cb29d9cc2d4c11623bbbbfa
Reviewed-on: http://gerrit.cloudera.org:8080/11972
Tested-by: Kudu Jenkins
Reviewed-by: Will Berkeley <wdberkeley@gmail.com>
5 weeks ago[java] Downgrade errorprone to 2.3.1
Grant Henke [Thu, 13 Dec 2018 23:25:06 +0000 (17:25 -0600)] 
[java] Downgrade errorprone to 2.3.1

Errorprone 2.3.2 can cause build issues on older
jdks. This patch downgrades it to 2.3.1 until the issue
is resolved.

Change-Id: I6162722ae2a37ebb8398cc7c6fdf4246eb2a18c5
Reviewed-on: http://gerrit.cloudera.org:8080/12080
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
5 weeks ago[Java] Add a Schema and Data Generator
Grant Henke [Mon, 10 Dec 2018 20:54:41 +0000 (14:54 -0600)] 
[Java] Add a Schema and Data Generator

This patch adds schema and data generator utility
classes that can be used to create random tables and
random data. These utilities are useful in fuzz tests
and for various load and scale test applications.

The initial implementation is inteneded to be
fairly flexble without being overengineered.
Follow on patches will improve the API and options.

The classes are currently marked private, but could be
changed in the future.

Change-Id: I750d2d346c3eeb7075b21c3fec0fd25236da4f56
Reviewed-on: http://gerrit.cloudera.org:8080/12061
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
5 weeks ago[Java] Upgrade dependencies
Grant Henke [Wed, 12 Dec 2018 20:02:41 +0000 (14:02 -0600)] 
[Java] Upgrade dependencies

Upgrades the Java dependencies and Gradle versions.

This patch also has a few build fixes to support
Gradle 5.0.

Minor version upgrades:
- ClojureToolsCli 0.3.5 -> 0.4.1
- Guava 26.0-android -> 27.0-android
- Mockito 2.22.0 -> 2.23.4

Maintenance version upgrades:
- Errorprone 2.3.1 -> 2.3.2
- Hive 2.3.3 -> 2.3.4

Gradle upgrades:
- Gradle 4.10.2 -> 5.0
- gradle-avro-plugin 0.15.1 -> 0.16.0
- shadow 2.0.4 -> 4.0.2
- protobuf-gradle-plugin 0.8.6 -> 0.8.7
- nebula-clojure-plugin 6.0.2 -> 7.0.1
- spotbugs-gradle-plugin 1.6.4 -> 1.6.5

Change-Id: Ic09189d7fee7f2348718375083c32fa7b00ec5c0
Reviewed-on: http://gerrit.cloudera.org:8080/12076
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
5 weeks agoKUDU-1375: [docs] Remove "binaries" in the Build from Source section
Alex Rodoni [Mon, 10 Dec 2018 20:52:18 +0000 (12:52 -0800)] 
KUDU-1375: [docs] Remove "binaries" in the Build from Source section

- make install does not install the Kudu binaries

Change-Id: I5b1e7337cdd9f04eea1c2b4da3d27b9815553121
Reviewed-on: http://gerrit.cloudera.org:8080/12060
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Adar Dembo <adar@cloudera.com>
5 weeks agojava: add unit test for RetryRule
Mike Percy [Tue, 4 Dec 2018 02:51:49 +0000 (18:51 -0800)] 
java: add unit test for RetryRule

Also did some minor refactoring of the RetryRule.

Change-Id: Ia4bb578b550c3ccb3ce5526a4ca87abafd7d4021
Reviewed-on: http://gerrit.cloudera.org:8080/12041
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Mike Percy <mpercy@apache.org>
6 weeks ago(05/05) delta_applier: support for diff scan style iteration
Adar Dembo [Fri, 2 Nov 2018 04:04:55 +0000 (21:04 -0700)] 
(05/05) delta_applier: support for diff scan style iteration

This patch incorporates all of the previous work to provide diff scan
support for an entire DiskRowSet. There's a new "fuzzy" test in
diskrowset-test that exercises this functionality.

Change-Id: I895350a3c7728df88a87a36b904ffdea4f3f6b7a
Reviewed-on: http://gerrit.cloudera.org:8080/11860
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy <mpercy@apache.org>
6 weeks ago(04/05) delta_store: support iteration with is_deleted virtual column
Adar Dembo [Fri, 2 Nov 2018 04:04:38 +0000 (21:04 -0700)] 
(04/05) delta_store: support iteration with is_deleted virtual column

This patch adds support to delta stores (i.e. DMS and delta files) for
iterating with an IS_DELETED virtual column. A diff scan will include
IS_DELETED in the projection so that clients can tell if a row was deleted
in the requested time range.

Change-Id: Ibafcae9f9f82a6c2d9a8afedb8fbf07d6e3069ec
Reviewed-on: http://gerrit.cloudera.org:8080/11859
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy <mpercy@apache.org>
6 weeks ago(03/05) delta_store: support iteration with snap_to_exclude
Adar Dembo [Fri, 2 Nov 2018 00:07:18 +0000 (17:07 -0700)] 
(03/05) delta_store: support iteration with snap_to_exclude

This patch changes the delta stores (DMS and delta files) to respect
snap_to_exclude during iteration. The key changes are:
- The introduction of the "selection" criteria, a new delta relevancy
  formula for determining whether a delta applies to a scan with both
  snap_to_exclude and snap_to_include. The existing "application" criteria
  was formalized and moved into delta_relevancy.h. There was also a
  non-trivial change to DeltaFileReader::IsRelevantForSnapshot() to use both
  criterias when culling entire delta files.
- A new SelectUpdates() method for using the selection criteria on a batch
  of prepared deltas. SelectUpdates() requires new in-memory state, the
  creation of which is gated behind a new PREPARE_FOR_SELECT flag so as not
  to affect regular scans.
- Updates to the delta fuzz testing logic to test iterators with two
  timestamps, and to provide SelectUpdates() coverage.

A future patch will modify DeltaApplier to use SelectUpdates for diff scans.

Change-Id: I7811e185fef270f40fdbbb38f491eee8b4aa043c
Reviewed-on: http://gerrit.cloudera.org:8080/11858
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy <mpercy@apache.org>
6 weeks ago(02/05) delta_store: convert DeltaIterator::PrepareBatch flags into bitfield
Adar Dembo [Tue, 11 Sep 2018 23:27:56 +0000 (16:27 -0700)] 
(02/05) delta_store: convert DeltaIterator::PrepareBatch flags into bitfield

A follow-on patch will introduce DeltaIterator::SelectUpdates(), a new
method for extracting delta information from a DeltaIterator. As it is
only to be used by diff scans and will require dedicated in-memory state,
it'll also come with a new DeltaIterator::PrepareBatch() flag.

However, diff scans will need to use both SelectUpdates() and
ApplyUpdates(), and it'd be a shame to have to reseek and reprepare just to
use both extraction methods. As such, this patch makes it possible to
prepare a batch for use in multiple delta extraction methods.

Change-Id: Id500ec3cc9459a78ae1098be3f3cd0cacb7a7a1a
Reviewed-on: http://gerrit.cloudera.org:8080/11857
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy <mpercy@apache.org>
6 weeks ago[test] Break up test output per test case.
Grant Henke [Tue, 4 Dec 2018 20:29:51 +0000 (14:29 -0600)] 
[test] Break up test output per test case.

Currently, when generating the JUnit XML for each
test suite Gradle appends all of the test output to
stderr and stdout at the suite level. This
change ensures Gradle will break up the output
by each test case instead.

Change-Id: If7564e48cddc7bf7ea7595d925f605802259dced
Reviewed-on: http://gerrit.cloudera.org:8080/12031
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
Reviewed-by: Will Berkeley <wdberkeley@gmail.com>
6 weeks ago[docs] Fix error gflags
Yingchun Lai [Sun, 2 Dec 2018 07:05:56 +0000 (02:05 -0500)] 
[docs] Fix error gflags

Change-Id: Id7d846fbacdd294b13139cfc17dba095aae7aa3e
Reviewed-on: http://gerrit.cloudera.org:8080/12023
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
7 weeks ago[tools] Add tool to dump memtrackers
Will Berkeley [Wed, 21 Nov 2018 12:08:26 +0000 (04:08 -0800)] 
[tools] Add tool to dump memtrackers

This adds a new tool, `kudu diagnose dump_mem_trackers`, that dumps the
mem-trackers information. It contains information equivalent to the
tracker table on /mem-trackers. There's a new RPC introduced to support
this tool. Output both as JSON and as a table is supported. The table
output type flattens the tracker hierarchy, but it can be
reconstructed from the 'id' and 'parent_id' columns.

Some Kudu community members have reported to me they have started using
the /mem-trackers page to investigate performance and memory issues with
Kudu clusters, and having a way to get the same information in a form
amenable to programmatic searching, sorting, and filtering is
convenient. This tool can also serve as a starting point for more
automated analysis of the tracker information.

The particular advantage of the memtracker dump over, say, the data
generated by heap sampling is that the memtracker shows where memory is
owned. The sampling just shows where it was allocated in a call stack.
So, for example, using the memtracker information it's possible to tell
very quickly exactly which tablets are using a lot of memory in
DeltaMemStores. Heap samples would indicate memory allocated in
update-related code paths, but it would not be possible to tell which
tablet replicas actually held the memory that was allocated.

Here's a small sample of output in csv table format:

tablet-2e02d00d46834b359c6ba0f8d471cbf9,server,-1,2318,2583
txn_tracker,tablet-2e02d00d46834b359c6ba0f8d471cbf9,67108864,0,0
MemRowSet-1,tablet-2e02d00d46834b359c6ba0f8d471cbf9,-1,265,265
DeltaMemStores,tablet-2e02d00d46834b359c6ba0f8d471cbf9,-1,265,265
tablet-2d40450f6556485eab92f64f90756b61,server,-1,2318,2583
txn_tracker,tablet-2d40450f6556485eab92f64f90756b61,67108864,0,0
MemRowSet-1,tablet-2d40450f6556485eab92f64f90756b61,-1,265,265
DeltaMemStores,tablet-2d40450f6556485eab92f64f90756b61,-1,265,265

The columns are id,parent-id,limit,current_consumption,peak_consumption
just like the table on /mem-trackers.

Change-Id: I3e54f809bb6434b8d8e8c95771fe089c9da122d0
Reviewed-on: http://gerrit.cloudera.org:8080/11975
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
7 weeks ago[docs] Contributing to blog
Attila Bukor [Tue, 27 Nov 2018 22:42:53 +0000 (23:42 +0100)] 
[docs] Contributing to blog

Submitting blog posts are not straightforward, especially if someone
hasn't used Jekyll and/or Gerrit. This commit adds a "blog posts"
section to our contributing docs.

Change-Id: Ifd8ccae4b15b1ad8b679e0d2d8eabdf5fb5e3a09
Reviewed-on: http://gerrit.cloudera.org:8080/11940
Reviewed-by: Mike Percy <mpercy@apache.org>
Tested-by: Attila Bukor <abukor@apache.org>
7 weeks agocompaction: remove dead code
Adar Dembo [Tue, 20 Nov 2018 23:42:13 +0000 (15:42 -0800)] 
compaction: remove dead code

As far as I can tell, this has been dead since 2014 (commit 07db3407e)
and 2013 (commit 95f6c9811).

Change-Id: I046d1c402f65803b6feb64983b52b6cccafdf2ab
Reviewed-on: http://gerrit.cloudera.org:8080/11978
Tested-by: Kudu Jenkins
Reviewed-by: Mike Percy <mpercy@apache.org>
7 weeks ago[raft_consensus-itest] fix flake in Test_KUDU_1735
Alexey Serbin [Thu, 22 Nov 2018 05:48:07 +0000 (21:48 -0800)] 
[raft_consensus-itest] fix flake in Test_KUDU_1735

Fixed flake in RaftConsensusParamReplicationModesITest.Test_KUDU_1735
scenario of the raft_consensus-itest suite, for both the 3-4-3 and 3-2-3
replica management schemes.

The flakiness was due to a few reasons, where the most prominent one was
the race in 3-4-3 mode between shutting down replica's consensus
upon DeleteTablet(TABLET_DATA_TOMBSTONED) and re-discovering the peer
by the leader once the catalog manager added the replica back as a
non-voter.  Another minor flake was due to not waiting for the manually
elected leader to commit an operation on its own term.  And the one
was due to waiting for RECEIVED_OPID instead of COMMITTED_OPID Raft
operation index before setting the --fault_crash_before_append_commit
flag to 1.

I ran the RaftConsensusParamReplicationModesITest.Test_KUDU_1735
scenario using dist_test with --stress_cpu_threads=16 before and after
the fix with the following results:

  before the fix (950 out of 1024 failed):
    http://dist-test.cloudera.org/job?job_id=aserbin.1542868617.80411

  after the fix  (  0 out of 1024 failed):
    http://dist-test.cloudera.org/job?job_id=aserbin.1542871109.113847

Change-Id: If44ad0e8363a3aead409484cff68843f1e5d6b6d
Reviewed-on: http://gerrit.cloudera.org:8080/11981
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <awong@cloudera.com>
7 weeks ago[compaction] KUDU-1400: Improve rowset compaction policy to consider merging small...
Will Berkeley [Tue, 30 Oct 2018 22:11:41 +0000 (15:11 -0700)] 
[compaction] KUDU-1400: Improve rowset compaction policy to consider merging small DRSs

This implements the small rowset compaction scheme explained in the
KUDU-1400 design doc [1]. While the implementation is simple, the
reasoning behind it is nuanced, and I'll defer the explanations of them
to the design doc rather than repeating them here.

The three relevant constant factors:

--compaction_minimum_improvement_score
--compaction_small_rowset_tradeoff
kSupportAdjust

have been tuned to work well with each other. See the relevant tests for
reasoning on why, and validation of what the results should be.

Since compaction policy's performance is important, I ran the YCSB
benchmark before and after the change using 'perf stat' to compare the
performance:

Command:

Before:

 Performance counter stats for 'bin/compaction_policy-test --gtest_filter=*Ycsb*' (10 runs):

       1231.095442 task-clock                #    0.997 CPUs utilized            ( +-  2.00% )
               148 context-switches          #    0.120 K/sec                    ( +-  2.43% )
                 9 cpu-migrations            #    0.007 K/sec                    ( +- 16.14% )
             3,630 page-faults               #    0.003 M/sec                    ( +-  0.00% )
     3,251,530,478 cycles                    #    2.641 GHz                      ( +-  2.04% )
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
     5,772,319,429 instructions              #    1.78  insns per cycle          ( +-  0.01% )
     1,070,627,520 branches                  #  869.654 M/sec                    ( +-  0.01% )
        13,583,368 branch-misses             #    1.27% of all branches          ( +-  0.10% )

       1.235037947 seconds time elapsed                                          ( +-  2.00% )

After:

Performance counter stats for 'bin/compaction_policy-test --gtest_filter=*Ycsb*' (10 runs):

       1297.749333 task-clock                #    0.994 CPUs utilized            ( +-  2.17% )
               158 context-switches          #    0.122 K/sec                    ( +-  3.01% )
                14 cpu-migrations            #    0.011 K/sec                    ( +- 17.48% )
             3,636 page-faults               #    0.003 M/sec
     3,509,480,140 cycles                    #    2.704 GHz                      ( +-  2.65% )
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
     6,800,316,126 instructions              #    1.94  insns per cycle          ( +-  0.01% )
     1,073,111,416 branches                  #  826.902 M/sec                    ( +-  0.01% )
        13,574,780 branch-misses             #    1.26% of all branches          ( +-  0.15% )

       1.305058206 seconds time elapsed                                          ( +-  2.16% )

A follow up will integrate the below design doc with the existing one,
docs/design-docs/compaction-policy.md.

[1]: https://docs.google.com/document/d/1yTfxt0_2p5EfIjCnjJCt3o-nB9xk-Kl2O8yKTA1LQrQ/edit?usp=sharing

Change-Id: I7b421c6ed77d28ebab9b91a4d6fcb1e825997e6c
Reviewed-on: http://gerrit.cloudera.org:8080/11869
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <awong@cloudera.com>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
7 weeks ago[consensus] fix message on unprepared dedup ops
Alexey Serbin [Thu, 22 Nov 2018 08:17:42 +0000 (00:17 -0800)] 
[consensus] fix message on unprepared dedup ops

A minor clean-up on RaftConsensus::UpdateReplica() aiming to fix
the warning message on the unprepared operations from deduplicated
request.

Change-Id: Ib9871068743b27720f839797ba6aa6f23cf03a7a
Reviewed-on: http://gerrit.cloudera.org:8080/11982
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <awong@cloudera.com>
8 weeks agodeflake TsRecoveryITest.TestTabletRecoveryAfterSegmentDelete
Andrew Wong [Wed, 21 Nov 2018 23:35:52 +0000 (15:35 -0800)] 
deflake TsRecoveryITest.TestTabletRecoveryAfterSegmentDelete

The test runs a write workload and waits for a certain number of WAL
segments to show up on disk. Recently, the test has become particularly
flaky (~10% flaky in the last couple days according to our test tracking
server), though I haven't been able to determine the cause of this new
flakiness.

Regardless, the test has been at least a little flaky for much longer
than the last couple of days, and the cause seems to be that in TSAN, we
might not always hit the expected number of WAL segments in the allotted
amount of time.

Upon inspecting a flamegraph of the test, it seems like a decent
percentage of cycles are spent compressing the WALs, so I've removed the
log compression codec for the test.

Without this fix, the test failed 100/100 times with 4 stress threads in
TSAN mode. With it, it passed 1000/1000.

Change-Id: Ic19d33a5e43aaae21c1cb6273a09a09b1b91f92c
Reviewed-on: http://gerrit.cloudera.org:8080/11979
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
8 weeks ago[webui] fixed build and corrected a typo
Alexey Serbin [Wed, 21 Nov 2018 04:22:56 +0000 (20:22 -0800)] 
[webui] fixed build and corrected a typo

Fixed build for TCMALLOC-enabled environments.
Fixed a typo in 'Memory usage by subsystem' WebUI page.
As additional tiny change, take advantage of the 'using' directive
for std::ostringstream.

This is a follow-up to 7437626f42b30811720d59e792bc905662efd2b3.

Change-Id: If1f0b3df0ac4178609914322741511d8fff87b24
Reviewed-on: http://gerrit.cloudera.org:8080/11973
Reviewed-by: Will Berkeley <wdberkeley@gmail.com>
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <awong@cloudera.com>
8 weeks agobuild: don't instantiate const objects with no user-defined constructor
Andrew Wong [Wed, 21 Nov 2018 00:37:53 +0000 (16:37 -0800)] 
build: don't instantiate const objects with no user-defined constructor

It seems that some compilers disallow instantiating const objects that
don't provide a user-provided constructor. This would surface as the
following error.

error: default initialization of an object of const type 'const ClusterInfo' requires a user-provided default constructor

Change-Id: Ia21baed51e035d9005453fd7d08274c9c23e74d7
Reviewed-on: http://gerrit.cloudera.org:8080/11971
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
8 weeks agobuild: don't use constexpr lambdas
Andrew Wong [Wed, 21 Nov 2018 00:37:30 +0000 (16:37 -0800)] 
build: don't use constexpr lambdas

We use constexpr lambdas, and some compilers are fine with it.
Officially, they aren't supported until C++17, so this patch replaces
such instances with const lambdas.

This would yield a build failure in some environments.

See https://en.cppreference.com/w/cpp/language/lambda for more details
about constexpr lambdas in C++17.

Change-Id: I9d1bdb84d9e6ee5c6a4a920d46f5daee68975208
Reviewed-on: http://gerrit.cloudera.org:8080/11970
Reviewed-by: Adar Dembo <adar@cloudera.com>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
Tested-by: Kudu Jenkins
8 weeks ago[webui] Fancy table for /mem-trackers and sortable tables
Will Berkeley [Tue, 20 Nov 2018 22:46:09 +0000 (14:46 -0800)] 
[webui] Fancy table for /mem-trackers and sortable tables

This fancifies the table of trackers on /mem-trackers in the style of
6ae9ecbe2595090c78e7afd271aae9d04dd4d0b5. It also adds the ability to
sort some tables by some numeric columns. Namely:

- the /mem-trackers trackers table is sortable by current consumption
  and peak consumption
- the /tablets page tablets tables are sortable by on-disk size
- the /maintenance-manager "non-running op" table is sortable by RAM
  anchored, logs retained, and perf.

I tested this change out manually, verifying that ascending and
descending sort looked good. Unfortunately, the bootstrap table
library doesn't seem to use a stable sort.

Change-Id: Ibdf8e7bd82fe2b95e699b8bb238a9cf0e5a7e727
Reviewed-on: http://gerrit.cloudera.org:8080/11968
Reviewed-by: Adar Dembo <adar@cloudera.com>
Reviewed-by: Andrew Wong <awong@cloudera.com>
Tested-by: Kudu Jenkins
8 weeks agobuild: explicitly declare decimal Set/Get<>s
Andrew Wong [Wed, 21 Nov 2018 00:36:01 +0000 (16:36 -0800)] 
build: explicitly declare decimal Set/Get<>s

Some compilers aren't clever enough to pick up on implicit instantiation
of templates based on usage in a given compilation unit, and instead we
need to explicitly instantiate them. This surfaced as the following
build error:

Undefined symbols for architecture x86_64:
  "kudu::Status kudu::KuduPartialRow::Set<kudu::TypeTraits<(kudu::DataType)15> >(int, kudu::TypeTraits<(kudu::DataType)15>::cpp_type const&, bool)", referenced from:
      kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)15> >::GenerateSplitRows(kudu::client::KuduSchema const&) const in all_types-itest.cc.o
      kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)15> >::GenerateRowKey(kudu::client::KuduInsert*, int, int) const in all_types-itest.cc.o
  "kudu::Status kudu::KuduPartialRow::Set<kudu::TypeTraits<(kudu::DataType)16> >(int, kudu::TypeTraits<(kudu::DataType)16>::cpp_type const&, bool)", referenced from:
      kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)16> >::GenerateSplitRows(kudu::client::KuduSchema const&) const in all_types-itest.cc.o
      kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)16> >::GenerateRowKey(kudu::client::KuduInsert*, int, int) const in all_types-itest.cc.o
  "kudu::Status kudu::KuduPartialRow::Set<kudu::TypeTraits<(kudu::DataType)17> >(int, kudu::TypeTraits<(kudu::DataType)17>::cpp_type const&, bool)", referenced from:
      kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)17> >::GenerateSplitRows(kudu::client::KuduSchema const&) const in all_types-itest.cc.o
      kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)17> >::GenerateRowKey(kudu::client::KuduInsert*, int, int) const in all_types-itest.cc.o
  "kudu::Status kudu::client::KuduScanBatch::RowPtr::Get<kudu::TypeTraits<(kudu::DataType)15> >(int, kudu::TypeTraits<(kudu::DataType)15>::cpp_type*) const", referenced from:
      kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)15> >::VerifyRowKey(kudu::client::KuduScanBatch::RowPtr const&, int, int) const in all_types-itest.cc.o
  "kudu::Status kudu::client::KuduScanBatch::RowPtr::Get<kudu::TypeTraits<(kudu::DataType)16> >(int, kudu::TypeTraits<(kudu::DataType)16>::cpp_type*) const", referenced from:
      kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)16> >::VerifyRowKey(kudu::client::KuduScanBatch::RowPtr const&, int, int) const in all_types-itest.cc.o
  "kudu::Status kudu::client::KuduScanBatch::RowPtr::Get<kudu::TypeTraits<(kudu::DataType)17> >(int, kudu::TypeTraits<(kudu::DataType)17>::cpp_type*) const", referenced from:
      kudu::client::IntKeysTestSetup<kudu::client::KeyTypeWrapper<(kudu::DataType)17> >::VerifyRowKey(kudu::client::KuduScanBatch::RowPtr const&, int, int) const in all_types-itest.cc.o

Change-Id: If280f69b5736737a55862486c98f434cfb8e8072
Reviewed-on: http://gerrit.cloudera.org:8080/11969
Reviewed-by: Adar Dembo <adar@cloudera.com>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
Tested-by: Kudu Jenkins
8 weeks ago[docs] note on unavailable tservers during rebalancing
Alexey Serbin [Tue, 20 Nov 2018 22:01:35 +0000 (14:01 -0800)] 
[docs] note on unavailable tservers during rebalancing

Change-Id: Ifa7a7590a90cee2e2d1399655621ec3618e51b25
Reviewed-on: http://gerrit.cloudera.org:8080/11966
Reviewed-by: Will Berkeley <wdberkeley@gmail.com>
Tested-by: Alexey Serbin <aserbin@cloudera.com>
8 weeks ago[docs] Add note that it is safe to stop and restart the rebalancer
Will Berkeley [Tue, 20 Nov 2018 18:33:30 +0000 (10:33 -0800)] 
[docs] Add note that it is safe to stop and restart the rebalancer

Change-Id: I14745e1b31565f31ff8ff7c7d465eee38f2a22bc
Reviewed-on: http://gerrit.cloudera.org:8080/11963
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
Reviewed-by: Mitch Barnett <mbarnett@cloudera.com>
Tested-by: Will Berkeley <wdberkeley@gmail.com>
8 weeks ago[util] more information on fault injection
Alexey Serbin [Tue, 20 Nov 2018 05:00:45 +0000 (21:00 -0800)] 
[util] more information on fault injection

Log more information on injected fault: name of the process and PID.
That helps to parse concatenated logs: e.g., that useful in parsing
logs from scenarios run via dist-test.

Change-Id: Id0f094006a89fd271091d1bde13d301ea359ae61
Reviewed-on: http://gerrit.cloudera.org:8080/11960
Tested-by: Kudu Jenkins
Reviewed-by: Will Berkeley <wdberkeley@gmail.com>
8 weeks ago[tools] Don't count tombstoned replicas in `kudu remote_replica check`
Will Berkeley [Tue, 20 Nov 2018 08:37:00 +0000 (00:37 -0800)] 
[tools] Don't count tombstoned replicas in `kudu remote_replica check`

Counting them as not running made the tool pretty useless on any
realistic cluster, where it's expected to have some tombstones. For
example, it wasn't possible to use this tool to tell when all the tablet
replicas on a tablet server had finished bootstrapping.

I tested this against a server that previously failed the check because
it had about 6000/10000 replicas tombstoned in STOPPED and INITIALIZED
states. It now passes the check.

Change-Id: Ibbdde768cce5f1d27064b7276944a585b41a2f03
Reviewed-on: http://gerrit.cloudera.org:8080/11962
Reviewed-by: Andrew Wong <awong@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Attila Bukor <abukor@apache.org>
8 weeks ago[tools] ksck: Add information about replica counts to plain ksck output
Will Berkeley [Mon, 19 Nov 2018 23:44:59 +0000 (15:44 -0800)] 
[tools] ksck: Add information about replica counts to plain ksck output

This adds some information about replica counts on tablet servers to the
output of ksck when ksck is in PLAIN_* mode (i.e. not JSON output). It
outputs a 5-number summary of the distribution of replicas and lists any
outliers:

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 1646
 First Quartile | 3672
 Median         | 4075
 Third Quartile | 4242
 Maximum        | 4600

Tablet Replica Count Outliers
 Type  |               UUID               |              Host              | Replica Count
-------+----------------------------------+--------------------------------+---------------
 Small | cc32936bc8594948a04fd4240da36aed | vc1304.halxg.cloudera.com:7050 | 1646

In PLAIN_FULL mode it additionally outputs the replica count for every
tablet server:

Tablet Replica Count by Tablet Server
               UUID               |              Host              | Replica Count
----------------------------------+--------------------------------+---------------
 09d6bf7a02124145b43f43cb7a667b3d | vc1314.halxg.cloudera.com:7050 | 100
 23d473f441674d43807fd9e631862bfd | vc1308.halxg.cloudera.com:7050 | 100
 2fb5cdac22b0418bb2df456906e42eb4 | vc1306.halxg.cloudera.com:7050 | 101
 70f7ee61ead54b1885d819f354eb3405 | vc1316.halxg.cloudera.com:7050 | 95
 72fcec63e96f4248ae39d114eb3cd7c9 | vc1318.halxg.cloudera.com:7050 | 94
 86708813b37a44bd8e92c711211c8685 | vc1310.halxg.cloudera.com:7050 | 96
 a662440710624c02bd5612df32cb0235 | vc1302.halxg.cloudera.com:7050 | 101
 c9633273962a4521a32d5e177a118a84 | vc1312.halxg.cloudera.com:7050 | 101
 cc32936bc8594948a04fd4240da36aed | vc1304.halxg.cloudera.com:7050 | 76

I also tested it against an empty cluster.

There's no unit tests added, just because our current testing setup for
ksck makes it really painful to add one for this, and it seemed easy
enough to check out manually. Probably, a follow up should straighten
out ksck-test to make testing ksck changes easier.

Change-Id: I7e5373033ab84c1e34f9519eb9bd4e04a652c595
Reviewed-on: http://gerrit.cloudera.org:8080/11958
Reviewed-by: Andrew Wong <awong@cloudera.com>
Tested-by: Kudu Jenkins
2 months ago[tests] fix flake in TestRandomHistoryGCWorkload
Alexey Serbin [Fri, 16 Nov 2018 19:55:39 +0000 (11:55 -0800)] 
[tests] fix flake in TestRandomHistoryGCWorkload

This patch fixes a flake most prominent in TSAN builds for the
RandomizedTabletHistoryGcITest.TestRandomHistoryGCWorkload scenario
of the tablet_history_gc-itest suite.

Before: dist_test --stress_cpu_threads=16: 256 out of 256 failed
  http://dist-test.cloudera.org/job?job_id=aserbin.1542394875.56611

After:  dist_test --stress_cpu_threads=16:   0 out of 256 failed
  http://dist-test.cloudera.org/job?job_id=aserbin.1542397028.71131

In addition, this patch also contains a minor code clean up of the
tablet_history_gc-itest suite to take advantage of the 'using'
declarations.

Change-Id: I8d146d4b83c8d488d3c1766a53fe4a4d322b6590
Reviewed-on: http://gerrit.cloudera.org:8080/11945
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
2 months agoFix some issues in HMS and Sentry test fixtures
Adar Dembo [Fri, 16 Nov 2018 18:04:47 +0000 (10:04 -0800)] 
Fix some issues in HMS and Sentry test fixtures

1. The HMS or Sentry client object may be null in TearDown. If we don't
   check, we'll deref a null pointer if there's an error during SetUp()
   (e.g. if the HMS or Sentry server failed to start).
2. The HMSCatalogTest fixture should properly chain to its superclass.

Change-Id: Ib0376b972fe6add6d9312aea6944c9ab1a03f25f
Reviewed-on: http://gerrit.cloudera.org:8080/11943
Reviewed-by: Hao Hao <hao.hao@cloudera.com>
Tested-by: Kudu Jenkins
2 months ago[kudu-jepsen] fixed typo in README.adoc
Alexey Serbin [Wed, 14 Nov 2018 23:47:00 +0000 (15:47 -0800)] 
[kudu-jepsen] fixed typo in README.adoc

For the kudu-jepsen module, './gradlew' should be run from the
upper-level (parent) directory (i.e. $KUDU_ROOT/java).

Also, updated the wording and path specification for the example
involving the 'sshKeyPath' run-time property.

Change-Id: I01a62d135aad560fa26f7a3ccf746813d3ec279e
Reviewed-on: http://gerrit.cloudera.org:8080/11932
Tested-by: Kudu Jenkins
Reviewed-by: Hao Hao <hao.hao@cloudera.com>
2 months agoraft_consensus_nonvoter-itest: deflake a bit
Adar Dembo [Fri, 9 Nov 2018 19:03:26 +0000 (11:03 -0800)] 
raft_consensus_nonvoter-itest: deflake a bit

I saw a failure in ReplicaBehindWalGcThresholdITest.ReplicaReplacement
(GetParam() was (1, false)) just after the master was restarted:

  raft_consensus_nonvoter-itest.cc:2070: Failure
  Failed
  Bad status: Service unavailable: Leader not yet ready to serve requests

This is odd as there's a WaitForCatalogManager() call in there, so why would
a subsequent GetTabletLocations RPC return this ServiceUnavailable? As best
I can tell, the only way for this to happen is if the attempt to grab the
leadership lock from within the ListTables RPC (sent from
WaitForCatalogManager()) returns IllegalState, which it'll do if the
UUID in the master's cstate doesn't match the UUID on disk. Perhaps this can
happen during a leader master election; maybe the cstate's UUID becomes
empty for a little while? If that's true, this should fix the problem by
considering IllegalState to be a non-final state and continuing the loop.

I couldn't repro this failure, but Alexey managed to do so in a dist-test
loop with special latency injection enabled. Without the fix, 93 out of 256
runs failed, and with the fix, none failed.

Change-Id: I8192bd669e7e309943ea82718dd715238d520bbd
Reviewed-on: http://gerrit.cloudera.org:8080/11918
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
Tested-by: Kudu Jenkins
2 months agotest: deflake TestRestartWithPendingCommitFromFailedOp
Andrew Wong [Wed, 14 Nov 2018 03:42:07 +0000 (19:42 -0800)] 
test: deflake TestRestartWithPendingCommitFromFailedOp

The test would fail to create the tablet due to the injected failure,
when it should have really waited until the tablet was bootstrapped
before injecting errors.

The test failed 6/1000 times with stress and with a seed of -922249903.
With the fix, 1000/1000 passed.

This isn't the only source of flakiness in ts_recovery-itest, but it's
something.

Change-Id: Ic76b5f120aee7327f6cee7324b862bb89c5662ab
Reviewed-on: http://gerrit.cloudera.org:8080/11929
Tested-by: Andrew Wong <awong@cloudera.com>
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
Reviewed-by: Adar Dembo <adar@cloudera.com>
2 months ago[test] Clear the KuduClientCache between tests
Grant Henke [Tue, 13 Nov 2018 14:54:22 +0000 (08:54 -0600)] 
[test] Clear the KuduClientCache between tests

The KuduClientCache is a Scala object, effectively
a singleton, that will live across tests. If the cache
is not cleared between tests, client state and
configurations could carry over from another test
and cause failures that are difficult to debug.

Change-Id: I1d989c00f9b0407e1db4b11df68080ef20cd471c
Reviewed-on: http://gerrit.cloudera.org:8080/11924
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
2 months ago[java] Upgrade to Spark 2.4
Grant Henke [Thu, 8 Nov 2018 22:04:28 +0000 (16:04 -0600)] 
[java] Upgrade to Spark 2.4

In Spark 2.4 spark-avro is now a part of Spark itself.
This change migrates the Kudu spark-avro
dependencies and adds a test to ensure that
the functionality does not break.

Change-Id: Id1f8de543276c4dc82a57c4a2228ae2374f2d87f
Reviewed-on: http://gerrit.cloudera.org:8080/11912
Reviewed-by: Adar Dembo <adar@cloudera.com>
Reviewed-by: Hao Hao <hao.hao@cloudera.com>
Tested-by: Grant Henke <granthenke@apache.org>
2 months agoKUDU-2599: fix flaky DefaultSourceTest.testSocketReadTimeoutPropagation
Adar Dembo [Fri, 9 Nov 2018 05:41:24 +0000 (21:41 -0800)] 
KUDU-2599: fix flaky DefaultSourceTest.testSocketReadTimeoutPropagation

This test is just about checking the propagation of an option from the
DefaultSource to the KuduRelation. However, building a DataFrame involves a
connection to the cluster, and if we set socketReadTimeoutMs to 1, it's
going to be very difficult to connect, especially if we're testing
TSAN-instrumented binaries whose thread creation is inherently slower.

The fix? Use a magic number larger than '1'. It's that simple.

Change-Id: I81a20f776c115caf3bd720bc532bbeb5fc69bc42
Reviewed-on: http://gerrit.cloudera.org:8080/11916
Reviewed-by: Andrew Wong <awong@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <granthenke@apache.org>
2 months ago(01/05) delta_store: avoid copying deleted row data in ApplyUpdates
Adar Dembo [Thu, 1 Nov 2018 22:58:56 +0000 (15:58 -0700)] 
(01/05) delta_store: avoid copying deleted row data in ApplyUpdates

When applying deltas, the scan path will first populate a selection vector
with deletes (i.e. a row is unset if there's a relevant DELETE for it), and
then use it to enable two optimizations:
1. Short-circuit out if all rows in the block have been deleted.
2. Skip predicate evaluation and base data copying for deleted rows.

To this we can add a third optimization: don't apply a delta whose mutation
is to a deleted row. Note that it's not incorrect to apply such deltas (as
we'll skip deleted rows when serializing the scan response to the client);
it's just unnecessary work.

I wrote a microbenchmark to evaluate the impact of this optimization. Below
is the timing and perf stat output of the microbenchmark before and after
applying the optimization (specifically, applying this patch, then
commenting out the filtering in DeltaPrepare::ApplyUpdates for the first
run, and uncommenting it for the next run).

  Time spent running 1000 scans with 0 deletes: real 0.523s user 0.524s sys 0.000s
  Time spent running 1000 scans with 10 deletes: real 0.515s user 0.516s sys 0.000s
  Time spent running 1000 scans with 100 deletes: real 0.512s user 0.512s sys 0.000s
  Time spent running 1000 scans with 1000 deletes: real 0.553s user 0.552s sys 0.000s

   Performance counter stats for 'bin/deltamemstore-test --gtest_filter=*Varying* --benchmark_num_passes=1000':

       2201.029290      task-clock (msec)         #    0.991 CPUs utilized
                 5      context-switches          #    0.002 K/sec
                 0      cpu-migrations            #    0.000 K/sec
             4,950      page-faults               #    0.002 M/sec
     8,276,723,832      cycles                    #    3.760 GHz
    24,539,935,941      instructions              #    2.96  insn per cycle
     4,709,709,705      branches                  # 2139.776 M/sec
        12,631,579      branch-misses             #    0.27% of all branches

       2.220370506 seconds time elapsed

  Time spent running 1000 scans with 0 deletes: real 0.474s user 0.475s sys 0.000s
  Time spent running 1000 scans with 10 deletes: real 0.475s user 0.472s sys 0.004s
  Time spent running 1000 scans with 100 deletes: real 0.478s user 0.476s sys 0.004s
  Time spent running 1000 scans with 1000 deletes: real 0.550s user 0.552s sys 0.000s

   Performance counter stats for 'bin/deltamemstore-test --gtest_filter=*Varying* --benchmark_num_passes=1000':

       2074.795741      task-clock (msec)         #    0.990 CPUs utilized
                23      context-switches          #    0.011 K/sec
                 1      cpu-migrations            #    0.000 K/sec
             4,951      page-faults               #    0.002 M/sec
     7,675,100,058      cycles                    #    3.699 GHz
    23,100,692,252      instructions              #    3.01  insn per cycle
     4,539,777,117      branches                  # 2188.060 M/sec
        11,819,267      branch-misses             #    0.26% of all branches

       2.096193851 seconds time elapsed

Note: I originally wrote this patch thinking it was necessary for diff scan
correctness, but have since convinced myself that it's just an optimization.

Change-Id: I6a646d0816a96e9aba486c0d0a1e6b4a7e15c144
Reviewed-on: http://gerrit.cloudera.org:8080/11856
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <granthenke@apache.org>
Reviewed-by: Mike Percy <mpercy@apache.org>
2 months agoKUDU-2378: fix another aligned load crash
Adar Dembo [Thu, 8 Nov 2018 20:18:36 +0000 (12:18 -0800)] 
KUDU-2378: fix another aligned load crash

The TestKuduBackup.testRandomBackupAndRestore test from the kudu-backup
project would crash a TSAN master from time to time, with a stack trace
ending in Variant::Reset. After some debugging, this turned out to be a
holdover from KUDU-2378, where clang emitted a movaps (Aligned Load)
instruction on a value that wasn't aligned. The holdover was missed
because it was a static_cast (the original search covered
reinterpret_casts); I did a search for other bad casts but found none.

Why TSAN? Probably because it's the only build to use libc++, whose
std::string uses the Small String Optimization to embed the data into the
string itself, thus raising the possibility of the string's data being
unaligned. By comparison, libstdc++'s COW std::string uses a separate
allocation for string data, which is likely always 16-byte aligned.

The patch includes a unit test that would repro the crash 100% of the time
when built for TSAN.

Change-Id: I13e23a9d88db878071faeff7b9de06d1bdd98357
Reviewed-on: http://gerrit.cloudera.org:8080/11911
Tested-by: Adar Dembo <adar@cloudera.com>
Reviewed-by: Todd Lipcon <todd@apache.org>
2 months ago[dist-test] build environment for tests in loop mode
Alexey Serbin [Wed, 7 Nov 2018 21:36:45 +0000 (13:36 -0800)] 
[dist-test] build environment for tests in loop mode

This patch fixes the issue of not deploying corresponding DATA_FILES
by dist-test when running a test via the loop sub-command.

I used the following command to run the ts_location_assignment-itest
in dist-test:

  KUDU_ALLOW_SLOW_TESTS=1 \
    ../../build-support/dist_test.py loop -n 32 \
    ./bin/ts_location_assignment-itest --stress_cpu_threads=16

Prior to this patch, all runs failed because of the absence of
assign-location.py script at dist-test slaves:
  http://dist-test.cloudera.org/job?job_id=aserbin.1541627808.8037

With this patch, all runs succeeded:
  http://dist-test.cloudera.org/job?job_id=aserbin.1541627254.115588

Change-Id: I487194a396635bbaab457795bb24c0063eebbe5d
Reviewed-on: http://gerrit.cloudera.org:8080/11905
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Alexey Serbin <aserbin@cloudera.com>
2 months agogradle: expose testRandomSeed property
Adar Dembo [Thu, 8 Nov 2018 08:40:27 +0000 (00:40 -0800)] 
gradle: expose testRandomSeed property

In order to read a system property in tests, it isn't enough to add a
System.getProperty() call to the test; we need to expose the property in
gradle so that it's properly passed through to the test. With this patch,
testRandomSeed can be set using either -D or -P.

Note that even with testRandomSeed overridden, the PRNG didn't produce
deterministic results in TestKuduBackup.testRandomBackupAndRestore, but
perhaps that's a quirk of Scala's Random class.

Change-Id: I4873d27998f770a45dd9cc85f84ec8c146261b3d
Reviewed-on: http://gerrit.cloudera.org:8080/11908
Reviewed-by: Grant Henke <granthenke@apache.org>
Tested-by: Adar Dembo <adar@cloudera.com>
2 months ago[backup] Ensure random tests are reproducible
Grant Henke [Thu, 8 Nov 2018 15:32:10 +0000 (09:32 -0600)] 
[backup] Ensure random tests are reproducible

Changes the use of Random in the backup tests to
ensure that passing the same seed will result in the
same test being run.

Change-Id: I4f376ba5709ec118c4327881f7c864b23f92ac4d
Reviewed-on: http://gerrit.cloudera.org:8080/11909
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Grant Henke <granthenke@apache.org>
2 months agoAdd "network_plane" as part of ConnectionId
Michael Ho [Sat, 13 Oct 2018 07:08:19 +0000 (00:08 -0700)] 
Add "network_plane" as part of ConnectionId

The motivation for doing so is to allow N services on the same host to
be multiplexed on M different connections. For instance, a server may
host multiple KRPC based services: one for control command and one for
data transfer. Separating the connections between the control channel
and the data channel prevents unnecessary delays of the control commands
due to being stuck behind large data transfers from client to server.

By default, the network_plane of a new ConnectionId is not set.
A user can change it to a different value by calling
Proxy::set_network_plane() on the ConnectionId.

Change-Id: I6767e631fd9530ea54f5ed63ff4c8c179ab216b2
Reviewed-on: http://gerrit.cloudera.org:8080/11681
Reviewed-by: Adar Dembo <adar@cloudera.com>
Tested-by: Kudu Jenkins
2 months ago[cmake] no empty KUDU_DATA_FILES ctest env variables
Alexey Serbin [Wed, 7 Nov 2018 23:52:07 +0000 (15:52 -0800)] 
[cmake] no empty KUDU_DATA_FILES ctest env variables

This patch introduces an update on ADD_KUDU_TEST() custom function:
if DATA_FILES is not specified for a test, don't add KUDU_DATA_FILES
variable into its ctest environment.

Prior to this patch, ctest environment of every test binary would
contain KUDU_DATA_FILES variable.  If DATA_FILES was specified for a
test, it was set to corresponding value, otherwise it was set to
an empty string.

Change-Id: I2b255f6148649cd282bab70fb19c3321fec01adb
Reviewed-on: http://gerrit.cloudera.org:8080/11907
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <adar@cloudera.com>
2 months agorun-test: don't report leaks that don't fail test
Andrew Wong [Mon, 29 Oct 2018 06:46:26 +0000 (23:46 -0700)] 
run-test: don't report leaks that don't fail test

LeakSanitizer will report a leak when allocating a string in
SuperviseThread.  It's unclear why this is the case, but upon inspecting
the code, it seems like a false positive. The stack trace is as follows:

=================================================================
==93677==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 58 byte(s) in 1 object(s) allocated from:
    #0 0x5318c8 in operator new(unsigned long) /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:92
    #1 0x3ae3a9c3c8 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (/usr/lib64/libstdc++.so.6+0x3ae3a9c3c8)
    #2 0x3ae3a9d19a in std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long) (/usr/lib64/libstdc++.so.6+0x3ae3a9d19a)
    #3 0x3ae3a9d5eb in std::string::reserve(unsigned long) (/usr/lib64/libstdc++.so.6+0x3ae3a9d5eb)
    #4 0x3ae3a9d770 in std::string::append(unsigned long, char) (/usr/lib64/libstdc++.so.6+0x3ae3a9d770)
    #5 0x7f518f799c12 in strings::SubstituteAndAppend(std::string*, StringPiece, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&) ../src/kudu/gutil/strings/substitute.cc:110:3
    #6 0x536e76 in strings::Substitute(StringPiece, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&, strings::internal::SubstituteArg const&) ../src/kudu/gutil/strings/substitute.h:188:3
    #7 0x7f5190590860 in kudu::Thread::SuperviseThread(void*) ../src/kudu/util/thread.cc:607:17
    #8 0x3ae0e079d0 in start_thread (/lib64/libpthread.so.0+0x3ae0e079d0)
    #9 0x3ae0ae88fc in clone (/lib64/libc.so.6+0x3ae0ae88fc)

This appears to be affecting a number tests, but generally only lines #0
and #1 are present in the logs, making them difficult to debug (a
developer would have to rerun the test with specific ASAN_OPTIONS to
unwind the stacktrace more). Namely, exactly_once_writes-itest
(KUDU-2517), kudu-admin-test (KUDU-2583), and rebalancer-tool-test
(untracked via Jira) all show the top of the above stack trace, and
based on the full stack trace, it seems these are all false positives.

The presence of issues like
https://github.com/google/sanitizers/issues/757 confirms that
LeakSanitizer can report false positives in workloads with high
concurrency. Generally, the test binary will return an error in the face
of real leaks, but in tests like the ones mentioned, the test may log
messages reporting leaks, but not actually return an error because the
"leak" was transient (e.g. see GenericServiceImpl::CheckLeaks).

We currently inject errors into JUnit XML report if any leaks are
reported, even for false positives, since the leak messages still find
their way into the logs. This patch updates this to only inject these
errors if the test also returned an error.

For clarity, I also threw in a log statement to
GenericServiceImpl::CheckLeaks denoting false positives.

Change-Id: I1f199795c48bd9b6106110aae132ec165eb0f647
Reviewed-on: http://gerrit.cloudera.org:8080/11886
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <awong@cloudera.com>
2 months ago[catalog_manager] cleanup on InitTokenSigner
Alexey Serbin [Wed, 7 Nov 2018 18:36:39 +0000 (10:36 -0800)] 
[catalog_manager] cleanup on InitTokenSigner

With this patch, the catalog manager no longer issues empty write
operations to the system tablet upon initializing TokenSigner.
Additionally, the catalog manager now logs about deleted TSK keys.

Change-Id: I68eb1972c0ab330c7739e6901ebe36e4190144cc
Reviewed-on: http://gerrit.cloudera.org:8080/11902
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <awong@cloudera.com>
2 months agodocs: Use a non-SSH method of installing Gerrit commit hook
Mike Percy [Wed, 7 Nov 2018 03:33:02 +0000 (19:33 -0800)] 
docs: Use a non-SSH method of installing Gerrit commit hook

Apparently, contributors in China cannot connect to the Gerrit SSH port.
Because of this, they cannot follow the How to Contribute directions for
installing the Gerrit commit hook.

This patch updates those instructions to use HTTPS instead of SSH for
installing the Gerrit commit hook, which should work for everyone.

Change-Id: Ibdb024aafcd601f740ae73d83c54f9659cc5a3a9
Reviewed-on: http://gerrit.cloudera.org:8080/11896
Reviewed-by: Alexey Serbin <aserbin@cloudera.com>
Reviewed-by: Grant Henke <granthenke@apache.org>
Tested-by: Grant Henke <granthenke@apache.org>