Mike Miller [Wed, 18 May 2022 15:19:09 +0000 (15:19 +0000)]
Create TabletDirectory classes (#2688)
* Create TabletDirectory and RelativeTabletDirectory
* Move Reference class and improve it so other classes can be children
* Add validate method to StoredTabletFile
* Use TabletDirectory in TabletFile
* Refactor some GC code
Christopher Tubbs [Tue, 17 May 2022 21:10:02 +0000 (17:10 -0400)]
Merge branch '1.10'
EdColeman [Tue, 17 May 2022 20:58:03 +0000 (16:58 -0400)]
Update documentation with purpose of the "cache" (#2714)
* Update documentation with purpose of the "cache"
- Remove the TODO with an explanation that the class supports legacy
clients.
* Update core/src/main/java/org/apache/accumulo/fate/zookeeper/ZooCacheFactory.java
Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
* Fix formatting
Co-authored-by: Ed Coleman etcoleman <edcoleman@apache.org>
Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
Mike Miller [Tue, 17 May 2022 19:26:00 +0000 (19:26 +0000)]
General improvements to CI (#2715)
Dave Marion [Tue, 17 May 2022 15:15:51 +0000 (11:15 -0400)]
Resolve todo by removing unused MajorCompactionRequest.openReader method (#2710)
Related to #2699
Christopher Tubbs [Mon, 16 May 2022 12:32:04 +0000 (08:32 -0400)]
Bump libthrift to 0.16.0 (#2706)
Christopher Tubbs [Mon, 16 May 2022 12:31:49 +0000 (08:31 -0400)]
Delete undefined thrift-generated files (#2705)
Update generate-thrift.sh to detect and delete any files that were
generated from a previous version of the `*.thrift` IDL files, but are
no longer generated in the current version.
EdColeman [Sat, 14 May 2022 23:09:38 +0000 (19:09 -0400)]
Update test documentation to clarify purpose. (#2704)
Christopher Tubbs [Fri, 13 May 2022 22:18:42 +0000 (18:18 -0400)]
Minor little cleanups (#2695)
* Remove or narrow unnecessary or broad warnings suppressions
* Shorten the lookup of annotations in Property.java
* Use Streams where they simplify or improve the processing of big loops
* Shorten a few lines with casts using var
* Improve use of generics in a few places
* Fix license header in proto generation to be consistent with project
* Avoid unnecessary reflection to load DistCp in ShellServerIT
* Remove unneeded super()
Emily [Fri, 13 May 2022 22:09:16 +0000 (18:09 -0400)]
Add front-end console logging in Monitor (#2679)
* Add console logging to REST calls
* Consolidate REST GET call code into single function
* Consolidate REST POST code and update in-line args
Dave Marion [Fri, 13 May 2022 21:47:39 +0000 (17:47 -0400)]
Consolidated duplicate Thrift client code (#2647)
* Consolidated Thrift client methods, removed ManagerClient, ServerClient and ReplicationClient
* Add missing dependency
* Removed unused ClientExec, moved ClientExecReturn into replication code
* Moved ThriftClientTypes to a new subpackage and created new classes for
each implementation that required special logic. Added an executeVoid
method and used that to remove the extraneous `return null` lines.
* Consolidated code added in #2622 into utility method
* Narrow the exceptions being thrown from Exec methods to just Thrift
TException (plus RuntimeExceptions)
* Remove catch clauses that aren't needed due to never being thrown or
redundant with another catch clause
* Remove unnecessary class nesting layers
* Remove unnecessary second generic parameter for ThriftClientTypes
* Use AtomicBoolean to force pass by reference, instead of trying to
pass boxed Boolean objects, which is a bit dubious, because it's not
clear to the reader that it's not being auto-boxed/auto-unboxed and
using pass by value instead of the intended pass by reference
* Make ThriftClientTypes abstract, to make implementing subclasses more
clear and to more easily identify classes that don't implement certain
critical methods
* IDE also optimized a few things, removing extra parens, converting
some anonymous inner classes to lambdas, using method references for
simple lambdas, and removing unnecessary return statements
* Get rid of unnecessary passing AccumuloConfiguration
* Avoid unnecessary rewrapping of RuntimeExceptions as another
RuntimeException when creating the processors
* Fix extremely trivial warnings suppression
* Use narrower exceptions
* Rethrow exception in missing case, where cause of
TableNotFoundException is not NamespaceNotFoundException
* Add compile-time dependency to shell (hopefully we can fix that in
future, but it's hard to track down)
* Implement the execute method in ReplicationCoordinatorThriftClient
* Fix failure in NamespacesIT
Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
Dave Marion [Fri, 13 May 2022 16:35:29 +0000 (12:35 -0400)]
Make variables/functions more clear in TabletLocatorImplTest (#2702)
Closes #2698
EdColeman [Thu, 12 May 2022 14:26:21 +0000 (10:26 -0400)]
Narrow static util method, writer not needed, use ZooReader (#2694)
Dom G [Wed, 11 May 2022 22:32:01 +0000 (18:32 -0400)]
Improve MiniAccumuloClusterTest (#2693)
Emily [Wed, 11 May 2022 22:06:24 +0000 (22:06 +0000)]
Add check for ReplicationTable online status in Monitor (#2666)
Dom G [Wed, 11 May 2022 18:32:24 +0000 (14:32 -0400)]
Convert deadTServers and badTServers tables to DataTables (#2685)
* replace tables with DataTables
* format code via jslint standards
Dave Marion [Wed, 11 May 2022 16:10:18 +0000 (12:10 -0400)]
Prevent NPE in TabletServer.run by starting Thrift server after CompactionManager (#2684)
Closes #2669
Dave Marion [Wed, 11 May 2022 16:09:26 +0000 (12:09 -0400)]
Execute Caffeine tasks in current Thread for tests (#2691)
* Execute Caffeine tasks in current Thread for tests
As described at https://github.com/ben-manes/caffeine/wiki/Testing, Caffeine
delegates maintenance tasks (eviction, refreshes, etc.) to background threads
to take advantage of batching. This change implements the suggestion at
https://github.com/ben-manes/caffeine/issues/455#issuecomment-
689398669 to
run all of the maintenance tasks in the current Thread for testing. Using
Cache.cleanup as suggested in the wiki page did not appear to work, but this
approach allows us to remove all of the calls to Thread.sleep in the tests.
Emily [Tue, 10 May 2022 21:59:48 +0000 (21:59 +0000)]
Add declaration for scansList prior to usage (#2687)
Christopher Tubbs [Tue, 10 May 2022 21:28:37 +0000 (17:28 -0400)]
Merge branch 'single-node-props'
This closes #1225, #1454, #1809
EdColeman [Tue, 10 May 2022 21:13:08 +0000 (17:13 -0400)]
Single node prop store (#2569)
Convert multiple node property configuration to a single-node encoded store.
This reduces the number of ZooKeeper watchers needed to track property
changes, and sets the groundwork for atomic changes to multiple properties
in the future.
Co-authored-by: Ed Coleman etcoleman <edcoleman@apache.org>
Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
Christopher Tubbs [Wed, 27 Apr 2022 19:51:56 +0000 (15:51 -0400)]
Minor enhancements to ZK utilities
Christopher Tubbs [Wed, 27 Apr 2022 19:31:22 +0000 (15:31 -0400)]
Include classloader context in log message
Christopher Tubbs [Wed, 27 Apr 2022 17:46:24 +0000 (13:46 -0400)]
Improve getting parent from AccumuloConfiguration
* Also improve the singleton loading of DefaultConfiguration to reduce
object creation
Christopher Tubbs [Tue, 26 Apr 2022 19:53:32 +0000 (15:53 -0400)]
Remove isPropertySet option for caching
This is an incremental change from the single-node property changes
EdColeman [Tue, 10 May 2022 20:26:40 +0000 (16:26 -0400)]
Remove json prettyPrint from RootGcCandidates (#2681)
- removes multi-line json output from logs
Co-authored-by: Ed Coleman etcoleman <edcoleman@apache.org>
Mike Miller [Tue, 10 May 2022 14:46:57 +0000 (10:46 -0400)]
Rename some confusing GC methods (#2686)
Christopher Tubbs [Mon, 9 May 2022 19:12:22 +0000 (15:12 -0400)]
Run shfmt and shellcheck on scripts (#2677)
* Run shfmt to format scripts
* Run shellcheck to check all scripts for bugs
* Add automated check for shfmt and shellcheck during CI
Christopher Tubbs [Sat, 7 May 2022 00:44:45 +0000 (20:44 -0400)]
Add VFS type mapping for new jar mime type (#2683)
Add VFS type mapping for the new jar mime type added in newer versions
of Java 17, application/java-archive, to ensure it is associated with
the "jar" scheme in VFS. This ensures VFS loads jars correctly when run
on a newer version of Java.
This fixes #2682
See related: https://issues.apache.org/jira/browse/VFS-819
Mike Miller [Fri, 6 May 2022 15:29:39 +0000 (11:29 -0400)]
GC Blip Count Follow up Fixes (#2678)
* Move blip counter to the correct place
* Improve GarbageCollectionTest
Dom G [Fri, 6 May 2022 15:10:14 +0000 (11:10 -0400)]
Bug fix for caption in tservers table in monitor (#2674)
* Patch bug with caption remaining highlighted
Mike Miller [Fri, 6 May 2022 10:33:34 +0000 (06:33 -0400)]
Clean up bulkImport js (#2675)
* Refactor bulkImport.js to be formatted properly and use concise jquery
Co-authored-by: Dom G. <domgarguilo@apache.org>
Emily [Wed, 4 May 2022 22:56:13 +0000 (22:56 +0000)]
Update Replication Table offline message in Monitor (#2676)
Mike Miller [Wed, 4 May 2022 20:12:13 +0000 (16:12 -0400)]
Add new blip count to GC (#2672)
* Update GarbageCollectionTest to count blips
* Split up confirmDeletes into 3 methods
Mike Miller [Tue, 3 May 2022 12:40:33 +0000 (08:40 -0400)]
Create new internal GC class (#2670)
* Pull out the GCEnv inner class of SimpleGarbageCollector into its own
class and call it GCRun.
* Keep the logic of GCEnv but make the stats private numbers to allow
dropping of synchronized block
* Create separate logger classes for each instance of GCRun
* Make GCRun return the stats gathered during that run
* Make SimpleGarbageCollector increment the current stats based on what
is returned in the GCRun class
Dom G [Fri, 29 Apr 2022 20:50:18 +0000 (16:50 -0400)]
Highlight rows with recovering tservers in the tserver status table (#2663)
* highlight recovering tservers
* add caption when there are recovering tservers
Mike Miller [Fri, 29 Apr 2022 20:21:12 +0000 (16:21 -0400)]
Add js strict directive to each js file (#2664)
Mike Miller [Fri, 29 Apr 2022 10:51:51 +0000 (06:51 -0400)]
Upgrade to Bootstrap v3.4.1 and include missing map files (#2662)
* Upgrade to Bootstrap v3.4.1
* Add missing map files to fix console errors
* Update License file
Christopher Tubbs [Fri, 29 Apr 2022 00:56:24 +0000 (20:56 -0400)]
Check for misplaced ITs
Christopher Tubbs [Fri, 29 Apr 2022 00:56:03 +0000 (20:56 -0400)]
Use getCount for COUNT property types
Christopher Tubbs [Fri, 29 Apr 2022 00:44:43 +0000 (20:44 -0400)]
Bump caffeine dependency
Dave Marion [Thu, 28 Apr 2022 20:06:57 +0000 (16:06 -0400)]
Add base objects for Tablet and TabletServer for the upcoming ScanServer feature (#2661)
This commit introduces TabletBase, a base class for Tablets, and TabletHostingServer,
an interface for server components that host Tablets. These changes will be used by
the ScanServer feature that is in the works.
Related to #2411
Co-authored-by: Keith Turner <kturner@apache.org>
Christopher Tubbs [Thu, 28 Apr 2022 17:47:21 +0000 (13:47 -0400)]
Use memoized suppliers to lazy load resources (#2658)
* Remove the need to synchronize on reads for lazily initialized
singleton resources, particularly in the configuration and context
utilities, using Suppliers.memoize
* Apply to DefaultConfiguration.getInstance() to avoid unnecessary
object creation whenever that is called
* Make all ServerContext fields final, and use memoize to lazily load
anything that was previously checking if it was set in a synchronized
getter method
* Use computeIfAbsent in ServerConfigurationFactory, and make its caches
of configuration objects non-static, so they only live as long as the
ServerContext that created it lives; this, along with using
ConcurrentHashMaps, dramatically simplifies this code
* Reduce ServerContext reliance on ServerConfigurationFactory when it
already has the information (notably, it has the instance of
SiteConfiguration it used to construct the ServerConfigurationFactory)
* Add a builder option for SiteConfiguration.empty() for use with tests
* Update related tests, and make ServerContextTest.testCanRun more
robust
Mark Owens [Thu, 28 Apr 2022 17:40:34 +0000 (13:40 -0400)]
Resolve unused format arguments (#2655)
* Resolve unused format arguments
Updated instances where the number of format arguments did not
match number of arguments.
In CompactionCoordinator the compactorAddress was a third argument for
a statement containing only two format arguments. The compactorAddress
was dropped as it did not seem to be necessary.
In AuditedSecurityOperation there were two occasions of mismatched
formats and arguments. In both cases, an additional format argument was
added to allow both the tableName and tableId to be displayed in the
audit log.
Changes to AuditMessage formatting affected the associated IT test as
well. Updated the IT test to work with the previous changes.
Mike Miller [Thu, 28 Apr 2022 14:46:01 +0000 (10:46 -0400)]
Replace more js with jquery in server.js (#2652)
* Replace javascript code with more readable and clearer jquery
* Add static rows and table cells to tserver tables instead of dynamically creating the rows
* Create clearAllTableCells() js function to be reused
Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
Dom G [Thu, 28 Apr 2022 13:05:52 +0000 (09:05 -0400)]
Replace iterator usage (#2646)
* Use streams to simplify iterator usage
* Add more short-circuit empty/non-empty checks
Dom G [Thu, 28 Apr 2022 11:24:11 +0000 (07:24 -0400)]
Address TODOs found in code (#2648)
Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
Dave Marion [Thu, 28 Apr 2022 11:05:28 +0000 (07:05 -0400)]
Modified Shell tracing to use native OpenTelemetry Tracer (#2656)
Removed TraceUtil.enable/disable and modified the Shell to
create a Tracer like a normal application would do when using
Accumulo
Closes #2623
Christopher Tubbs [Thu, 28 Apr 2022 06:30:43 +0000 (02:30 -0400)]
More improvements to new SecurityOperation
* Make all fields private final, and non-static
* Lazily load root username so synchronization isn't needed to read it
after it's loaded, using Suppliers.memoize()
* Limit visibility of methods to private that are intended as internal
Dave Marion [Wed, 27 Apr 2022 20:48:22 +0000 (20:48 +0000)]
Moved AuditedSecurityOperation to ServerContext
Closes #2504
Christopher Tubbs [Wed, 27 Apr 2022 16:55:13 +0000 (12:55 -0400)]
Rely on WADL for dynamic REST endpoint listing (#2654)
This removes the http page that shows a static list of hard-coded REST
endpoints added in #2639 and #2649. Instead, this adds the missing
dependency to ensure the WADL endpoint at /rest/application.wadl is
generated properly, so that developers can get an accurate description
of not only the REST endpoint locations, but also their parameters and
return types. Furthermore, since this is using the WADL standards-based
approach, it will be more useful to web-developer tooling to help them
write web views using these endpoints.
Mark Owens [Wed, 27 Apr 2022 14:36:24 +0000 (10:36 -0400)]
Added missing space in string literal
Added missing space between 'lzo,lz4' in outCompressionOpt option description.
Mark Owens [Wed, 27 Apr 2022 11:06:18 +0000 (07:06 -0400)]
Update ChangeSecret to work as documentation states (#2651)
* Update ChangeSecret to work as documentation states
There are inconsistencies in how ChangeSecret should be run, as well as
exceptions thrown when attempting to execute ChangeSecret.
In two locations, instructions for executing ChangeSecret indicate
./bin/accumulo org.apache.accumulo.server.util.ChangeSecret
But within accumulo-site.xml, instructions indicate
./bin/accumulo org.apache.accumulo.server.util.ChangeSecret [oldpasswd] [newpasswd]
Neither one of the two methods work properly.
Running with no parameters throws a NullPointerException. Running with
the indicated parameters presents a usage message with an error
message provided. Additionally, the usage message makes no mention of
the old and new password parameters.
This PR updates site.xml to match the instructions within the rest of
the code, i.e., execute the command with no parameters.
It also updates the method by which the old and new parameters
are collected. This allowed the `Opts` class to be removed along
with other unneeded code.
The use of `password` is replaced with the string `secret` to
make the intent more clear.
Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
nikita [Tue, 26 Apr 2022 13:35:50 +0000 (06:35 -0700)]
Handle thread interrupt in server client execute loop (#2622)
Re-throw ClosedByInterruptException as UncheckedIOException to
exit out of interrupted retry loops to improve the client side user
experience when interrupting the process with `Ctrl+C`
This fixes #2621
Co-authored-by: Nikita Sirohi <nikita@gh.st>
Mike Miller [Tue, 26 Apr 2022 09:46:10 +0000 (05:46 -0400)]
Fix js in server.js
Mike Miller [Tue, 26 Apr 2022 08:24:10 +0000 (04:24 -0400)]
Replace javascript code with jquery for server (#2650)
Mike Miller [Mon, 25 Apr 2022 17:39:17 +0000 (13:39 -0400)]
Make rest endpoints sorted (#2649)
Dave Marion [Thu, 21 Apr 2022 12:03:15 +0000 (08:03 -0400)]
Separated tablet scan functions from TabletClientService into a new Thrift service (#2643)
related to #2640
Dave Marion [Thu, 21 Apr 2022 11:00:18 +0000 (07:00 -0400)]
Fix NullTServer and ZombieTServer thrift processor setup (#2645)
Christopher Tubbs [Wed, 20 Apr 2022 20:59:02 +0000 (16:59 -0400)]
Update GitHub Actions workflows (#2644)
* Bump to latest versions of actions
* Specify Java distribution explicitly
* Use 'adopt' JDK distro with hotspot
(works with 11 and 17; zulu was flaky with 17)
Dom G [Wed, 20 Apr 2022 17:51:06 +0000 (13:51 -0400)]
Improvements to streams usages (#2638)
* Improvements to streams
* Simplify propertyTypeTest
Mike Miller [Wed, 20 Apr 2022 12:46:20 +0000 (08:46 -0400)]
Add all endpoints page on Monitor (#2639)
* Create a debug page on the monitor and an endpoint to display all
of the REST endpoints available on the monitor
* Also create a variable to store the live port number when the server
starts up
Dave Marion [Tue, 19 Apr 2022 18:25:35 +0000 (14:25 -0400)]
Use Thrift Multiplex components to host multiple services (#2620)
This commit changes all server side Thrift Processors to be
TMultiplexedProcessors and changes all clients to use the
TMultiplexedProtocol to support splitting TabletClientService
into at least two services in a later commit for the ScanServer
feature.
This commit also modifies the Thrift IDL for TabletClientService
such that it no longer extends ClientService and modifies
ManagerClientService such that it no longer extends FateService.
The TabletServer and Manager processes then needed to be
modified to start two Thrift TProcessors instead of one. This
required the use of the TMultiplexedProcessor object on the server
side, which takes one or more Processors and places them in
a map keyed on the Thrift service name. Clients, when
communicating with the TabletServer or Manager, then need to
use a TMultiplexedProtocol with the name of the Thrift service
that they want to call. This server and client setup logic is located
in two new classes: ThriftProcessorTypes and ThriftClientTypes
Mike Miller [Tue, 19 Apr 2022 11:31:11 +0000 (07:31 -0400)]
Add specific timeouts to tests in SslIT (#2631)
Christopher Tubbs [Mon, 18 Apr 2022 17:03:00 +0000 (13:03 -0400)]
Update Apache parent POM, modernizer (#2636)
* Update modernizer and fix new recommendations, primarily replacing
Guava Iterables and making more use of Stream
* Add ConfigurationCopy constructor to accept a Stream
* Add sequential stream() to ScannerBase to easily stream results and
replace `new StreamSupport(scanner.spliterator(), false)` with this
* Fix incorrect imports of dependencies shaded into Hadoop
* Remove no longer needed m2e ignores in the POM
* Also bump sortpom and datasketches and add two new enforcer QA rules
* Update core/src/main/java/org/apache/accumulo/core/conf/ConfigurationCopy.java
Co-authored-by: Dom G. <domgarguilo@apache.org>
* Fix dependency analysis
* Mark test-only dependencies in the test scope
* Move MapReduceClientOpts class to src/test/java - There is an issue
with the dependency analyzer where it doesn't properly detect that
MapReduceClientOpts requires jcommander to compile, even though it
does. It requires it because its parent class, ClientOpts, uses
JCommander annotations. There does not appear to be a way to add an
exception to force the analyzer to ignore this class. However, it
turns out that this class was only ever used it tests, where
JCommander is explicitly required for compilation already, so this
class was moved to the test directory.
* Incidentally, shorten the array element names in the dependency
analyzer configuration, because the names are ignored for
configuration types that are arrays, and because this makes it easier
to read. Also remove an extra stray dash character found after doing
this.
* Fix RowHashIT
Inline MapReduceClientOpts into test where it is used and squash it with
its one and only subclass there, to avoid jar sealing issues.
Co-authored-by: Dom G. <domgarguilo@apache.org>
Dave Marion [Mon, 18 Apr 2022 13:33:17 +0000 (09:33 -0400)]
Modified Initialize to add separator between root table dir and root tablet (#2637)
Closes #2605
Dom G [Fri, 15 Apr 2022 00:55:33 +0000 (20:55 -0400)]
Check for non Junit5 usage in CI (#2630)
* Fix found incorrect assert being used in a test
* Check for both vintage stuff in unapproved classes, as well as jupiter
stuff in the approved vintage classes
Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
Mike Miller [Thu, 14 Apr 2022 15:25:14 +0000 (11:25 -0400)]
Create IteratorBuilder to clean up internals (#2600)
* Create builder classes for iterator stack creation
* Replace IterLoad class with new IteratorBuilder and refactor classes
that use it to determine whether or not there is a bug in iterator code
Christopher Tubbs [Thu, 14 Apr 2022 01:59:41 +0000 (21:59 -0400)]
Fix ThriftTransportPool client idle timeout default (#2629)
* Create new client-side property for ThriftTransportPool's idle
transport timeout. This allows the default value for clients to be
configured independently of the default value for servers.
* Set default value for clients back to 3 seconds, like it was
hard-coded prior to the changes in #2612
* Fix debug logging of the transport pool idle time value
This fixes #2626
AlbertWhitlock [Wed, 13 Apr 2022 19:23:02 +0000 (15:23 -0400)]
Convert Compaction Coordinator module to JUnit5 (#2628)
Christopher Tubbs [Wed, 13 Apr 2022 04:33:53 +0000 (00:33 -0400)]
Merge pull request #2584
Migrate iterator-test-harness module JUnit5
Christopher Tubbs [Wed, 13 Apr 2022 04:06:03 +0000 (00:06 -0400)]
Make improvements to iterator test harness
* Rename base test class, so it's not coupled to JUnit in its API (just
as a dependency)
* Use Streams to provide tuples of input-expectedOutput-testCase
* Make it easy to get the built-in test cases as a stream, so it can be
used to create a stream of tuples
* Remove unnecessary classes and move some utility code into private
methods
* Inline small methods and update the iterator tests that use this
harness prepare their input/expectedOutput in similar ways so they are
easier to maintain
* Use the default parameter naming for the tests, and ensure this
includes the test case name and input in the test name
* Make things package-private if they don't need to be public
domgarguilo [Tue, 22 Mar 2022 20:23:19 +0000 (16:23 -0400)]
Convert iterator test harness module to JUnit5
AlbertWhitlock [Tue, 12 Apr 2022 21:59:59 +0000 (17:59 -0400)]
Convert accumulo-start module to use JUnit5 (#2608)
* Convert all tests to JUnit5 Jupiter engine, except for AccumuloVFSClassLoaderTest
* Leave AccumuloVFSClassLoaderTest using the vintage engine because it uses
PowerMock and cannot be converted until PowerMock works with JUnit5
Christopher Tubbs [Tue, 12 Apr 2022 21:47:55 +0000 (17:47 -0400)]
MiniAccumuloCluster improvements (#2624)
* Use a memoized Supplier to lazily load client properties and server
context and remove now unnecessary synchronization for these
* Make a protective copy of the client properties when returning in the
public API method, getClientProperties(), and update javadoc
* Make fields final where possible
* Relocate constructors to just under the fields, prior to other
methods, for easier class navigation
This fixes #2619
Christopher Tubbs [Tue, 12 Apr 2022 04:45:39 +0000 (00:45 -0400)]
Remove unnecessary uses of ClientInfo in ITs (#2625)
* Remove conversions of client Properties into ClientInfo, only to
convert them back to Properties again; just use Properties directly
* Remove unnecessary ClientInfo parameter from test utilities where
client would suffice (client.properties() was also available)
* Replace uses of clusterHarness.getClientInfo().getProperties() with
just-as-good clusterHarness.getClientProps()
Christopher Tubbs [Tue, 12 Apr 2022 04:23:03 +0000 (00:23 -0400)]
Use CLASSPATH env instead of `-cp` (#2617)
* Replace use of `-cp` for specifying the class path with use of the
CLASSPATH environment variable
* Removing redundant display of extraJvmOpts and classpath from the
debug log message when MiniAccumuloCluster starts (they're already
available in the args and the environment)
* Use Streams to avoid intermediate list objects and to make the
process' argList construction more intuitive
Christopher Tubbs [Tue, 12 Apr 2022 00:36:22 +0000 (20:36 -0400)]
Start spans when inside a valid span (#2611)
Start a span, even if tracing is disabled on our end, if we find our
code inside a valid span. This implies that we're running inside a
user's span, started in user code.
This prevents us from needing to expose Accumulo internal
enabling/disabling utilities to client code. Users can enable
client-side tracing merely by wrapping our code with their own valid
spans.
Christopher Tubbs [Tue, 12 Apr 2022 00:24:02 +0000 (20:24 -0400)]
Prepare for automatic module names
* Prepare the pom.xml to populate the jar manifest with automatic module
names, but don't actually activate it, due to the bug reported in the
maven-javadoc-plugin: https://issues.apache.org/jira/browse/MJAVADOC-707
* This relates to #2498 (helps make that PR more minimal)
Christopher Tubbs [Mon, 11 Apr 2022 23:57:10 +0000 (19:57 -0400)]
Minor cleanup of ThriftTransportPool (#2612)
* Minor cleanup of ThriftTransportPool
* Remove unnecessary memoization of the checker thread
* Inline the Closer Runnable type and make it a lambda
* Use primitive long for `ERROR_THRESHOLD`
* Reorder all the class members before all the inner-classes, and place
the fields above the methods to make it easier to navigate the class
* Make constructor private and use a static method to construct and
start the checker thread after construction, then return the instance
* Make ThriftTransportPool immutable by passing in the maximum age of
idle transports, rather than setting it after it is constructed; this
also changes the default to the general purpose RPC timeout config
value for all servers, rather than just the manager; clients continue
to use the hard-coded 3 seconds, just as before this change
* Changes from code review
* Change the default idle transport max age to be configurable, no
longer hard-coded at 3 seconds, but now uses
`Property.GENERAL_RPC_TIMEOUT` if set in the client configuration,
which has a default of 120. The longer default is acceptable due to
the fact that the client has a close method to forcibly clean up all
transports, if they wish, so there's no urgency to clean them up in
the background for good resource management. This configurability
fixes ACCUMULO-2069
* Use a LongSupplier so the configurable max age can respond to dynamic
configuration changes
* Use the same max age supplier to determine how frequently the checker
thread runs
* Fix ThriftTransportPool checker loop
Ensure the checker loop checks for shutdown frequently, even if the
configured maximum age for idle transports is longer. This ensures it is
responsive to shutdowns, terminating quickly when a shutdown is
detected, but it doesn't waste time checking for expired idle
connections too often if the max age is relatively long.
Christopher Tubbs [Mon, 11 Apr 2022 21:18:50 +0000 (17:18 -0400)]
Add details to ZooKeeper watcher logs (#2615)
Add more details to the log messages produced while processing ZooKeeper
WatchedEvents. Also, try to standardize some messages by adding the
event at the end, after whatever core message is conveyed. Most of the
time, we won't see these messages, but it definitely helps to have
greater insight into what event triggered the situation, rather than
picking out and printing only certain details from the WatchedEvent.
Dave Marion [Wed, 6 Apr 2022 14:50:24 +0000 (10:50 -0400)]
Refactor ExternalCompactionMetricsIT to reduce probability of timeout (#2610)
Closes #2406
Mark Owens [Tue, 5 Apr 2022 06:59:15 +0000 (02:59 -0400)]
Use returned values (#2558)
ErrorProne insists that return values be checked/used or specifically
acknowledged as unused. The two error patterns are:
* [CheckReturnValue](https://errorprone.info/bugpattern/CheckReturnValue)
* [ReturnValueIgnored](https://errorprone.info/bugpattern/ReturnValueIgnored)
To remove unused values:
* Log the loaded class name in CounterSummary at trace level
* Move precomputeHashCode method into ThriftTransportKey's constructor
* Assigned return value to checkThreadFactory.get() call within
startCheckerThread.
* Wrapped Iterators.get call within CloseScannerIT with assertNotNull to
remove CheckReturnVal/ReturnValueIgnored error when using ErrorProne
analysis.
* Use assertThrows in tests where return values were ignored because exceptions
were expected
* Avoid use of Iterators.get and Iterators.size in tests, using `forEach(e -> {})` pattern
for scanner iterations where we don't care about the actual data
Also:
* Add errorprone profile to GitHub Actions pull request build checks
Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
Christopher Tubbs [Mon, 4 Apr 2022 17:12:28 +0000 (13:12 -0400)]
Bump surefire and failsafe plugins to 3.0.0-M6 (#2607)
* Bypass buggy 3.0.0-M5 version; this fixes #2555
Keith Turner [Mon, 4 Apr 2022 15:56:40 +0000 (11:56 -0400)]
fixes #2567 corrects race condition in tablet metadata verification (#2574)
Mike Miller [Mon, 4 Apr 2022 12:38:04 +0000 (08:38 -0400)]
Rename new property for better sorting (#2604)
Dave Marion [Fri, 1 Apr 2022 16:21:07 +0000 (12:21 -0400)]
Modified accumulo-cluster to pass -a argument to server processes (#2603)
Closes #2602
Mike Miller [Thu, 31 Mar 2022 15:10:51 +0000 (11:10 -0400)]
Add new property tserver.max.scan.result.timeout (#2599)
Dom G [Wed, 30 Mar 2022 15:21:19 +0000 (11:21 -0400)]
Convert defaultTimeoutSeconds() to defaultTimeout() (#2580)
* Make timeouts in AccumuloITBase use Duration instead of int
Dave Marion [Tue, 29 Mar 2022 20:07:37 +0000 (16:07 -0400)]
Use ServerContext.getScheduledExecutor where appropriate (#2593)
Reviewed the direct usage of ThreadPools.createGeneralScheduledExecutorService
and compared it to the usage of Timer and SimpleTimer in version 2.0.1. Found
and fixed a case in SessionManager where a new SheduledThreadPool was being created
in each SessionManager but a shared instance of SimpleTimer was being used in
the previous version.
Mike Miller [Tue, 29 Mar 2022 15:32:06 +0000 (11:32 -0400)]
Fix ZooZap (#2597)
Dave Marion [Tue, 29 Mar 2022 11:03:40 +0000 (07:03 -0400)]
Bump Micrometer from 1.8.3 to 1.8.4 (#2596)
Dave Marion [Tue, 29 Mar 2022 11:03:05 +0000 (07:03 -0400)]
Bump OpenTelemetry version from 1.11 to 1.12 (#2595)
Diego Marcilio [Mon, 28 Mar 2022 13:28:48 +0000 (15:28 +0200)]
Add missing javadoc for exceptions thrown on invalid arguments (#2594)
* Fixing wrong javadoc for null outputStream in SerializationUtil.serialize()
* Adding a couple @throws javadocs and tests for invalid arguments
Mike Miller [Fri, 25 Mar 2022 17:24:35 +0000 (13:24 -0400)]
Make ZooZap keyword executable (#2591)
Dave Marion [Fri, 25 Mar 2022 17:05:13 +0000 (13:05 -0400)]
Change location where Thread.start() is called to resolve ErrorProne warning (#2590)
Mike Miller [Fri, 25 Mar 2022 11:35:47 +0000 (07:35 -0400)]
Use new ThreadPools method to clean up code (#2589)
Mike Miller [Thu, 24 Mar 2022 14:05:44 +0000 (10:05 -0400)]
Add trace and debug log to consistency check (#2583)
* Closes #2577
* Add trace span and time measurement around consistency check
so we get an idea of how long metadata scans are taking
* Create new property tserver.health.check.interval to make it configurable
* Create new method watchCriticalFixedDelay() in ThreadPools
Co-authored-by: Keith Turner <kturner@apache.org>
Dom G [Thu, 24 Mar 2022 13:46:25 +0000 (09:46 -0400)]
Convert monitor module tests to JUnit5 (#2588)
Dom G [Thu, 24 Mar 2022 13:30:08 +0000 (09:30 -0400)]
Convert AccumuloUncaughtExceptionHandlerTest to JUnit5 (#2585)