accumulo.git
18 months ago[maven-release-plugin] prepare release rel/1.10.1 rel/1.10.1
Christopher Tubbs [Thu, 17 Dec 2020 22:06:50 +0000 (17:06 -0500)] 
[maven-release-plugin] prepare release rel/1.10.1

18 months agoFix #1792 flaky/broken GcMetricsIT
Christopher Tubbs [Thu, 17 Dec 2020 03:29:17 +0000 (22:29 -0500)] 
Fix #1792 flaky/broken GcMetricsIT

Rewrite portions of GcMetricsIT to improve its reliability and to add
comments and rename methods to make it more clear how it is intended to
work, so it's easier to debug in future.

Also use daemon threads for the tailer, and ensure it is started
consistently (using a dedicated method) and closed properly when done.

Remove unnecessary catching of exceptions, and allow them to be thrown
from the tests instead, because they will be handled by the test
framework.

18 months agoAdd additional permission checks (#1832)
Jeffrey Manno [Wed, 16 Dec 2020 03:27:06 +0000 (22:27 -0500)] 
Add additional permission checks (#1832)

* Update AuditMessageIT to test auditing of flush commands
* Add new ManagerApiIT to test permissions checks of
  MasterClientServiceHandler RPC endpoints
* Fill in missing checks for SYSTEM permissions in PermissionsIT
  (and add flush tests for WRITE and ALTER_TABLE permissions)

Co-authored-by: Dave Marion <dlmarion@apache.org>
Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
18 months agoBackport JUnit upgrade from #1562 to 1.10.1
Christopher Tubbs [Tue, 15 Dec 2020 18:25:08 +0000 (13:25 -0500)] 
Backport JUnit upgrade from #1562 to 1.10.1

Update JUnit dependency, and use assertThrows instead of
ExpectedException Rule and hamcrest library, to avoid deprecation
warnings and an unnecessary test dependency.

18 months agoThrow sercurity exceptions when permissions checks fail. Backport to 1.10 (#1830)
Jeffrey Manno [Wed, 9 Dec 2020 15:26:20 +0000 (10:26 -0500)] 
Throw sercurity exceptions when permissions checks fail. Backport to 1.10 (#1830)

* backport sercurity exceptions to 1.10

* fix AuditMessageIT

18 months agoImprove log message in Delete CleanUp FATE. (#1829)
Mike Miller [Tue, 8 Dec 2020 19:39:17 +0000 (14:39 -0500)] 
Improve log message in Delete CleanUp FATE. (#1829)

Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
19 months agoBackport some GcMetricsIT test stability to 1.10
Christopher Tubbs [Thu, 26 Nov 2020 18:04:01 +0000 (13:04 -0500)] 
Backport some GcMetricsIT test stability to 1.10

Backport some test stability fixes to GcMetricsIT implemented in #1745
to Accumulo 1.10.1

19 months agoMinor fixup of build
Christopher Tubbs [Thu, 26 Nov 2020 17:58:28 +0000 (12:58 -0500)] 
Minor fixup of build

* Show rat output in console (partial backport to 1.x from #1767)
* Don't skip enforcer when skipQA profile is active, because it is
  necessary for enforcing build requirements (and isn't costly to run)
* Enforce the JDK version to less than 15 for Accumulo 1.x, because
  Concurrent Mark and Sweep garbage collector options were removed from
  JDK 15, which Accumulo 1.x uses in its scripts and in Mini. While
  Accumulo 1.x can still compile and build in JDK 15, it will not be
  able to run as-is, including running tests, resulting in hard-to-track
  and confusing errors for developers. It is easier to enforce the JDK
  version to being at least 11, and strictly less than 15, during
  development.

19 months agoMake skipQA profile more human-proof (#1807)
Christopher Tubbs [Wed, 25 Nov 2020 16:56:50 +0000 (11:56 -0500)] 
Make skipQA profile more human-proof (#1807)

Support activation of skipQA profile using either of two options:
1. `-PskipQA` (existing option) OR
2. `-DskipQA` (new option in this patch)

20 months agoBack port #1754 to 1.10
Mike Miller [Wed, 28 Oct 2020 18:26:12 +0000 (14:26 -0400)] 
Back port #1754 to 1.10

* Update jvm option for JDWP in Mini

20 months agoConfigure surefire/failsafe forkCount separately (#1756)
Christopher Tubbs [Wed, 28 Oct 2020 16:55:22 +0000 (12:55 -0400)] 
Configure surefire/failsafe forkCount separately (#1756)

* Create new configuration properties in the build configuration for
  surefire.forkCount and failsafe.forkCount to enable configuring them
  independently of one another
* Create a profile to set both surefire.forkCount and failsafe.forkCount
  together, if desired
* Remove forkCount settings from the build workflow, instead selecting
  sane defaults that are equivalent
* Force minicluter to use one surefire fork only (along with start
  module) to avoid potential conflicts

20 months agoAdd libraries in tool.sh script for 1.10 examples (#1737)
Mark Owens [Thu, 15 Oct 2020 16:01:04 +0000 (12:01 -0400)] 
Add libraries in tool.sh script for 1.10 examples (#1737)

When attempting to run the 1.10.x Accumulo examples with Hadoop version 3.x, there were several NoClassDefFoundError exceptions thrown when using the tool.sh script to run MR jobs. This PR updates the tool.sh script to add the necessary jar files to the lib path allowing the MR jobs to run successfully. The additions work with both Hadoop 2.x and Hadoop 3.x.

20 months agoUse mktemp arg that works on on alpine-based distros (#1734)
larsw [Wed, 14 Oct 2020 18:42:54 +0000 (20:42 +0200)] 
Use mktemp arg that works on on alpine-based distros (#1734)

* Fix build_native_library.sh so that it runs on alpine-based (e.g. Alpine)-based distros
* Fixes #1731

20 months agoUpdate README.combiner
Mark Owens [Wed, 14 Oct 2020 17:44:28 +0000 (13:44 -0400)] 
Update README.combiner

Update readme to display missing output line from call to setiter.

20 months agoUpdate regex expression in tool.sh (#1729)
Mark Owens [Fri, 9 Oct 2020 20:36:48 +0000 (16:36 -0400)] 
Update regex expression in tool.sh (#1729)

* Update tool.sh for 1.x to modify the regex to successfully determine the zookeeper version

20 months agoAdd missing script for creatin the IT matrix
Christopher Tubbs [Fri, 9 Oct 2020 20:19:31 +0000 (16:19 -0400)] 
Add missing script for creatin the IT matrix

20 months agoUpdate CI jobs to build PRs and see failure logs
Christopher Tubbs [Fri, 9 Oct 2020 20:10:51 +0000 (16:10 -0400)] 
Update CI jobs to build PRs and see failure logs

* Add `if: failure()` to upload-artifact tasks so we get the logs for
  failed tasks
* Allow input for the ref to build for manually triggered jobs (so we
  can build PRs)
* Show the first git log message to ensure we can verify upon inspection
  that the chosen ref was built

20 months agoUpdate release candidate script (#1724)
Christopher Tubbs [Wed, 7 Oct 2020 17:18:21 +0000 (13:18 -0400)] 
Update release candidate script (#1724)

Improving generated output:
* show link to staging repo website
* show full commits
* show reminders of which buttons to click on the staging site
* show reminder to replace `$origin` with actual remote name
* place `-s` in `git tag` command with the other options
* update draft release notes link to point to staged site

Optimizing performance:
* create the `-next` branch and switch to it in one command
* skip tests during release prep because the release can (and should) be
  tested outside of the actual release candidate creation process and
  this speeds things up significantly
* change release prep goal to only do `package`, because it's not
  necessary to run `verify` during `release:prepare`; all the QA checks
  done at `verify` can be done before/after release candidate staging

Being more interactive:
* prompt for gpg key to use
* prompt for the `remote` to push to (do not assume `origin`)

Removing junk:
* remove check for `gpg2` (modern Linux uses version 2 for the `gpg` binary)
* remove unnecessary `--test` option and related `runOrFail` function
* remove `extraReleaseArguments` to simplify things

21 months agoUpdate run-verify.sh to not fail when optional arguments passed are empty (#1717)
Karthick Narendran [Tue, 29 Sep 2020 20:09:54 +0000 (21:09 +0100)] 
Update run-verify.sh to not fail when optional arguments passed are empty (#1717)

* Use Array variable as suggested by Christopher

21 months agoUpdate tool.sh to set correct ZOOKEEPER_CMD. (#1716)
Mark Owens [Mon, 28 Sep 2020 19:44:13 +0000 (15:44 -0400)] 
Update tool.sh to set correct ZOOKEEPER_CMD. (#1716)

ZooKeeper changed directory structure between versions 3.4 and 3.5. This
update sets the correct ZooKeeper depending upon version used.

* Updated tool.sh to set correct ZOOKEEPER_CMD.

* Added  suggested change to guarantee ZOOKEEPER_VERSION is defined. Also update to use regex pattern checking for version.

Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
22 months agoMerge branch '1.10.0-rc2-next' into 1.10
Christopher Tubbs [Thu, 3 Sep 2020 02:26:56 +0000 (22:26 -0400)] 
Merge branch '1.10.0-rc2-next' into 1.10

22 months agoUpdate formatter plugin version
Christopher Tubbs [Tue, 1 Sep 2020 17:45:54 +0000 (13:45 -0400)] 
Update formatter plugin version

22 months ago[maven-release-plugin] prepare for next development iteration
Mike Miller [Thu, 27 Aug 2020 15:56:15 +0000 (11:56 -0400)] 
[maven-release-plugin] prepare for next development iteration

22 months ago[maven-release-plugin] prepare release rel/1.10.0 rel/1.10.0
Mike Miller [Thu, 27 Aug 2020 15:56:15 +0000 (11:56 -0400)] 
[maven-release-plugin] prepare release rel/1.10.0

22 months agoFix file permissions in source assembly tarball (#1692)
Christopher Tubbs [Thu, 27 Aug 2020 12:18:02 +0000 (08:18 -0400)] 
Fix file permissions in source assembly tarball (#1692)

Fix permissions mangling caused by the latest versions of the assembly
plugin.

Use maven-assembly-plugin-3.1.1 because of MASSEMBLY-941
https://issues.apache.org/jira/browse/MASSEMBLY-941

22 months agoUpdate powermock to 2.0.7 (#1691)
Mike Miller [Mon, 24 Aug 2020 18:48:39 +0000 (14:48 -0400)] 
Update powermock to 2.0.7 (#1691)

22 months agoUpdate bouncycastle version to 1.66 (#1690)
Mike Miller [Mon, 24 Aug 2020 18:48:15 +0000 (14:48 -0400)] 
Update bouncycastle version to 1.66 (#1690)

22 months agoTune timeouts for some flakey tests
Christopher Tubbs [Fri, 21 Aug 2020 15:20:44 +0000 (11:20 -0400)] 
Tune timeouts for some flakey tests

Increase default timeouts for some tests which routinely time out during
CI testing on Jenkins.

22 months agoFix issues in prep for 1.10 release (#1688)
Christopher Tubbs [Thu, 20 Aug 2020 19:47:19 +0000 (15:47 -0400)] 
Fix issues in prep for 1.10 release (#1688)

* Bump snapshot versions to 1.10.0-SNAPSHOT
* Converge the Hadoop 2 and Hadoop 3 assemblies, so only one is
  necessary (tested both on Hadoop 2.6.5, 3.0.3, and 3.3.0 using
  fluo-uno)
* Bumped plugin versions and added timestamp flag for increased build
  reproducibility; timestamp should automatically update during release
  process; verified the built jars differ only in the contents of the
  generated META-INF/DEPENDENCIES file when building with the Hadoop2
  vs. the Hadoop3 profiles, and since this file doesn't matter, they are
  now provably equivalent builds
* Fixed some Hadoop profile issues, and updated Hadoop versions to
  latest patch version in the minor release lines we had previously been
  building with as our minimum supported versions
* Update Guava-related code to work with both Hadoop 2 and 3;
  suppress warnings or migrate code to avoid code paths that won't work
  across Guava versions, for fewer class path surprises related to
  Guava
* Update asciidoc plugin, and fix various doc build issues warned about
  by the plugin
* Use last commons-io that doesn't break vfs2
* Make Guava version configurable at build time, so it's easier to build a
  package that contains a version of guava suitable for Hadoop 3.1 and later.

22 months agoFix warnings from #1687
Christopher Tubbs [Mon, 17 Aug 2020 16:40:40 +0000 (12:40 -0400)] 
Fix warnings from #1687

22 months agoUpdate commons-* libraries (#1687)
Mike Miller [Mon, 17 Aug 2020 14:40:15 +0000 (10:40 -0400)] 
Update commons-* libraries (#1687)

* Update commons-cli to 1.4
* Update commons-codec to 1.14
* Update commons-io to 2.7
* Update commons-lang to 2.6
* Update commons-logging to 1.2

22 months agoFix typo in workflow file
Christopher Tubbs [Sun, 16 Aug 2020 19:11:56 +0000 (15:11 -0400)] 
Fix typo in workflow file

22 months agoAdd workflow for triggering a custom build
Christopher Tubbs [Sun, 16 Aug 2020 19:05:09 +0000 (15:05 -0400)] 
Add workflow for triggering a custom build

Add a `workflow_dispatcher` action that can be triggered manually to run
the full ITs, a single test, or anything in-between, on demand.

22 months agoFix JCommander warnings from #1684
Christopher Tubbs [Sat, 15 Aug 2020 20:56:00 +0000 (16:56 -0400)] 
Fix JCommander warnings from #1684

22 months agoUpdate jcommander to 1.78 (#1684)
Mike Miller [Fri, 14 Aug 2020 19:24:07 +0000 (15:24 -0400)] 
Update jcommander to 1.78 (#1684)

22 months agoUpdate gson to 2.8.6 (#1686)
Mike Miller [Fri, 14 Aug 2020 18:39:55 +0000 (14:39 -0400)] 
Update gson to 2.8.6 (#1686)

22 months agoSupport IteratorEnvironment in test harness (#1681)
Christopher Tubbs [Fri, 14 Aug 2020 00:57:35 +0000 (20:57 -0400)] 
Support IteratorEnvironment in test harness (#1681)

* Adds the ability to provide a custom IteratorEnvironment to the
  iterator-test-harness
* This fixes a test failure in SummingCombinerTest caused by recent
  changes to Combiner in #1680

22 months agoModify Combiner init method to honor all options. Fixes #1678 (#1680)
Mike Miller [Thu, 13 Aug 2020 12:06:47 +0000 (08:06 -0400)] 
Modify Combiner init method to honor all options. Fixes #1678 (#1680)

22 months agoMove MinorCompaction method call into retry catch block. Fixes #1298 (#1675)
Mike Miller [Wed, 12 Aug 2020 12:30:40 +0000 (08:30 -0400)] 
Move MinorCompaction method call into retry catch block. Fixes #1298 (#1675)

* Fixes the situation when MinorCompactionTask creates a
new File and an IOException leaves the Tablet in a bad state
* Moves the new file code into the try/catch block that will
retry during an IOException

22 months agoIgnore some warnings in CI workflow (#1671)
Christopher Tubbs [Fri, 7 Aug 2020 08:40:20 +0000 (04:40 -0400)] 
Ignore some warnings in CI workflow (#1671)

Ignore warnings produced by upload-artifact step if no files are
available to upload. Not all jobs in the matrix build are expected to
produce artifacts, so this isn't a problem worth warning about.

22 months agoPublish test results in CI workflow (#1671)
Christopher Tubbs [Fri, 7 Aug 2020 08:30:12 +0000 (04:30 -0400)] 
Publish test results in CI workflow (#1671)

22 months agoFix javadoc html header elements on newer JDKs
Christopher Tubbs [Wed, 5 Aug 2020 23:06:33 +0000 (19:06 -0400)] 
Fix javadoc html header elements on newer JDKs

As of JDK 13, the javadoc tool's linter requires user headers in
javadocs to start at h2, and checks that h1 appears before h2 in
generated HTML. The side-effect of this check is that any raw (not
generated by javadoc) doc-files html needs to start at h1, before using
h2.

22 months agoImprove CI workflow (#1671)
Christopher Tubbs [Wed, 5 Aug 2020 18:53:22 +0000 (14:53 -0400)] 
Improve CI workflow (#1671)

* Add timeout
* Make matrix similar across branches
* Better naming of matrix jobs
* Better branch pattern matching
* Exclude accumulo artifacts from cache
* Suppress transfer progress when downloading artifacts

22 months agoUse GitHub Actions for CI testing (#1671)
Christopher Tubbs [Wed, 5 Aug 2020 16:22:01 +0000 (12:22 -0400)] 
Use GitHub Actions for CI testing (#1671)

Add GitHub Actions for CI and remove Travis CI configuration, for the
1.9/1.10 maintenance branch.

22 months agoFix #1663 Missing metrics files running CI tests (#1667)
EdColeman [Tue, 4 Aug 2020 20:41:58 +0000 (16:41 -0400)] 
Fix #1663 Missing metrics files running CI tests (#1667)

- excluded hadoop-metrics2-accumulo.properties from jar plugin
- added gc metrics config sample to template

Co-authored-by: Ed Coleman <edcoleman.apache.org>
Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
23 months agoRemove obsolete workaround for Eclipse bug
Christopher Tubbs [Thu, 16 Jul 2020 18:22:49 +0000 (14:22 -0400)] 
Remove obsolete workaround for Eclipse bug

Remove unneeded Maven profile for runtime dependency resolution in the
compile scope while using Eclipse/M2E. Newer versions of Eclipse have
fixed the bug.

Related to #1652 (but for the 1.9 branch).

23 months agoClose unclosed Scanners
Christopher Tubbs [Thu, 16 Jul 2020 06:11:52 +0000 (02:11 -0400)] 
Close unclosed Scanners

Close unclosed scanners detected by Eclipse IDE

23 months agoRename yield method (future restricted identifier)
Christopher Tubbs [Thu, 16 Jul 2020 04:58:16 +0000 (00:58 -0400)] 
Rename yield method (future restricted identifier)

(Backport of c5ecd474dbed016cc6653870fedcce863550a44f)

Rename a private yield method in SourceSwitchingIteratorTest to avoid
conflicting with future versions of Java. In future Java releases,
'yield' is a restricted keyword.

Verified failure (and fix) using the following on OpenJDK13:

```
mvn clean package -DskipTests -Dmaven.compiler.failOnWarning=true
```

23 months agoMerge pull request #1646 from ivakegg/ACCUMULO-1644
Ivan Bella [Tue, 7 Jul 2020 12:22:58 +0000 (08:22 -0400)] 
Merge pull request #1646 from ivakegg/ACCUMULO-1644

fixes #1644: minor compaction retry

2 years agofixes #1644: minor compaction retry 1646/head
Ivan Bella [Tue, 30 Jun 2020 18:33:22 +0000 (18:33 +0000)] 
fixes #1644: minor compaction retry
* Updated the minor compaction to gracefully handle temporary VFS class loader issues while minor compacting

2 years agoUpdate spotbugs-maven-plugin (for JDK14)
Christopher Tubbs [Tue, 21 Apr 2020 20:35:05 +0000 (16:35 -0400)] 
Update spotbugs-maven-plugin (for JDK14)

2 years agoUpdate maven-enforcer-plugin (for JDK14)
Christopher Tubbs [Tue, 21 Apr 2020 20:32:04 +0000 (16:32 -0400)] 
Update maven-enforcer-plugin (for JDK14)

2 years agoFix idempotency bug in importtable (#1555)
Arvind Shyamsundar [Thu, 12 Mar 2020 18:02:45 +0000 (11:02 -0700)] 
Fix idempotency bug in importtable (#1555)

* Fix idempotency bug in importtable
The previous implementation would fail when an interrupted importtable
operation was retried by FATE, as it would not find (already moved)
files in the exported directory. Additionally, this commit also removes
a chatty fs.exists() check.

* Use Set operations to streamline code
This code uses set operations to validate if there are missing files in
the exported (source) directory. In addition the usage of sets improves
perf compared to the lambda scanning the file status arrays.

* Fix formatting and use inbuilt Java joiner

2 years agoUpdate copyright dates for 2020 releases
Christopher Tubbs [Sun, 8 Mar 2020 14:11:05 +0000 (10:11 -0400)] 
Update copyright dates for 2020 releases

2 years agoFix unused variable warning from #1536
Christopher Tubbs [Fri, 6 Mar 2020 23:57:32 +0000 (18:57 -0500)] 
Fix unused variable warning from #1536

2 years agoImprove master metrics rported (#1536)
EdColeman [Fri, 6 Mar 2020 20:16:32 +0000 (15:16 -0500)] 
Improve master metrics rported (#1536)

Adds FATE transaction state and op types reported via hadoop metrics2
Improves metrics tests
Add IT to verify that enabling legacy metrics does not crash master
Added ZooReaderWriter getInstance() for testing.

2 years agoFixes #1525 - remove lines added in b67d3f9 in VolumeUtil (#1539)
Jeffrey Zeiberg [Tue, 3 Mar 2020 16:58:41 +0000 (11:58 -0500)] 
Fixes #1525 - remove lines added in b67d3f9 in VolumeUtil (#1539)

* Revert optimization that broke special tablet directory volume code
that is still needed in 1.X

2 years agoEnable logs for accumulo-maven-plugin's IT
Christopher Tubbs [Thu, 27 Feb 2020 19:42:22 +0000 (14:42 -0500)] 
Enable logs for accumulo-maven-plugin's IT

2 years agoFix #1532 Remove need for Ant on class path
Christopher Tubbs [Thu, 27 Feb 2020 19:32:32 +0000 (14:32 -0500)] 
Fix #1532 Remove need for Ant on class path

Make MetricsConfiguration use a MapConfiguration with System.getenv()
rather than EnvironmentConfiguration, to get the environment properties.
EnvironmentConfiguration uses Ant code, which is not available on the
class path (typically) to get the environment, so it should be avoided.

2 years agoRe #1492 Disable unreliable replication tests (#1524)
Christopher Tubbs [Mon, 24 Feb 2020 21:55:19 +0000 (16:55 -0500)] 
Re #1492 Disable unreliable replication tests (#1524)

Fixes #1226 Disables frequently failing CyclicReplicationIT

2 years agoDecrease recovery cache time from 45s to 15s (#1526)
Mike Miller [Thu, 20 Feb 2020 14:40:48 +0000 (09:40 -0500)] 
Decrease recovery cache time from 45s to 15s (#1526)

* This should help timing of ITs which deal with recovery but will
still prevent too many exist calls to NN in production

2 years agoFix #1520 Limit UnsynchronizedBuffer max size (#1523)
Christopher Tubbs [Wed, 19 Feb 2020 23:25:44 +0000 (18:25 -0500)] 
Fix #1520 Limit UnsynchronizedBuffer max size (#1523)

Limit UnsynchronizedBuffer maximum size to Integer.MAX_VALUE - 8, the
same limit as ArrayList, because of some JVM behavior being unable to
allocate arrays with Integer.MAX_VALUE.

2 years agoMake Accumulo 1.9 compatible with ZK 3.5 (#1506)
Karthick Narendran [Wed, 12 Feb 2020 17:43:39 +0000 (12:43 -0500)] 
Make Accumulo 1.9 compatible with ZK 3.5 (#1506)

2 years agoOptimize Tablet volume replacement (#1505)
Mike Miller [Wed, 12 Feb 2020 13:28:02 +0000 (08:28 -0500)] 
Optimize Tablet volume replacement (#1505)

* Stop volume replacements from checking every file when not used

2 years agoClean-up / remove comments in code from gc_metrics_backport when merged. (#1499)
EdColeman [Tue, 4 Feb 2020 19:40:17 +0000 (14:40 -0500)] 
Clean-up / remove comments in code from gc_metrics_backport when merged. (#1499)

2 years agoBackport gc metrics to 1.9.x to use hadoop metrics2. (#1406)
EdColeman [Tue, 4 Feb 2020 17:53:21 +0000 (12:53 -0500)] 
Backport gc metrics to 1.9.x to use hadoop metrics2. (#1406)

* Backport gc metrics to 1.9.x to use hadoop metrics2.

This backport bring the gc metrics from master / 2.0 into the 1.9.x branch. The metrics
can be enabled / disabled with GC_METRICS_ENABLED property (default false for 1.9.x)  The
metrics changes include:

- Publish current gc metrics to hadoop 2 metrics reporting system.
- Add gc run counter that increments on each gc cycle run
- Add metric to track time required for gc post op (compact, flush, none)

Also Includes:

- Move IT resource file for PR #1381 to correct directory so test passes
in CI and on the command-line. Previously, it may have only worked in
some IDEs, because of the way those IDEs put test resources on the class
path.

* checkpoint: warning clean-up

* Code clean-up and prep for review

one open todo - check if metadata consistency check can be added as a metric.

* defer master check reproting

Co-authored-by: Christopher Tubbs <ctubbsii@apache.org>
2 years agoFix #1482 Increase timeouts on some slow ITs
Christopher Tubbs [Fri, 31 Jan 2020 12:47:26 +0000 (07:47 -0500)] 
Fix #1482 Increase timeouts on some slow ITs

Stabilize some ITs by increasing timeouts to values which cause them to
pass more frequently on modest developer hardware and adding a timeout
to one that was missing it.

Also backport the removal of some flakey performance-related tests,
which were already removed in newer branches.

2 years agoFix #1491 Stabilize ChaoticBalancerIT in 1.x
Christopher Tubbs [Fri, 31 Jan 2020 10:32:29 +0000 (05:32 -0500)] 
Fix #1491 Stabilize ChaoticBalancerIT in 1.x

Backport fixes from ACCUMULO-4602
(520d23d85dfce9297bab887726322861163a1fb2)

2 years agoFix unused var and generics warning
Christopher Tubbs [Thu, 30 Jan 2020 18:19:11 +0000 (13:19 -0500)] 
Fix unused var and generics warning

2 years agofixes #1474: Option to leave cloned tables offline (#1475)
Ivan Bella [Tue, 28 Jan 2020 18:59:09 +0000 (13:59 -0500)] 
fixes #1474: Option to leave cloned tables offline (#1475)

* Added an optional keepOffline option to TableOperations.clone via a CloneConfiguration
* Updated the shell clone command with a -o option
* Updated the CloneTestIT to test the keepOffline option
* Updated the ShellServerIT to test the -o option

2 years agoMake ITs override defaultTimeoutSeconds (#1483)
Mike Miller [Mon, 27 Jan 2020 21:27:18 +0000 (16:27 -0500)] 
Make ITs override defaultTimeoutSeconds (#1483)

* Some ITs were setting timeout using junit, instead of the miniaccumulo
method defaultTimeoutSeconds which allows use of system prop timeout.factor
* Decrease timeout of DeleteRowsIT

2 years agoRequire x tservers on start (#1158)
EdColeman [Mon, 27 Jan 2020 19:22:06 +0000 (14:22 -0500)] 
Require x tservers on start (#1158)

* Adding tserver availability blocking on startup

Check for tserver availability when master gets lock and when configured, block until threshold tserver count reached, or wait time-out expires.  Adds two properties:

MASTER_STARTUP_TSERVER_AVAIL_MIN_COUNT - the number of desired tservers
MASTER_STARTUP_TSERVER_AVAIL_MAX_WAIT - the max time willing to wait

2 years agoIncrease timeout for GarbageCollectWALIT to allow recovery
Mike Miller [Fri, 17 Jan 2020 20:14:03 +0000 (15:14 -0500)] 
Increase timeout for GarbageCollectWALIT to allow recovery

2 years agoAdd color back to travis CI console
Christopher Tubbs [Thu, 23 Jan 2020 00:12:03 +0000 (19:12 -0500)] 
Add color back to travis CI console

2 years agoValidate ClientOpts instance name and Zookeeper paths (#1478)
Arvind Shyamsundar [Tue, 21 Jan 2020 16:48:24 +0000 (08:48 -0800)] 
Validate ClientOpts instance name and Zookeeper paths (#1478)

* Only override instance name and ZK paths if those were specified by caller.

2 years agoUpdate to Java 1.8 and related fixes (#1467)
Mike Miller [Wed, 15 Jan 2020 19:14:53 +0000 (14:14 -0500)] 
Update to Java 1.8 and related fixes (#1467)

* Update to maven compiler source and target to Java 1.8
* Add optional dependency to pom for modernizer annotations
* Add SuppressModernizer annotation to uses of Guava Predicate in
AccumuloConfiguration and its subclasses, Optional in VolumeChooser,
and Function in GroupBalancer
* Replace Guava Function, Predicate, Preconditions, Optional and
Iterators where able with java.util equivalents
* Replace core.util.Base64 class with java.util equivalent
* Fix warnings introduced by Java 1.8
* Drop method in MockConfiguration causing compiler warning and made
PropertyFilter in AccumuloConfiguration extend guava Predicate
* Rename execute methods in MasterClient and ServerClient, like the 2.0
changes to fix the overload warnings

Co-Authored-By: Christopher Tubbs <ctubbsii@apache.org>
2 years agofixes #1456 by adding rate limiter to shutdown queue (#1457)
alerman [Thu, 2 Jan 2020 21:05:18 +0000 (16:05 -0500)] 
fixes #1456 by adding rate limiter to shutdown queue (#1457)

2 years agoTemporarily cache existence of recovery wals (#1462)
Keith Turner [Thu, 2 Jan 2020 20:04:17 +0000 (15:04 -0500)] 
Temporarily cache existence of recovery wals (#1462)

For the case where a lot of tablet servers died, the master was
frequently checking for the existence of a recovery log for each tablet.
Its very likely that many tablets point to the same recovery logs and
the existence checks are redundant.  This patch caches the result of
existence checks for a short period.

2 years agoTurn off trim stacktrace on failsafe plugin (#1452)
Mike Miller [Tue, 10 Dec 2019 20:26:19 +0000 (15:26 -0500)] 
Turn off trim stacktrace on failsafe plugin (#1452)

2 years agoBackport license and typo fixes for bloomfilter code
Christopher Tubbs [Sat, 16 Nov 2019 03:23:07 +0000 (22:23 -0500)] 
Backport license and typo fixes for bloomfilter code

* Remove incorrect redundant license headers from bloomfilter code
* Add exceptions for rat (because it doesn't recognize the style of the
  BSD license header for the bloomfilter code)
* Backport any spelling typos fixed in bloomfilter code in master branch
  back to 1.9

2 years agoFix #1308 - Refactor fate concurrency IT (#1414)
EdColeman [Thu, 7 Nov 2019 22:30:32 +0000 (17:30 -0500)] 
Fix #1308 - Refactor fate concurrency IT (#1414)

fix error that caused test failure.

2 years agoFix #1401, trace servlet parameter handling (#1409)
Richard W. Eggert II [Wed, 6 Nov 2019 05:16:40 +0000 (00:16 -0500)] 
Fix #1401, trace servlet parameter handling (#1409)

* corrected servlet parameter handling so that it does not break things

This resolves #1401.

I removed the previous implementation of  string sanitization, replacing it with more specific checks at the places where the parameters are used.

BasicTest was deleted because it was exclusively testing the broken
sanitization mechanism.

* Update server/monitor/src/main/java/org/apache/accumulo/monitor/servlets/trace/ShowTrace.java

Added null check to avoid NPE

Co-Authored-By: Christopher Tubbs <ctubbsii@apache.org>
* automatically adjusted formatting

2 years agoImproved metadata table consitency checking and testing (#1379)
Keith Turner [Thu, 10 Oct 2019 14:55:08 +0000 (10:55 -0400)] 
Improved metadata table consitency checking and testing (#1379)

Added unit test to exercise the code that checks the consistency of the
metadata table during GC scans.  Also added a check to ensure the last
tablet was seen.  This should cause an error to be thrown if the scanner
stops returning data for any reason.  These changes were already made in
2.0 in #1266.  However the code in 1.9 is so different that they could
not be back-ported, but had to be re-implemented.

2 years agoExpanded InputConfigurator permissions checks to include Namespace.READ (#1371)
Richard W. Eggert II [Fri, 4 Oct 2019 23:03:36 +0000 (19:03 -0400)] 
Expanded InputConfigurator permissions checks to include Namespace.READ (#1371)

* created tests to cover #1370

* expanded checks in InputConfigurator.validatePermissions to include Namespace.READ and resolve #1370

* auto-adjusted formatting

* removed unnecessary assertions from testGetSplitsWithNamespaceReadPermission

2 years agoAdd colVis file to ContinuousInputFormat. Fixes #1368 (#1369)
Mike Miller [Thu, 19 Sep 2019 22:46:49 +0000 (18:46 -0400)] 
Add colVis file to ContinuousInputFormat. Fixes #1368 (#1369)

2 years agoFix visibilities option in run-bulk-generate.sh
Mike Miller [Wed, 18 Sep 2019 21:08:04 +0000 (17:08 -0400)] 
Fix visibilities option in run-bulk-generate.sh

2 years agoFix #1345 - Allow gc initiated compaction as option. (#1352)
EdColeman [Thu, 12 Sep 2019 19:06:06 +0000 (15:06 -0400)] 
Fix #1345 - Allow gc initiated compaction as option. (#1352)

* Fix #1345 - Allow gc initiated compaction as option.

Added a property - gc.use.full.compaction that defaults to current behaviour. When false,
performs a blocking flush instead.

* fix type in property description.

* wip: use enum, compact, flush or none

* fix test failure by adding PropertyTest

* added timing log messsgae

* update durations to use nanoseconds

2 years agoFix #1338 Reduce verbose logging of merge operations in Master log
EdColeman [Fri, 6 Sep 2019 18:57:43 +0000 (14:57 -0400)] 
Fix #1338 Reduce verbose logging of merge operations in Master log

* reduce merge operations logging of extent info at debug level - issue #1338.

* minor cleanup removed always true param

* address comments, clean-up of MergeStats to use substition.

* code cleanup and logging changes for discussion

* code cleanup and logging changes for discussion

 - address review comments
 - merge changes m.miller removing unused code in TableGroupWatcher

* Stop logging during merges

* address review comments

* clarify comment after clean-up removed empty else clause.

2 years agoFix #1347 Update log statement for user compactions without user iterators
EdColeman [Thu, 5 Sep 2019 19:59:11 +0000 (15:59 -0400)] 
Fix #1347 Update log statement for user compactions without user iterators

2 years agoRemoved unused code in TabletGroupWatcher (#1341)
Mike Miller [Tue, 3 Sep 2019 15:48:11 +0000 (11:48 -0400)] 
Removed unused code in TabletGroupWatcher (#1341)

* Also make some local vars private

2 years agoFix importdirectory tab completion (#1335)
EdColeman [Mon, 26 Aug 2019 14:52:38 +0000 (10:52 -0400)] 
Fix importdirectory tab completion (#1335)

2 years agoClean up importDirectory IT (remove unneeded method)
Christopher Tubbs [Fri, 16 Aug 2019 19:09:55 +0000 (15:09 -0400)] 
Clean up importDirectory IT (remove unneeded method)

2 years agoImprove importDirectory tests from #1299
Christopher Tubbs [Fri, 16 Aug 2019 00:29:18 +0000 (20:29 -0400)] 
Improve importDirectory tests from #1299

* Remove unused variables in test
* Verify mock objects in test

2 years agoAdd optional -t tablename to importdirectory shell command. (#1299)
EdColeman [Thu, 15 Aug 2019 19:41:15 +0000 (15:41 -0400)] 
Add optional -t tablename to importdirectory shell command. (#1299)

* Add optional -t tablename to importdirectory shell command.

the -t tablename is optional to preserve original behaviour. Without
the option, the current table is used.

* added getOptions

* wip - tests cmd format, need to evaluate testing functionality

2 years agoFix #1314 fixes gc thread issue by creating one timed task instance (#1318)
EdColeman [Thu, 15 Aug 2019 19:00:04 +0000 (15:00 -0400)] 
Fix #1314 fixes gc thread issue by creating one timed task instance (#1318)

* issue #1314 - call scanServers directly, no schedule tasks necessary

* create one tserver set with timed task outside of run loop.

* addressed review comment

* Added call to scanServers, updated unit tests.

* updated logging with added detail at trace level.

* merge changes from 1.9

2 years agofix #1326 Copy instead of clone metadata table in IT
Keith Turner [Thu, 15 Aug 2019 17:25:12 +0000 (13:25 -0400)] 
fix #1326 Copy instead of clone metadata table in IT

After the changes in #1309 cloning of metdata table is no longer
allowed. TabletStateChangeIteratorIT relied on cloning and
was changed to copy instead.

Also the test was very sensitive to concurrent chnages in the metadata
table.  Suspect that cloning used to introduce a delay that hid this.
The change from cloning to copying caused the test to fail often
because of these timing issues.  To avoid this, the test was refactored
to tolerate concurrent changes to the metadata table.

2 years agoBackport BulkIngest test. Closes #1288 (#1312)
Mike Miller [Thu, 15 Aug 2019 13:52:57 +0000 (09:52 -0400)] 
Backport BulkIngest test. Closes #1288 (#1312)

* Created classes for running the bulk generate job
* Created run-bulk-generate.sh
* Also updated the example test files

2 years agoExamine WALs used by root tablet during GC (#1310)
Keith Turner [Wed, 14 Aug 2019 20:07:07 +0000 (16:07 -0400)] 
Examine WALs used by root tablet during GC (#1310)

2 years agofix #1309 prevent cloning accumulo.metadata (#1311)
Keith Turner [Wed, 14 Aug 2019 20:06:26 +0000 (16:06 -0400)] 
fix #1309 prevent cloning accumulo.metadata (#1311)

2 years agoAdd patch file for Java 11 generated apidocs
Christopher Tubbs [Sun, 4 Aug 2019 15:16:44 +0000 (11:16 -0400)] 
Add patch file for Java 11 generated apidocs

Adds javadoc11.patch to patch broken javadoc in Java 11. These were
patched in Java 12 and later, but have not yet been backported to 11 at
the time of this writing, or they have not been patched upstream at all
yet. So, it may be necessary to apply this patch wnen generating
javadocs (using the aggregate-javadocs profile) for the API published on
the accumulo-website, if they were generated using Java 11.

The relevant upstream issues are:

* https://bugs.openjdk.java.net/browse/JDK-8227487 (Missing dejavu font)
* https://bugs.openjdk.java.net/browse/JDK-8214856 (JSZip API misuse)
* https://bugs.openjdk.java.net/browse/JDK-8215291 (Broken search links)

The patch can be applied by doing:

```bash
# generate the javadocs for the site
mvn clean package -DskipTests javadoc:aggregate -Paggregate-javadocs
# apply the patch
patch -p1 < contrib/javadoc11.patch
```