bookkeeper.git
2 days agoUpdated organization to Streamlio former members (#3280) master
AlvaroStream [Fri, 13 May 2022 16:03:07 +0000 (18:03 +0200)] 
Updated organization to Streamlio former members (#3280)

Update the organization to former Streamlio members to StreamNative

2 days ago4.14.5 release notes and doc (#3199)
Nicolò Boschi [Fri, 13 May 2022 08:03:35 +0000 (10:03 +0200)] 
4.14.5 release notes and doc (#3199)

2 days ago[website] Add script for cut new release (release-notes, new docs, sidebar..) (#3198)
Nicolò Boschi [Fri, 13 May 2022 07:50:22 +0000 (09:50 +0200)] 
[website] Add script for cut new release (release-notes, new docs, sidebar..) (#3198)

3 days ago[website] fix hyperlink (#3276)
赤月 [Thu, 12 May 2022 12:48:28 +0000 (20:48 +0800)] 
[website] fix hyperlink (#3276)

Co-authored-by: zwen <zwen@didiglobal.com>
4 days agoBump prometheus client version to 0.15.0 (#3266)
ZhangJian He [Wed, 11 May 2022 23:50:13 +0000 (07:50 +0800)] 
Bump prometheus client version to 0.15.0 (#3266)

### Motivation
prometheus client `0.8.1` is release on Jan,2020, Keep update

### Changes
Bump prometheus client version from 0.8.1 to 0.15.0

### Test
Update prometheus client to 0.15.0, compare the metrics txt, the results are same.

4 days agoEnable Netty and BookKeeper IO optimizations on jdk17 (#3234)
Nicolò Boschi [Wed, 11 May 2022 17:17:44 +0000 (19:17 +0200)] 
Enable Netty and BookKeeper IO optimizations on jdk17 (#3234)

* Enable netty optimization and BK native IO on jdk17

* zip and better docs

4 days agoFix link Apache Pulsar#persistent-storage (#3271)
赤月 [Wed, 11 May 2022 11:38:32 +0000 (19:38 +0800)] 
Fix link Apache Pulsar#persistent-storage  (#3271)

5 days agocleaning up leftover 4.15-snapshot versions (#3267)
Andrey Yegorov [Tue, 10 May 2022 19:13:33 +0000 (12:13 -0700)] 
cleaning up leftover 4.15-snapshot versions (#3267)

* cleaning up leftover 4.15-snapshot versions

* updated news page

5 days ago[website] Set 4.15.0 as latest release (#3265)
Nicolò Boschi [Tue, 10 May 2022 06:58:36 +0000 (08:58 +0200)] 
[website] Set 4.15.0 as latest release (#3265)

6 days agoRelease notes for BK 4.15
Andrey Yegorov [Mon, 9 May 2022 18:51:58 +0000 (11:51 -0700)] 
Release notes for BK 4.15

Descriptions of the changes in this PR:
### Motivation

release notes for BK release 4.15

### Changes

Doc updates per release procedure.

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Yong Zhang <zhangyong1025.zy@gmail.com>, Hang Chen <chenhang@apache.org>, Nicolò Boschi <boschi1997@gmail.com>

This closes #3155 from dlg99/bk-4.15-release-notes

8 days agoFix spelling mistake in bk_server.conf (#3262)
Tian Luo [Sat, 7 May 2022 12:11:38 +0000 (20:11 +0800)] 
Fix spelling mistake in bk_server.conf (#3262)

### Motivation

Fix spelling mistake in bk_server.conf

### Changes

Master Issue: #3261

10 days agoBP-47 (task5): Garbage collection support direct IO entrylogger (#3256)
Hang Chen [Thu, 5 May 2022 06:29:11 +0000 (14:29 +0800)] 
BP-47 (task5): Garbage collection support direct IO entrylogger (#3256)

* Garbage Collection support for DirectIO entrylogger

GC support requires that the entrylogger provides a way to retrieve
all entrylogs which have been completely flushed to disk. Previously
this was done by returning the least unflushed log id. However, this
is problematic as it doesn't support the log ids wrapping around. It
also means that GC has to start checking for log id existence from
zero every time it boots.

This change replaces getLeastUnflushedLogId() with getFlushedLogIds(),
to give the entrylogger full control of which logs should be
considered for GC.

It also changes the CompactableLedgerStorage interface, removing
getEntryLogger() and adding injection of the entrylogger to the
GarbageCollectionThread. This makes testing easier.

Co-authored-by: Ivan Kelly <ikelly@splunk.com>
11 days ago[minor] [optimize] Remove redundant toString call (#3254)
ZhangJian He [Thu, 5 May 2022 00:26:43 +0000 (08:26 +0800)] 
[minor] [optimize] Remove redundant toString call (#3254)

11 days agoAllow run LocalBookkeeper directly in `bookkeeper-server` module or on IDE (#3255)
ZhangJian He [Wed, 4 May 2022 13:20:11 +0000 (21:20 +0800)] 
Allow run LocalBookkeeper directly in `bookkeeper-server` module or on IDE (#3255)

11 days agoBP-47 (task4): Aligned native buffer wrapper (#3253)
Hang Chen [Wed, 4 May 2022 06:18:23 +0000 (14:18 +0800)] 
BP-47 (task4): Aligned native buffer wrapper (#3253)

2 weeks agoFix: shell tests are broken and not running on CI (#3251)
Andrey Yegorov [Fri, 29 Apr 2022 16:10:42 +0000 (09:10 -0700)] 
Fix: shell tests are broken and not running on CI (#3251)

2 weeks agoBP-47 (task3): Abstract interface for entrylogger (#3197)
Hang Chen [Fri, 29 Apr 2022 05:44:14 +0000 (13:44 +0800)] 
BP-47 (task3): Abstract interface for entrylogger (#3197)

* Interface for entrylogger

Define the interface and contract for entrylogger. This is mostly
taking the entrylogger methods used by other components and prettying
them up a bit.

Notable changes are:
- internalReadEntry is now readEntry. There is no 'validate'
  flag. Instead there are two overloads for the method, and validation
  only runs if ledgerId and entryId are passed in.
- shutdown has been renamed to close.
- the compaction entrylog methods have been put behind an
  interface. As it was they were leaking implementation
  details. Ultimitely compaction itself should be hidden behind the
  entrylogger, but that's a larger refactor.

(cherry picked from commit c927f4b0128e2b1f0f4b30c41ce442b366d5009c)

* abstract interface for entrylogger

* format code

* format code

* address comments

* Apply suggestions from code review

Co-authored-by: Andrey Yegorov <8622884+dlg99@users.noreply.github.com>
* address comments

* format code

Co-authored-by: Ivan Kelly <ikelly@splunk.com>
Co-authored-by: Andrey Yegorov <8622884+dlg99@users.noreply.github.com>
2 weeks agoMove bookkeeper-stats and bookkeeper-stats-providers as submodules of stats
ZhangJian He [Thu, 28 Apr 2022 23:00:53 +0000 (07:00 +0800)] 
Move bookkeeper-stats and bookkeeper-stats-providers as submodules of stats

### Motiviations
Fix todo to make the project layout more reasonable and easier to fix bug/dev new feature
### Changes
move `bookkeeper-stats` and `bookkeeper-stats-providers` as submodules of `stats`

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Nicolò Boschi <boschi1997@gmail.com>, Andrey Yegorov <None>

This closes #3246 from Shoothzj/module-04-25

2 weeks agofix some jdk17 test problem (#3242)
ZhangJian He [Wed, 27 Apr 2022 07:09:35 +0000 (15:09 +0800)] 
fix some jdk17 test problem (#3242)

2 weeks ago[maven-release-plugin] prepare for next development iteration
Andrey Yegorov [Tue, 26 Apr 2022 21:48:53 +0000 (14:48 -0700)] 
[maven-release-plugin] prepare for next development iteration

2 weeks ago[maven-release-plugin] prepare branch branch-4.15
Andrey Yegorov [Tue, 26 Apr 2022 21:48:51 +0000 (14:48 -0700)] 
[maven-release-plugin] prepare branch branch-4.15

2 weeks agoRevert "[maven-release-plugin] prepare branch branch-4.15"
Andrey Yegorov [Tue, 26 Apr 2022 21:47:58 +0000 (14:47 -0700)] 
Revert "[maven-release-plugin] prepare branch branch-4.15"

This reverts commit d9b98ad8ccec6a9ba65cfc00ef319ee51454c0c4.

2 weeks agoRevert "[maven-release-plugin] prepare for next development iteration"
Andrey Yegorov [Tue, 26 Apr 2022 21:47:48 +0000 (14:47 -0700)] 
Revert "[maven-release-plugin] prepare for next development iteration"

This reverts commit e5d5ddac8a10acca031eed0221d925d54713a6ab.

2 weeks ago[maven-release-plugin] prepare for next development iteration
Andrey Yegorov [Tue, 26 Apr 2022 21:31:55 +0000 (14:31 -0700)] 
[maven-release-plugin] prepare for next development iteration

2 weeks ago[maven-release-plugin] prepare branch branch-4.15
Andrey Yegorov [Tue, 26 Apr 2022 21:31:52 +0000 (14:31 -0700)] 
[maven-release-plugin] prepare branch branch-4.15

2 weeks agoAdd get under replicated ledger count
Hang Chen [Tue, 26 Apr 2022 06:17:37 +0000 (14:17 +0800)] 
Add get under replicated ledger count

### Motivation
Fix #3224

### Changes
Add flag for `ListUnderreplicatedCmd` to control only display under replicated ledgers count

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Yong Zhang <zhangyong1025.zy@gmail.com>, Andrey Yegorov <None>

This closes #3228 from hangc0276/chenhang/add_get_under_replicate_ledger_count

2 weeks agoCorrect the method signature within annotations
iamazy [Tue, 26 Apr 2022 01:14:46 +0000 (09:14 +0800)] 
Correct the method signature within annotations

Descriptions of the changes in this PR:

### Motivation

Some interfaces have modified the method signature but the annotations have not.

### Changes

Correct the method signature within annotations and fix typos

Reviewers: Andrey Yegorov <None>, ZhangJian He <shoothzj@gmail.com>

This closes #3201 from iamazy/fix-typo

2 weeks agoFix bookie shell help command exception
Hang Chen [Mon, 25 Apr 2022 22:35:16 +0000 (06:35 +0800)] 
Fix bookie shell help command exception

### Motivation
When run `bin/bookkeeper shell ledgermetadata -h` command, it throw the following exception.
```
23:02:41,236 ERROR Error parsing command line arguments :
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: -h
at org.apache.commons.cli.Parser.processOption(Parser.java:363) ~[commons-cli-commons-cli-1.2.jar:1.2]
at org.apache.commons.cli.Parser.parse(Parser.java:199) ~[commons-cli-commons-cli-1.2.jar:1.2]
at org.apache.commons.cli.Parser.parse(Parser.java:85) ~[commons-cli-commons-cli-1.2.jar:1.2]
at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:240) ~[org.apache.bookkeeper-bookkeeper-server-4.16.0-SNAPSHOT.jar:4.16.0-SNAPSHOT]
at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:2372) ~[org.apache.bookkeeper-bookkeeper-server-4.16.0-SNAPSHOT.jar:4.16.0-SNAPSHOT]
at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:2463) ~[org.apache.bookkeeper-bookkeeper-server-4.16.0-SNAPSHOT.jar:4.16.0-SNAPSHOT]
ledgermetadata: Print the metadata for a ledger, or optionally dump to a file.
usage: ledgermetadata -ledgerid <ledgerid> [--dump-to-file
                      FILENAME|--restore-from-file FILENAME]
 -dumptofile <arg>        Dump metadata for ledger, to a file
 -l,--ledgerid <arg>      Ledger ID
 -restorefromfile <arg>   Restore metadata for ledger, from a file
```

### Changes
Add `-h` and `--help` support.

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, ZhangJian He <shoothzj@gmail.com>, Andrey Yegorov <None>

This closes #3227 from hangc0276/chenhang/fix_bookieshell_help_exception

2 weeks agoFormat log statements
Hang Chen [Mon, 25 Apr 2022 22:32:20 +0000 (06:32 +0800)] 
Format log statements

### Motivation
In PR #2455 , It convert all `System.*.print` statements to `LOG.* statements`. However, in `LOG.*` statements, it uses string concatenation to print messages, which is low performance.

### Changes
Change the string concatenation format to `LOG.*` standard statements

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Andrey Yegorov <None>, ZhangJian He <shoothzj@gmail.com>, Nicolò Boschi <boschi1997@gmail.com>, Yong Zhang <zhangyong1025.zy@gmail.com>

This closes #3213 from hangc0276/chenhang/format_log_info

2 weeks agoFix read empty ledger exception using bookkeeper admin
Hang Chen [Mon, 25 Apr 2022 22:28:04 +0000 (06:28 +0800)] 
Fix read empty ledger exception using bookkeeper admin

### Motivation
Fix #3222

The root cause is that BookKeeperAdmin use `asyncReadEntriesInternal` to read entries. However, the `asyncReadEntriesInternal` doesn't check the lastAddConfirm, which will lead to the exception list in issue 3222

### Changes

Add lastAddConfirm check before call `asyncReadEntriesInternal`

Reviewers: Yong Zhang <zhangyong1025.zy@gmail.com>, Andrey Yegorov <None>

This closes #3240 from hangc0276/chenhang/fix_BKBookieHandleNotAvailableException_for_empty_ledger

2 weeks agoBump jetty to 9.4.46.v20220331 to get rid of CVE-2021-34429
Nicolò Boschi [Mon, 25 Apr 2022 22:25:35 +0000 (00:25 +0200)] 
Bump jetty to 9.4.46.v20220331 to get rid of CVE-2021-34429

### Motivation

Current jetty version is vulnerable to https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-34429

### Changes

* Upgrade to the latest 9.x one 9.4.46.v20220331

Reviewers: Andrey Yegorov <None>, ZhangJian He <shoothzj@gmail.com>, Tian Luo <futurer@outlook.com>, Yong Zhang <zhangyong1025.zy@gmail.com>

This closes #3232 from nicoloboschi/bump-jetty-9.4.46.v20220331

3 weeks ago[ci] [github] upload test log when fail (#3241)
ZhangJian He [Sun, 24 Apr 2022 10:42:21 +0000 (18:42 +0800)] 
[ci] [github] upload test log when fail (#3241)

3 weeks agoIssue 2518:fix error link (#3217)
leiwingqueen [Sun, 24 Apr 2022 02:52:45 +0000 (10:52 +0800)] 
Issue 2518:fix error link (#3217)

### Motivation

(Explain: why you're making that change, what is the problem you're trying to solve)
[issue 2518](https://github.com/apache/bookkeeper/issues/2518)

### Changes

just try to fix the error link.

3 weeks ago[build] make build passes on JDK17 (#3053)
Nicolò Boschi [Fri, 22 Apr 2022 16:32:11 +0000 (18:32 +0200)] 
[build] make build passes on JDK17 (#3053)

3 weeks ago[minor] cleanup on InterleavedLedgerStorage (#3210)
ZhangJian He [Fri, 22 Apr 2022 16:10:59 +0000 (00:10 +0800)] 
[minor] cleanup on InterleavedLedgerStorage (#3210)

3 weeks agoRemove log4j from bookkeeper (#3225)
ZhangJian He [Fri, 22 Apr 2022 16:10:18 +0000 (00:10 +0800)] 
Remove log4j from bookkeeper (#3225)

3 weeks agobookeeper all missing stream binary (#3238)
ZhangJian He [Fri, 22 Apr 2022 16:06:17 +0000 (00:06 +0800)] 
bookeeper all missing stream binary (#3238)

3 weeks agofix logger number not correct (#3211)
ZhangJian He [Fri, 22 Apr 2022 01:00:01 +0000 (09:00 +0800)] 
fix logger number not correct (#3211)

### Motivation

fix logger number not correct

### Changes

fix logger number not correct

3 weeks ago[minor] replace deprecated api use MockitoAnnotations.initMocks (#3177)
ZhangJian He [Thu, 21 Apr 2022 11:42:58 +0000 (19:42 +0800)] 
[minor] replace deprecated api use MockitoAnnotations.initMocks (#3177)

3 weeks agofix broken local bookie (#3230)
ZhangJian He [Thu, 21 Apr 2022 11:42:33 +0000 (19:42 +0800)] 
fix broken local bookie (#3230)

3 weeks ago[ZooKeeper] [Conflict] Bump dropwizard metrics to zookeeper needed (#3229)
ZhangJian He [Thu, 21 Apr 2022 07:50:09 +0000 (15:50 +0800)] 
[ZooKeeper] [Conflict] Bump dropwizard metrics to zookeeper needed (#3229)

3 weeks agofix cli command doc issue (#3226)
Hang Chen [Thu, 21 Apr 2022 00:27:08 +0000 (08:27 +0800)] 
fix cli command doc issue (#3226)

### Motivation
Fix #3221

### Changes
Fix some cli command doc issue, and keep sync with code.

3 weeks agoBump jackson databind version to 2.13.2.2
ZhangJian He [Wed, 20 Apr 2022 14:18:20 +0000 (22:18 +0800)] 
Bump jackson databind version to 2.13.2.2

### Changes
- jackson bom will release a version when everytime its components releases, so we don't need to define variables like `jackson-data.version`
- bump jackson databind version to 2.13.2.2

Reviewers: Nicolò Boschi <boschi1997@gmail.com>, Enrico Olivelli <eolivelli@gmail.com>

This closes #3216 from Shoothzj/bump-jackson-databind-version-2-13-2-2

3 weeks agoBump async from 2.6.3 to 2.6.4 in /site3/website (#3219)
dependabot[bot] [Wed, 20 Apr 2022 14:15:35 +0000 (16:15 +0200)] 
Bump async from 2.6.3 to 2.6.4 in /site3/website (#3219)

Bumps [async](https://github.com/caolan/async) from 2.6.3 to 2.6.4.
- [Release notes](https://github.com/caolan/async/releases)
- [Changelog](https://github.com/caolan/async/blob/v2.6.4/CHANGELOG.md)
- [Commits](https://github.com/caolan/async/compare/v2.6.3...v2.6.4)

---
updated-dependencies:
- dependency-name: async
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
3 weeks agoFix force GC doesn't work under forceAllowCompaction when disk is full (#3205)
Hang Chen [Wed, 20 Apr 2022 13:43:15 +0000 (21:43 +0800)] 
Fix force GC doesn't work under forceAllowCompaction when disk is full (#3205)

## Motivation
When I set `forceAllowCompaction=true` and one ledger disk reaches max usage threshold and transfer bookie to readOnly mode, I expire some pulsar topics or delete some topics to free up disk space. I found that ledger compression cannot be triggered when using `curl -XPUT http://localhost:8000/api/v1/bookie/gc` command.

The root cause is that when one ledger disk reaches max usage threshold, it will suspend minor and major compaction
https://github.com/apache/bookkeeper/blob/f7579fd13d62ce630ea26638e73f5884da505ec8/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/SingleDirectoryDbLedgerStorage.java#L1041-L1058

When we use `curl -XPUT http://localhost:8000/api/v1/bookie/gc` command to trigger compaction, it will be filtered by `suspendMajor` and `suspendMinor` flag.

https://github.com/apache/bookkeeper/blob/f7579fd13d62ce630ea26638e73f5884da505ec8/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java#L416-L444

It will lead to
- The bookie won't clean up deleted ledgers
- Ledger disk can't free up disk usage
- Bookie can't recover from readOnly state into Writeable state.

And then we can only trigger compaction by the following steps.
- Increase max disk usage threshold
- Restart the bookie
- Use command `curl -XPUT http://localhost:8000/api/v1/bookie/gc` to trigger compaction

### Changes
1. Don't take the `suspendMajor` and `suspendMinor` flag into consideration when setting `forceAllowCompaction=true` and triggered by force GC.

3 weeks agofix numLedgersReplicated metric not update (#3218)
Hang Chen [Tue, 19 Apr 2022 19:25:04 +0000 (03:25 +0800)] 
fix numLedgersReplicated metric not update (#3218)

3 weeks agoRemove log4j1 from bk test code
ZhangJian He [Tue, 19 Apr 2022 16:34:50 +0000 (00:34 +0800)] 
Remove log4j1 from bk test code

### Motivation
this is a part work of remove log4j1 from bookeeper, first remove it from our test code, unit test and integrate test

### Changes
- remove log4j1 scope test dependency
- use log4j2.xml instread of log4j.properties
- introduce testtools module to reduce duplication,  if a module needs particular configuration (very unlikely) we can add a log4j2-test.xml file which has precedence over log4j2.xml one. idea from [nicoloboschi](https://github.com/nicoloboschi)

Reviewers: Andrey Yegorov <None>, Enrico Olivelli <eolivelli@gmail.com>, Nicolò Boschi <boschi1997@gmail.com>

This closes #3208 from Shoothzj/remove-log4j1-from-bk-test-code2

4 weeks agoFix testCompactionPersistence do not cleanup created directory (#3207)
Hang Chen [Fri, 15 Apr 2022 06:00:44 +0000 (14:00 +0800)] 
Fix testCompactionPersistence do not cleanup created directory (#3207)

4 weeks ago[website] Make site3 the official website (deploy scripts - master) (#3187)
Nicolò Boschi [Thu, 14 Apr 2022 09:49:57 +0000 (11:49 +0200)] 
[website] Make site3 the official website (deploy scripts - master) (#3187)

* [website] Make site3 the official website

* fix action

4 weeks agoFix master broken ci due to zookeeper CVE fp (#3204)
ZhangJian He [Wed, 13 Apr 2022 12:37:08 +0000 (20:37 +0800)] 
Fix master broken ci due to zookeeper CVE fp (#3204)

4 weeks agosimplify throttle code in metadata scanning (#3196)
Hang Chen [Wed, 13 Apr 2022 01:49:15 +0000 (09:49 +0800)] 
simplify throttle code in metadata scanning (#3196)

### Motivation
The throttler has been checked whether throttle by bytes in `acquire` method, we doesn't need double check.

### Changes

Remove the double check of whether throttle by bytes.

4 weeks agoexpose netty threads num config to bk_server.conf (#3173)
Hang Chen [Wed, 13 Apr 2022 01:47:54 +0000 (09:47 +0800)] 
expose netty threads num config to bk_server.conf (#3173)

### Motivation
https://github.com/apache/bookkeeper/pull/1612 this PR make the Netty io threads number configurable, but doesn't expose to the `bk_server.conf`. This parameter will be usually changed in performance tuning especially deploy this bookie on bare metal which have many cpus. For operation engineer who is unfamiliar with the source code will be hard to do performance tuning.

### Changes
Expose this parameter configuration into `bk_server.conf` and website. And add more explanation about it.

4 weeks agoremove unnessary exception catch (#3169)
Hang Chen [Wed, 13 Apr 2022 01:47:20 +0000 (09:47 +0800)] 
remove unnessary exception catch (#3169)

### Motivation
When read Lac from ledger, we won't throw NoEntryException, remove this exception catch.

### Changes
Remove the NoEntryException catch in ReadLac operation.

5 weeks agoAllow running units test on jdk17 (#3195)
ZhangJian He [Mon, 11 Apr 2022 01:03:22 +0000 (09:03 +0800)] 
Allow running units test on jdk17 (#3195)

* Allow running units test on jdk17

* Allow running units test on jdk17

5 weeks agoBP-47 (task2): Add structured logger support (#3190)
Hang Chen [Sun, 10 Apr 2022 23:50:18 +0000 (07:50 +0800)] 
BP-47 (task2): Add structured logger support (#3190)

* Add slogger support

* update pom.xml

* fix check style

5 weeks agoBP-47 (task1): Add native IO library support (#3189)
Hang Chen [Sun, 10 Apr 2022 23:49:39 +0000 (07:49 +0800)] 
BP-47 (task1): Add native IO library support (#3189)

* Add native io library support

* fix jni compile issue

* fix findbugs failed

5 weeks agoUpgrade to log4j 2.17.2. (#3188)
Raúl Gracia [Thu, 7 Apr 2022 06:56:36 +0000 (08:56 +0200)] 
Upgrade to log4j 2.17.2. (#3188)

Signed-off-by: Raúl Gracia <raul.gracia@emc.com>
5 weeks agoonly update topology when bookie rack changed
gaozhangmin [Wed, 6 Apr 2022 15:58:47 +0000 (23:58 +0800)] 
only update topology when bookie rack changed

It is unnecessary to update  topology, removing and adding the same bookieNode, if the rack of bookie stay unchanged.

Reviewers: Yong Zhang <zhangyong1025.zy@gmail.com>, Nicolò Boschi <boschi1997@gmail.com>, Andrey Yegorov <None>, ZhangJian He <shoothzj@gmail.com>, Enrico Olivelli <eolivelli@gmail.com>

This closes #2790 from gaozhangmin/remove-unnecessary-update-rack

5 weeks ago[BUILD] fix master branch broken http-core license check
ZhangJian He [Wed, 6 Apr 2022 15:37:28 +0000 (23:37 +0800)] 
[BUILD] fix master branch broken http-core license check

### Changes
- the bkctl and bk-server has different `http-core` version
- unify the `http-core` version to 4.4.15

Reviewers: Andrey Yegorov <None>, Nicolò Boschi <boschi1997@gmail.com>, Enrico Olivelli <eolivelli@gmail.com>

This closes #3183 from Shoothzj/fix-broken-license-check

5 weeks ago[website] add CI checks to validate the website (#3164)
Nicolò Boschi [Wed, 6 Apr 2022 09:17:07 +0000 (11:17 +0200)] 
[website] add CI checks to validate the website (#3164)

5 weeks agoBump minimist from 1.2.5 to 1.2.6 in /site3/website (#3179)
dependabot[bot] [Wed, 6 Apr 2022 09:07:50 +0000 (11:07 +0200)] 
Bump minimist from 1.2.5 to 1.2.6 in /site3/website (#3179)

Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)

---
updated-dependencies:
- dependency-name: minimist
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
5 weeks agoBump node-forge from 1.2.1 to 1.3.1 in /site3/website (#3180)
dependabot[bot] [Wed, 6 Apr 2022 09:06:54 +0000 (11:06 +0200)] 
Bump node-forge from 1.2.1 to 1.3.1 in /site3/website (#3180)

Bumps [node-forge](https://github.com/digitalbazaar/forge) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/digitalbazaar/forge/releases)
- [Changelog](https://github.com/digitalbazaar/forge/blob/main/CHANGELOG.md)
- [Commits](https://github.com/digitalbazaar/forge/compare/v1.2.1...v1.3.1)

---
updated-dependencies:
- dependency-name: node-forge
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
5 weeks agoUse netty maxDirectMemory instead of DirectMemoryUtils
ZhangJian He [Wed, 6 Apr 2022 08:13:13 +0000 (16:13 +0800)] 
Use netty maxDirectMemory instead of DirectMemoryUtils

### Motivation

Our `DirectMemoryUtils` has huge limit, it can't work well with other jvm. The Netty `PlatformDependent.maxDirectMemory();` is more generic.

### Changes
Use `PlatformDependent.maxDirectMemory();` instead of `DirectMemoryUtils`

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Andrey Yegorov <None>, Matteo Merli <mmerli@apache.org>, Nicolò Boschi <boschi1997@gmail.com>

This closes #2989 from Shoothzj/direct-memory

5 weeks ago[build] support apple m1 build (#3175)
ZhangJian He [Wed, 6 Apr 2022 05:46:17 +0000 (13:46 +0800)] 
[build] support apple m1 build (#3175)

6 weeks ago[netty] remove no longer used properties io.netty.recycler.linkCapacity and io.netty...
Nicolò Boschi [Sun, 3 Apr 2022 16:09:49 +0000 (18:09 +0200)] 
[netty] remove no longer used properties io.netty.recycler.linkCapacity and io.netty.recycler.maxCapacity.default (#3172)

* [netty] remove no longer used properties io.netty.recycler.linkCapacity and io.netty.recycler.maxCapacity.default

* gradle

6 weeks agoBump netty version to 4.1.75.Final, grpc to 1.45.1 (#3163)
ZhangJian He [Sun, 3 Apr 2022 08:00:21 +0000 (16:00 +0800)] 
Bump netty version to 4.1.75.Final, grpc to 1.45.1 (#3163)

6 weeks ago[build] Complement missing maven plugin version (#3166)
ZhangJian He [Sat, 2 Apr 2022 19:22:13 +0000 (03:22 +0800)] 
[build] Complement missing maven plugin version (#3166)

6 weeks ago[security] Bump bc fips version from 1.0.2.1 to 1.0.2.3 (#3087)
ZhangJian He [Sat, 2 Apr 2022 19:21:18 +0000 (03:21 +0800)] 
[security] Bump bc fips version from 1.0.2.1 to 1.0.2.3 (#3087)

6 weeks ago[WEBSITE] Update current stable version to 4.14.4
Nicolò Boschi [Fri, 1 Apr 2022 19:54:24 +0000 (21:54 +0200)] 
[WEBSITE] Update current stable version to 4.14.4

### Motivation

The current stable version is 4.14.4 but in the website is still 4.11.1

### Changes

* Edit variables to point to 4.14.4
* Cleanup staging website "download" page

Reviewers: Yong Zhang <zhangyong1025.zy@gmail.com>

This closes #3154 from nicoloboschi/update-stable-version

6 weeks ago[build] Fix various spotbugs warnings (#3160)
Nicolò Boschi [Fri, 1 Apr 2022 07:37:57 +0000 (09:37 +0200)] 
[build] Fix various spotbugs warnings  (#3160)

6 weeks agoFix region/rack aware placement police replace bookie bug (#2642)
Hang Chen [Thu, 31 Mar 2022 13:39:47 +0000 (21:39 +0800)] 
Fix region/rack aware placement police replace bookie bug (#2642)

6 weeks ago[website] Update committers page
Nicolò Boschi [Thu, 31 Mar 2022 11:45:12 +0000 (13:45 +0200)] 
[website] Update committers page

### Changes
Update committers info

Reviewers: Andrey Yegorov <None>

This closes #3161 from nicoloboschi/update-committers-nicoloboschi

6 weeks agoRevert rocksdb compaction on checkpoint to reduce cpu intensive (#3144)
Hang Chen [Thu, 31 Mar 2022 07:08:13 +0000 (15:08 +0800)] 
Revert rocksdb compaction on checkpoint to reduce cpu intensive (#3144)

6 weeks agocatch onBookieRackChange exception (#3060)
Hang Chen [Thu, 31 Mar 2022 05:42:34 +0000 (13:42 +0800)] 
catch onBookieRackChange exception (#3060)

### Motivation
When we update the bookie rack info, it will use all the bookie list to update rack topology. However If one bookie update failed and throw exception out, it will throw the exception out and the remains bookie info won't be updated into the rack topology, which will affect the ledger ensemble selection.

### Changes

Catch the bookie topology update exception to ensure the remaining bookies' info can be updated into the rack topology.

6 weeks agouse mockito.any instead of deprecated mockito.anyObject
ZhangJian He [Wed, 30 Mar 2022 23:10:25 +0000 (07:10 +0800)] 
use mockito.any instead of deprecated mockito.anyObject

### Changes
use mockito.any instead of deprecated mockito.anyObject

Reviewers: Andrey Yegorov <None>, Nicolò Boschi <boschi1997@gmail.com>

This closes #3152 from Shoothzj/use-mockito-any-instead-of-anyObject

6 weeks agoUpgrade ZooKeeper to 3.8.0 (#3145)
Enrico Olivelli [Wed, 30 Mar 2022 10:41:33 +0000 (12:41 +0200)] 
Upgrade ZooKeeper to 3.8.0 (#3145)

6 weeks agofix duplicate typeline for prometheus type (#3137)
ZhangJian He [Tue, 29 Mar 2022 06:29:00 +0000 (14:29 +0800)] 
fix duplicate typeline for prometheus type (#3137)

6 weeks ago[maven-release-plugin] prepare for next development iteration
Andrey Yegorov [Mon, 28 Mar 2022 23:01:51 +0000 (16:01 -0700)] 
[maven-release-plugin] prepare for next development iteration

6 weeks ago[maven-release-plugin] prepare branch branch-4.15
Andrey Yegorov [Mon, 28 Mar 2022 23:01:48 +0000 (16:01 -0700)] 
[maven-release-plugin] prepare branch branch-4.15

6 weeks agoFix NPE while reordering read-sequence for local-bookie ensemble policy
Rajan Dhabalia [Mon, 28 Mar 2022 22:54:35 +0000 (15:54 -0700)] 
Fix NPE while reordering read-sequence for local-bookie ensemble policy

### Motivation

When Bookie sanity and autoreovery use the same conf file which has flag `reorderReadSequenceEnabled=true` then bookie-sanity command throws NPE as `LocalBookieEnsemblePlacementPolicy::reorderReadLACSequence` returns null writesets which causes the sanity failure.

```
00:46:46.202 [BookKeeperClientWorker-OrderedExecutor-11-0] ERROR o.a.b.common.util.SafeRunnable       - Unexpected throwable caught
java.lang.NullPointerException: null
at org.apache.bookkeeper.client.PendingReadOp$SequenceReadRequest.sendNextRead(PendingReadOp.java:399)
at org.apache.bookkeeper.client.PendingReadOp$SequenceReadRequest.read(PendingReadOp.java:385)
at org.apache.bookkeeper.client.PendingReadOp.initiate(PendingReadOp.java:529)
at org.apache.bookkeeper.client.LedgerRecoveryOp.doRecoveryRead(LedgerRecoveryOp.java:148)
at org.apache.bookkeeper.client.LedgerRecoveryOp.access$000(LedgerRecoveryOp.java:37)
at org.apache.bookkeeper.client.LedgerRecoveryOp$1.readLastConfirmedDataComplete(LedgerRecoveryOp.java:109)
at org.apache.bookkeeper.client.ReadLastConfirmedOp.readEntryComplete(ReadLastConfirmedOp.java:135)
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion$1.readEntryComplete(PerChannelBookieClient.java:1829)
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion.handleReadResponse(PerChannelBookieClient.java:1910)
at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion.handleV3Response(PerChannelBookieClient.java:1885)
at org.apache.bookkeeper.proto.PerChannelBookieClient$3.safeRun(PerChannelBookieClient.java:1446)
at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

```

### Modification
Fix NPE for local ensemble policy while reading entry with `reorderReadSequenceEnabled` flag enabled.

Reviewers: Andrey Yegorov <None>, Enrico Olivelli <eolivelli@gmail.com>, Nicolò Boschi <boschi1997@gmail.com>

This closes #3127 from rdhabalia/repl_seq

6 weeks agoFix doc code problem.
赵延 [Mon, 28 Mar 2022 22:23:14 +0000 (06:23 +0800)] 
Fix doc code problem.

fix docs code demo problem.

Reviewers: Nicolò Boschi <boschi1997@gmail.com>, Andrey Yegorov <None>

This closes #3142 from horizonzy/fix-docs-code-problem

6 weeks agoBookieAutoRecoveryTest.testEmptyLedgerLosesQuorumEventually fix flaky test, ensure...
Enrico Olivelli [Mon, 28 Mar 2022 15:53:04 +0000 (17:53 +0200)] 
BookieAutoRecoveryTest.testEmptyLedgerLosesQuorumEventually fix flaky test, ensure that the Auditor is alive (#3149)

7 weeks ago[security] Upgrade jackson-databind to get rid of CVE-2020-36518 (#3140)
Nicolò Boschi [Sun, 27 Mar 2022 12:37:16 +0000 (14:37 +0200)] 
[security] Upgrade jackson-databind to get rid of CVE-2020-36518 (#3140)

* [security] Upgrade jackson-databind to get rid of CVE-2020-36518

7 weeks agoUpgrade rocksdb to 6.29.4.1 (#3143)
Hang Chen [Sun, 27 Mar 2022 09:53:36 +0000 (17:53 +0800)] 
Upgrade rocksdb to 6.29.4.1 (#3143)

7 weeks agoBringing back maven build (#3130)
Andrey Yegorov [Thu, 24 Mar 2022 16:15:09 +0000 (09:15 -0700)] 
Bringing back maven build (#3130)

* Revert "[build] remove Maven POM files (#3009)"

This reverts commit e089b51ab5e1cf5f061c81463d27f33a21198271.

* rxjava: add maven dependency

(cherry picked from commit ac73541ce79953141a08c60642cd39c7984ade1e)

* Bring guava to the same version as gradle

* ignore deprecation warnings in tests

* mockito-inline, as in gradle + suppress warnings

* suppressed warning

* Exclude site3/ from RAT check

* CI to use (mostly) maven

* OWASP check with maven

* Up'd versions to match gradle, corrected license files: looks like gradle build didn't force versions consistently

* Removed current-version-image to match https://github.com/apache/bookkeeper/pull/3027

* Shading patetrn to match gradle

* Fixed/suppressed CVEs

* Attempt to fix failing tests in CompactionByEntriesWithMetadataCacheTest

Co-authored-by: lushiji <lushiji@didiglobal.com>
7 weeks agoPendingReadOp: Fix ledgerEntryImpl reuse problem (#3110)
congbo [Thu, 24 Mar 2022 08:22:02 +0000 (16:22 +0800)] 
PendingReadOp: Fix ledgerEntryImpl reuse problem (#3110)

7 weeks agoSet BOOKIE_HTTP_PORT to make it optional in docker run (#3096)
Kezhu Wang [Wed, 23 Mar 2022 08:01:53 +0000 (16:01 +0800)] 
Set BOOKIE_HTTP_PORT to make it optional in docker run (#3096)

Fixes #3075.

7 weeks agoIssue #3105: Optimize OrderedExecutor performance by using GrowableArrayBlockingQueue...
Lari Hotari [Wed, 23 Mar 2022 08:01:00 +0000 (10:01 +0200)] 
Issue #3105: Optimize OrderedExecutor performance by using GrowableArrayBlockingQueue (#3108)

Fixes #3105

7 weeks agoRemove unused site2 directory (#3116)
Nicolò Boschi [Wed, 23 Mar 2022 08:00:34 +0000 (09:00 +0100)] 
Remove unused site2 directory (#3116)

7 weeks agoLog NoLedgerException on debug level (#3117)
Andras Beni [Mon, 21 Mar 2022 15:06:01 +0000 (16:06 +0100)] 
Log NoLedgerException on debug level (#3117)

NoLedgerException does not signify an error in the Bookie that needs
to be fixed. Instead it is - at most - a user error that the user is
notified about via the status code ENOLEDGER.
Logging this problem at error level introduces an odd difference
between the behavior of readLac using v2 versus v3 protocol version.
In the former case ReadEntryProcessor logs the same problem at debug
level. As a result changing protocol version appers to be introducing
an error.

8 weeks ago[WEBSITE] Staging website: images are not visible
Nicolò Boschi [Fri, 18 Mar 2022 16:40:43 +0000 (17:40 +0100)] 
[WEBSITE] Staging website: images are not visible

### Motivation

In the staging website image links are broken

### Changes

* Fixed the links with the correct syntax (I checked every `img` reference)

Reviewers: Andrey Yegorov <None>

This closes #3126 from nicoloboschi/website/fix-imgs

8 weeks ago[website] fix GITHUB_TOKEN used for deployment (#3125)
Nicolò Boschi [Fri, 18 Mar 2022 10:43:42 +0000 (11:43 +0100)] 
[website] fix GITHUB_TOKEN used for deployment (#3125)

8 weeks ago[website] use GH_TOKEN to push to asf-site and asf-staging branches (#3124)
Nicolò Boschi [Fri, 18 Mar 2022 08:59:11 +0000 (09:59 +0100)] 
[website] use GH_TOKEN to push to asf-site and asf-staging branches (#3124)

8 weeks ago[website] Fix deploy push to git - use ssh (#3123)
Nicolò Boschi [Thu, 17 Mar 2022 16:27:50 +0000 (17:27 +0100)] 
[website] Fix deploy push to git - use ssh (#3123)

8 weeks ago[website] fix deploy staging script (#3122)
Nicolò Boschi [Thu, 17 Mar 2022 16:09:04 +0000 (17:09 +0100)] 
[website] fix deploy staging script (#3122)

8 weeks ago[website] fix current site deployment (#3120)
Nicolò Boschi [Thu, 17 Mar 2022 16:08:09 +0000 (17:08 +0100)] 
[website] fix current site deployment (#3120)

8 weeks ago[website] deploy staging after every change (#3118)
Nicolò Boschi [Thu, 17 Mar 2022 11:40:36 +0000 (12:40 +0100)] 
[website] deploy staging after every change (#3118)

8 weeks agoGradle build: add mavenLocal() repository
Enrico Olivelli [Wed, 16 Mar 2022 18:15:49 +0000 (19:15 +0100)] 
Gradle build: add mavenLocal() repository

### Motivation

In Gradle you need to add `mavenLocal()` repository if you want to test local versions of third party libraries built with Maven (like ZooKeeper, Curator...)

### Changes
Add `mavenLocal()`  repository

Reviewers: Matteo Merli <mmerli@apache.org>, Nicolò Boschi <boschi1997@gmail.com>, Andrey Yegorov <None>

This closes #3114 from eolivelli/impl/gradle-maven-local

8 weeks ago[website] New Website built with Docusaurus v2 (#3088)
Nicolò Boschi [Wed, 16 Mar 2022 16:06:24 +0000 (17:06 +0100)] 
[website] New Website built with Docusaurus v2 (#3088)

8 weeks agoadd stats for throttled-write (#3102)
StevenLuMT [Wed, 16 Mar 2022 10:10:59 +0000 (18:10 +0800)] 
add stats for throttled-write (#3102)

Descriptions of the changes in this PR:

### Motivation

method:triggerFlushAndAddEntry costing time is a changing,so add a stats metric focus on this method

### Changes

1.the previous counter metrics(throttledWriteRequests) are retained
2.add throttledWriteStats to record cost time and count for the method(triggerFlushAndAddEntry)