bookkeeper.git
28 hours agoFix the Intellij key nouns. (#3381) master
liuzhuang2017 [Sat, 2 Jul 2022 06:02:07 +0000 (14:02 +0800)] 
Fix the Intellij key nouns. (#3381)

2 days agoadd and check indexDirs in Cookie meta (#3372)
StevenLuMT [Fri, 1 Jul 2022 06:13:46 +0000 (14:13 +0800)] 
add and check indexDirs in Cookie meta (#3372)

Co-authored-by: lushiji <lushiji@didiglobal.com>
2 days agouse estimateMaxDirectMemory instead of maxDirectMemory (#3375)
Hang Chen [Thu, 30 Jun 2022 15:19:53 +0000 (23:19 +0800)] 
use estimateMaxDirectMemory instead of maxDirectMemory (#3375)

### Motivation
https://github.com/apache/bookkeeper/pull/2989

### Modification
Use estimateMaxDirectMemory instead of maxDirectMemory

3 days agoFix a typo in bk_server.conf (#3370)
ZhangJian He [Thu, 30 Jun 2022 06:02:23 +0000 (14:02 +0800)] 
Fix a typo in bk_server.conf (#3370)

4 days agoconvert testcase to junit 4 (#3371)
StevenLuMT [Tue, 28 Jun 2022 17:58:39 +0000 (01:58 +0800)] 
convert testcase to junit 4 (#3371)

* Basic function: directories to store index files

* add test case for indexDirs fix

* checkstyle fixed

* use junit 5 api to run testcase

* convert this testcase to junit 4

Co-authored-by: lushiji <lushiji@didiglobal.com>
4 days agoBasic function: directories to store index files (#3352)
StevenLuMT [Tue, 28 Jun 2022 11:23:06 +0000 (19:23 +0800)] 
Basic function: directories to store index files (#3352)

Co-authored-by: lushiji <lushiji@didiglobal.com>
5 days agoFix bookie CI test not run (#3367)
Hang Chen [Tue, 28 Jun 2022 01:02:19 +0000 (09:02 +0800)] 
Fix bookie CI test not run (#3367)

### Motivation
The `bookie-tests` CI runs the following tests.
```
- name: Run bookie tests
        run: mvn -B -nsu -pl bookkeeper-server test -Dtest="org.apache.bookkeeper.bookie.*Test" -DfailIfNoTests=false -Dorg.slf4j.simpleLogger.defaultLogLevel=INFO
```

For the `remaining-tests` CI runs the following tests.
```bash
- name: Run remaining tests
        run: mvn -B -nsu -am -pl bookkeeper-server clean install test -Dtest="!org.apache.bookkeeper.client.**,!org.apache.bookkeeper.bookie.**,!org.apache.bookkeeper.replication.**,!org.apache.bookkeeper.tls.**" -DfailIfNoTests=false -Dorg.slf4j.simpleLogger.defaultLogLevel=INFO
```

The problem is that `bookie-tests` only run tests with the name `org.apache.bookkeeper.bookie.*Test`, however, the `remaining-tests` exclude tests with name `org.apache.bookkeeper.bookie.**`. It leads to a lot of tests in `org.apache.bookkeeper.bookie.*` package doesn't run at all in CI.

### Modification
Turn on `org.apache.bookkeeper.bookie.**` tests in `bookie-tests` CI.

I have been testing on my laptop, there are many tests can't pass, I may need more time to make those failed tests to be passed.

The following tests need to fixed.
- [x] org.apache.bookkeeper.bookie.storage.ldb.ConversionTest.test
- [x] org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageTest.testLimboStateThrowsInLimboWhenNoEntry
- [x] org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageTest.testLimboStateSucceedsWhenFenced
- [x] org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageTest.testLimboStateSucceedsWhenInLimboButHasEntry
- [x] org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageTest.testLimboStateThrowsInLimboWhenNotFenced
- [x] org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorageTest.testStorageStateFlags

10 days agoBP-47 (task6): Direct I/O entrylogger support (#3263)
Hang Chen [Thu, 23 Jun 2022 02:19:15 +0000 (10:19 +0800)] 
BP-47 (task6): Direct I/O entrylogger support (#3263)

* Direct I/O entry log support

The implementation uses JNI to do direct I/O to files via posix
syscalls. Fallocate is used if running on linux, otherwise this is
skipped (at the cost of more filesystem operates during writing).

There are two calls to write, writeAt and writeDelimited. I expect
writeAt to be used for the entrylog headers, which entries will go
through writeDelimited. In both cases, the calls may return before the
syscalls occur. #flush() needs to be called to ensure things are
actually written.

The entry log format isn't much changed from what is used by the
existing entrylogger. The biggest difference is the padding. Direct
I/O must write in aligned blocked. The size of the alignment varies by
machine configuration, but 4K is a safe bet on most. As it is unlikely
that entry data will land exactly on the alignment boundary, we need
to add padding to writes. The existing entry logger has been changed
to take this padding into account. When read as a signed int/long/byte
the padding will aways parse to a negative value, which distinguishes
it from valid entry data (the entry size will always be positive) and
also from preallocated space (which is always 0).

Another difference in the format is that the header is now 4K rather
than 1K. Again, this is to allow aligned rights. No changes are
necessary to allow the existing entry logger to deal with the header
change, as we create a dummy entry in the extra header space that the
existing entry logger already knows to ignore.

To enable, set dbStorage_directIOEntryLogger=true in the configuration.

(cherry picked from commit 09a1c81b6894d4398c5a59e54cc18784550badb5)

* format code

* format code

* fix spotbugs check

* fix jnilib not found

* fix jnilib not found

* address comments

* fix so lib not found

* address comments

* format code

* add compat test

* add compat test

* address comments

* fix findbugs failed

* format code

Co-authored-by: Ivan Kelly <ikelly@splunk.com>
10 days agoupgrade groovy from 2.5.17 to 3.0.11 to fix CVE-2019-11358(7.5) (#3346)
Hang Chen [Wed, 22 Jun 2022 13:09:40 +0000 (21:09 +0800)] 
upgrade groovy from 2.5.17 to 3.0.11 to fix CVE-2019-11358(7.5) (#3346)

12 days agoAutorecovery to rereplicate empty ledgers (#3239)
Andrey Yegorov [Tue, 21 Jun 2022 09:44:38 +0000 (02:44 -0700)] 
Autorecovery to rereplicate empty ledgers (#3239)

13 days agoreduce unnecessary checkpoints (#3341)
LinChen [Mon, 20 Jun 2022 07:01:47 +0000 (15:01 +0800)] 
reduce unnecessary checkpoints (#3341)

13 days agofix: broken link of code guide in README (#3347)
Max Xu [Mon, 20 Jun 2022 06:43:21 +0000 (14:43 +0800)] 
fix: broken link of code guide in README (#3347)

13 days agoallow build on jdk17 (#3343)
ZhangJian He [Sun, 19 Jun 2022 18:33:02 +0000 (02:33 +0800)] 
allow build on jdk17 (#3343)

13 days agoRemove unused value assign in BookKeeperTestClient (#3331)
LeKe [Sun, 19 Jun 2022 14:54:05 +0000 (22:54 +0800)] 
Remove unused value assign in BookKeeperTestClient (#3331)

2 weeks agoRemove gradle build script (#3340)
ZhangJian He [Fri, 17 Jun 2022 00:39:28 +0000 (08:39 +0800)] 
Remove gradle build script (#3340)

### Motivation
Remove gradle build script
See
https://lists.apache.org/thread/k3pdlpbvkzt061338fkxtppq3z70nk03
https://lists.apache.org/thread/os28qm4t0kf74q2k3lzsmnnpr0lnm6zc
### Changes
- delete gradle build script
- add `netty-buffer` dependency on module `distributedlog-core-shaded-test` to fix broken master tests.

2 weeks agoRemove gradle from workflow (#3314)
ZhangJian He [Thu, 16 Jun 2022 08:11:58 +0000 (16:11 +0800)] 
Remove gradle from workflow (#3314)

* Remove gradle from workflow

* add dep

2 weeks agoApply the backpressure changes on the V2 requests (#3324)
Yong Zhang [Thu, 16 Jun 2022 00:17:19 +0000 (08:17 +0800)] 
Apply the backpressure changes on the V2 requests (#3324)

---

*Motivation*

If one bookie is slow (not down, just slow), the BK client
will the acks to the user that the entries are written after
the first 2 acks. In the meantime, it will keep waiting for
the 3rd bookie to respond. If the bookie responds within the
timeout, the entries can now be dropped from memory, otherwise
the write will timeout internally and it will get replayed
to a new bookie.

In the V3 request, we have [server-side backpressure](https://github.com/apache/bookkeeper/pull/1410)
to impact the client-side behaviors. We should apply the same
changes to the V2 request. That would help this [issue](https://github.com/apache/pulsar/issues/14861)
to be resolved.

*Modification*

- Apply the change https://github.com/apache/bookkeeper/pull/1410 to V2 protocol

Descriptions of the changes in this PR:

2 weeks ago Fix the V2 AddRequest object leak issue (#3323)
Yong Zhang [Mon, 13 Jun 2022 07:27:51 +0000 (15:27 +0800)] 
 Fix the V2 AddRequest object leak issue (#3323)

---

**Motivation**

If the request is a V2 add request, we retained the data's
reference when creating the AddRequest object. To avoid the
object leak, we need to release the reference if we met
any errors before sending it.

3 weeks agoAdd Pravega as a Bookkeeper use case. (#3328)
Raúl Gracia [Fri, 10 Jun 2022 09:30:49 +0000 (11:30 +0200)] 
Add Pravega as a Bookkeeper use case. (#3328)

3 weeks ago[Client] Deduplicate error log for SSLException (#3320)
Michael Marshall [Thu, 9 Jun 2022 05:08:07 +0000 (00:08 -0500)] 
[Client] Deduplicate error log for SSLException (#3320)

### Motivation

While testing #3310, I noticed that the `PerChannelBookieClient#exceptionCaught` logic contains redundant logs when the exception is an `SSLException`. This PR removes a redundant log from client.

Based on reading through the rest of the method, this should be a trivial change with no other side effects. My one question is how closing a channel and closing the context differ. Technically, returning early skips the `ctx.close()`, which could change the behavior.

### Changes

* Return early to prevent a redundant log

3 weeks agoFix the PendingAddOp is not recycled when LedgerHandler closed (#3321)
Yong Zhang [Wed, 8 Jun 2022 10:02:32 +0000 (18:02 +0800)] 
Fix the PendingAddOp is not recycled when LedgerHandler closed (#3321)

---

**Motivation**

When adding an entry into a bookie, the entry data lifecycle
is handled by the bookie client. The data buffer will be
released after receiving a corresponding response from the
bookie server. So the user doesn't care about the entry
buffer releasing.
But when the ledgerHandler is closed, the PendingAddOp is not
recycled which leads to the data buffer never being released.
We should release that after the callback executed.

3 weeks agoTry to use Java9 CRC32C when JNI based CRC is not available (#3309)
Matteo Merli [Wed, 8 Jun 2022 01:26:50 +0000 (18:26 -0700)] 
Try to use Java9 CRC32C when JNI based CRC is not available (#3309)

* Try to use Java9 CRC32C when JNI based CRC is not available

* Fixed shading test dependency

* Fixed dlog shading deps

3 weeks agoEnable maven batch mode to reduce log in mac and windows workflow (#3318)
TingTing Wang [Tue, 7 Jun 2022 12:51:53 +0000 (20:51 +0800)] 
Enable maven batch mode to reduce log in mac and windows workflow (#3318)

3 weeks agoFix maven javadoc generate (#3317)
ZhangJian He [Tue, 7 Jun 2022 10:54:02 +0000 (18:54 +0800)] 
Fix maven javadoc generate (#3317)

### Motivation
Currently, the maven javadoc generate is broken both on master and branch-4.14

### Changes
- fix the javadoc generate, the gradle script has already use `delombok`
- delete unneeded `package-info` file
- delete unused dependency `spotbugs-annotations`, it's not worth to open other pr(IMO)

3 weeks agoConsider consider Bookie ID when validating the Cookie. (#3308)
Raúl Gracia [Tue, 7 Jun 2022 10:53:54 +0000 (12:53 +0200)] 
Consider consider Bookie ID when validating the Cookie. (#3308)

Signed-off-by: Raúl Gracia <raul.gracia@emc.com>
3 weeks agoadd missing link to paxos algorithm (#3298)
Vladislav Khakin [Tue, 7 Jun 2022 03:14:34 +0000 (06:14 +0300)] 
add missing link to paxos algorithm (#3298)

Fixes #3293

3 weeks agoBump apache version from 19 to 26 (#3315)
ZhangJian He [Mon, 6 Jun 2022 07:07:51 +0000 (15:07 +0800)] 
Bump apache version from 19 to 26 (#3315)

3 weeks agoClose journal channel in testJunkEndedJournal (#3307)
ZhangJian He [Mon, 6 Jun 2022 05:41:47 +0000 (13:41 +0800)] 
Close journal channel in testJunkEndedJournal (#3307)

4 weeks agoBump grpc from 1.45.1 to 1.47.0, solve dependency check FP (#3305)
ZhangJian He [Sun, 5 Jun 2022 07:29:26 +0000 (15:29 +0800)] 
Bump grpc from 1.45.1 to 1.47.0, solve dependency check FP (#3305)

* Bump grpc from 1.45.1 to 1.47.0

* add dependency check suppression

4 weeks ago[Client] Switch to rely on Netty for Hostname Verification (#3310)
Michael Marshall [Sun, 5 Jun 2022 06:39:35 +0000 (01:39 -0500)] 
[Client] Switch to rely on Netty for Hostname Verification (#3310)

### Motivation

Currently, we initiate hostname verification for the Bookkeeper Client in the `PerChannelBookieClient` class. In order to simplify the code, I propose that we refactor the client so it relies on Netty, its SslHandler/SslEngine, and the JVM, to perform the hostname verification.

When HTTPS is configured as the endpoint verification algorithm, it uses [RFC 2818](https://datatracker.ietf.org/doc/html/rfc2818) to perform hostname verification. This is defined by the Java Security Standard Algorithm Names documentation for JDK versions 8, 11, and 17. Here are the official docs:

* https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html
* https://docs.oracle.com/en/java/javase/11/docs/specs/security/standard-names.html
* https://docs.oracle.com/en/java/javase/17/docs/specs/security/standard-names.html

### Changes

* Rely on Netty and the SslEngine to perform hostname verification. With this change, CN matching is now deprecated, which brings the bookkeeper client in alignment with RFC 2818.
* Add new method to the `SecurityHandlerFactory` interface. It is named `newTLSHandler` and takes the `host` and `port` of the remote peer when creating a new SslEngine. To ensure backwards compatibility, the default implementation will call the original method. Note that the remote host and port are only needed when a client is using them for hostname verification.

4 weeks agoClean up javadoc mojos execution (#2906)
Fangbin Sun [Fri, 3 Jun 2022 05:23:06 +0000 (13:23 +0800)] 
Clean up javadoc mojos execution (#2906)

Descriptions of the changes in this PR:

### Motivation

Fixes #1514.

### Changes

1. Configuring maven-javadoc-plugin with `detectOfflineLinks=false` eliminates the Maven invocations (refer to: [https://issues.redhat.com/browse/ISPN-8629](https://issues.redhat.com/browse/ISPN-8629)).
2. Use option `notimestamp` to replace `additionalparam` since the latter is not allowed from version `3.x`.

4 weeks ago[build] Support windows build (#3303)
ZhangJian He [Thu, 2 Jun 2022 15:05:22 +0000 (23:05 +0800)] 
[build] Support windows build (#3303)

### Motivation
Support windows build to let more people easily contributed in this project. Test on my windows10 and windows11 PC.
I think that Windows now can be used for development, but not for production.
### Changes
- add some win32 compat code
- add windows build check
### owasp ci failure
both `google-http-client-gson` and `maven-settings` are not related to this PR

4 weeks ago[ci] [build] add macos build check (#3301)
ZhangJian He [Wed, 1 Jun 2022 11:09:14 +0000 (19:09 +0800)] 
[ci] [build] add macos build check (#3301)

4 weeks ago[build] [windows] fix broken master code compile error strerror_s (#3300)
ZhangJian He [Tue, 31 May 2022 02:06:21 +0000 (10:06 +0800)] 
[build] [windows] fix broken master code compile error strerror_s (#3300)

### Motivation
master compile is broken by #2549

### Changes

Add windows `#ifdef _WIN32` directive on `define`. Test on my m1 mac and windows 10.

4 weeks agoFix javadoc and build warnings (#3297)
Vladislav Khakin [Mon, 30 May 2022 23:47:45 +0000 (02:47 +0300)] 
Fix javadoc and build warnings (#3297)

Fix #472 #2155

4 weeks ago[build] support circe-checksum and cpu-affinity build on Windows (#2549)
magiccao [Mon, 30 May 2022 11:03:38 +0000 (19:03 +0800)] 
[build] support circe-checksum and cpu-affinity build on Windows (#2549)

Co-authored-by: dongcao <dongcao@trip.com>
4 weeks ago[build] misc bump several maven plugin version (#3294)
ZhangJian He [Mon, 30 May 2022 06:21:55 +0000 (14:21 +0800)] 
[build] misc bump several maven plugin version (#3294)

### Motivation
Keep plugin version update. I am trying to make bookkeeper can run on windows(more developers can join this project). I believe update some maven plugin versions can help a little.

### Changes
- bump jacoco maven plugin from 0.8.0 to 0.8.8
- bump maven-clean-plugin from 2.5 to 3.2.0
- bump maven-compiler version to 3.10.1
- bump maven-jar-plugin from 2.4 to 3.2.2
- bump nar-maven-plugin from 3.5.2 to 3.10.1
- delete a unnecessary dependency, it's already defined in parent pom

### ci failure
both `google-http-client-gson` and `maven-settings` are not related to this PR

5 weeks agoFixed build instruction to Maven since Gradle usage was reverted (#3296)
Asaf Mesika [Sun, 29 May 2022 09:46:22 +0000 (12:46 +0300)] 
Fixed build instruction to Maven since Gradle usage was reverted (#3296)

### Motivation

Getting started instructions for building the project were wrong since Gradle is not used anymore. I reverted to the old Maven one

5 weeks ago[website] Make download page fully ASF compliant (#3291)
Nicolò Boschi [Wed, 25 May 2022 13:01:07 +0000 (15:01 +0200)] 
[website] Make download page fully ASF compliant (#3291)

6 weeks agoBump netty version to 4.1.77.Final (#3273)
ZhangJian He [Thu, 19 May 2022 14:55:34 +0000 (22:55 +0800)] 
Bump netty version to 4.1.77.Final (#3273)

6 weeks ago[doc] Improve release guide (#3277)
Nicolò Boschi [Thu, 19 May 2022 14:54:21 +0000 (16:54 +0200)] 
[doc] Improve release guide (#3277)

* Improve release guide

* also fix website validation action

6 weeks ago[doc] Remove DC/OS installation guide (#3278)
Nicolò Boschi [Thu, 19 May 2022 14:53:36 +0000 (16:53 +0200)] 
[doc] Remove DC/OS installation guide (#3278)

* [doc] Remove DC/OS installation guide

* remove from release guide

6 weeks ago[doc] Remove old site source code (#3281)
Nicolò Boschi [Mon, 16 May 2022 13:56:24 +0000 (15:56 +0200)] 
[doc] Remove old site source code (#3281)

7 weeks agoUpdated organization to Streamlio former members (#3280)
AlvaroStream [Fri, 13 May 2022 16:03:07 +0000 (18:03 +0200)] 
Updated organization to Streamlio former members (#3280)

Update the organization to former Streamlio members to StreamNative

7 weeks ago4.14.5 release notes and doc (#3199)
Nicolò Boschi [Fri, 13 May 2022 08:03:35 +0000 (10:03 +0200)] 
4.14.5 release notes and doc (#3199)

7 weeks ago[website] Add script for cut new release (release-notes, new docs, sidebar..) (#3198)
Nicolò Boschi [Fri, 13 May 2022 07:50:22 +0000 (09:50 +0200)] 
[website] Add script for cut new release (release-notes, new docs, sidebar..) (#3198)

7 weeks ago[website] fix hyperlink (#3276)
赤月 [Thu, 12 May 2022 12:48:28 +0000 (20:48 +0800)] 
[website] fix hyperlink (#3276)

Co-authored-by: zwen <zwen@didiglobal.com>
7 weeks agoBump prometheus client version to 0.15.0 (#3266)
ZhangJian He [Wed, 11 May 2022 23:50:13 +0000 (07:50 +0800)] 
Bump prometheus client version to 0.15.0 (#3266)

### Motivation
prometheus client `0.8.1` is release on Jan,2020, Keep update

### Changes
Bump prometheus client version from 0.8.1 to 0.15.0

### Test
Update prometheus client to 0.15.0, compare the metrics txt, the results are same.

7 weeks agoEnable Netty and BookKeeper IO optimizations on jdk17 (#3234)
Nicolò Boschi [Wed, 11 May 2022 17:17:44 +0000 (19:17 +0200)] 
Enable Netty and BookKeeper IO optimizations on jdk17 (#3234)

* Enable netty optimization and BK native IO on jdk17

* zip and better docs

7 weeks agoFix link Apache Pulsar#persistent-storage (#3271)
赤月 [Wed, 11 May 2022 11:38:32 +0000 (19:38 +0800)] 
Fix link Apache Pulsar#persistent-storage  (#3271)

7 weeks agocleaning up leftover 4.15-snapshot versions (#3267)
Andrey Yegorov [Tue, 10 May 2022 19:13:33 +0000 (12:13 -0700)] 
cleaning up leftover 4.15-snapshot versions (#3267)

* cleaning up leftover 4.15-snapshot versions

* updated news page

7 weeks ago[website] Set 4.15.0 as latest release (#3265)
Nicolò Boschi [Tue, 10 May 2022 06:58:36 +0000 (08:58 +0200)] 
[website] Set 4.15.0 as latest release (#3265)

7 weeks agoRelease notes for BK 4.15
Andrey Yegorov [Mon, 9 May 2022 18:51:58 +0000 (11:51 -0700)] 
Release notes for BK 4.15

Descriptions of the changes in this PR:
### Motivation

release notes for BK release 4.15

### Changes

Doc updates per release procedure.

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Yong Zhang <zhangyong1025.zy@gmail.com>, Hang Chen <chenhang@apache.org>, Nicolò Boschi <boschi1997@gmail.com>

This closes #3155 from dlg99/bk-4.15-release-notes

8 weeks agoFix spelling mistake in bk_server.conf (#3262)
Tian Luo [Sat, 7 May 2022 12:11:38 +0000 (20:11 +0800)] 
Fix spelling mistake in bk_server.conf (#3262)

### Motivation

Fix spelling mistake in bk_server.conf

### Changes

Master Issue: #3261

8 weeks agoBP-47 (task5): Garbage collection support direct IO entrylogger (#3256)
Hang Chen [Thu, 5 May 2022 06:29:11 +0000 (14:29 +0800)] 
BP-47 (task5): Garbage collection support direct IO entrylogger (#3256)

* Garbage Collection support for DirectIO entrylogger

GC support requires that the entrylogger provides a way to retrieve
all entrylogs which have been completely flushed to disk. Previously
this was done by returning the least unflushed log id. However, this
is problematic as it doesn't support the log ids wrapping around. It
also means that GC has to start checking for log id existence from
zero every time it boots.

This change replaces getLeastUnflushedLogId() with getFlushedLogIds(),
to give the entrylogger full control of which logs should be
considered for GC.

It also changes the CompactableLedgerStorage interface, removing
getEntryLogger() and adding injection of the entrylogger to the
GarbageCollectionThread. This makes testing easier.

Co-authored-by: Ivan Kelly <ikelly@splunk.com>
8 weeks ago[minor] [optimize] Remove redundant toString call (#3254)
ZhangJian He [Thu, 5 May 2022 00:26:43 +0000 (08:26 +0800)] 
[minor] [optimize] Remove redundant toString call (#3254)

8 weeks agoAllow run LocalBookkeeper directly in `bookkeeper-server` module or on IDE (#3255)
ZhangJian He [Wed, 4 May 2022 13:20:11 +0000 (21:20 +0800)] 
Allow run LocalBookkeeper directly in `bookkeeper-server` module or on IDE (#3255)

8 weeks agoBP-47 (task4): Aligned native buffer wrapper (#3253)
Hang Chen [Wed, 4 May 2022 06:18:23 +0000 (14:18 +0800)] 
BP-47 (task4): Aligned native buffer wrapper (#3253)

2 months agoFix: shell tests are broken and not running on CI (#3251)
Andrey Yegorov [Fri, 29 Apr 2022 16:10:42 +0000 (09:10 -0700)] 
Fix: shell tests are broken and not running on CI (#3251)

2 months agoBP-47 (task3): Abstract interface for entrylogger (#3197)
Hang Chen [Fri, 29 Apr 2022 05:44:14 +0000 (13:44 +0800)] 
BP-47 (task3): Abstract interface for entrylogger (#3197)

* Interface for entrylogger

Define the interface and contract for entrylogger. This is mostly
taking the entrylogger methods used by other components and prettying
them up a bit.

Notable changes are:
- internalReadEntry is now readEntry. There is no 'validate'
  flag. Instead there are two overloads for the method, and validation
  only runs if ledgerId and entryId are passed in.
- shutdown has been renamed to close.
- the compaction entrylog methods have been put behind an
  interface. As it was they were leaking implementation
  details. Ultimitely compaction itself should be hidden behind the
  entrylogger, but that's a larger refactor.

(cherry picked from commit c927f4b0128e2b1f0f4b30c41ce442b366d5009c)

* abstract interface for entrylogger

* format code

* format code

* address comments

* Apply suggestions from code review

Co-authored-by: Andrey Yegorov <8622884+dlg99@users.noreply.github.com>
* address comments

* format code

Co-authored-by: Ivan Kelly <ikelly@splunk.com>
Co-authored-by: Andrey Yegorov <8622884+dlg99@users.noreply.github.com>
2 months agoMove bookkeeper-stats and bookkeeper-stats-providers as submodules of stats
ZhangJian He [Thu, 28 Apr 2022 23:00:53 +0000 (07:00 +0800)] 
Move bookkeeper-stats and bookkeeper-stats-providers as submodules of stats

### Motiviations
Fix todo to make the project layout more reasonable and easier to fix bug/dev new feature
### Changes
move `bookkeeper-stats` and `bookkeeper-stats-providers` as submodules of `stats`

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Nicolò Boschi <boschi1997@gmail.com>, Andrey Yegorov <None>

This closes #3246 from Shoothzj/module-04-25

2 months agofix some jdk17 test problem (#3242)
ZhangJian He [Wed, 27 Apr 2022 07:09:35 +0000 (15:09 +0800)] 
fix some jdk17 test problem (#3242)

2 months ago[maven-release-plugin] prepare for next development iteration
Andrey Yegorov [Tue, 26 Apr 2022 21:48:53 +0000 (14:48 -0700)] 
[maven-release-plugin] prepare for next development iteration

2 months ago[maven-release-plugin] prepare branch branch-4.15
Andrey Yegorov [Tue, 26 Apr 2022 21:48:51 +0000 (14:48 -0700)] 
[maven-release-plugin] prepare branch branch-4.15

2 months agoRevert "[maven-release-plugin] prepare branch branch-4.15"
Andrey Yegorov [Tue, 26 Apr 2022 21:47:58 +0000 (14:47 -0700)] 
Revert "[maven-release-plugin] prepare branch branch-4.15"

This reverts commit d9b98ad8ccec6a9ba65cfc00ef319ee51454c0c4.

2 months agoRevert "[maven-release-plugin] prepare for next development iteration"
Andrey Yegorov [Tue, 26 Apr 2022 21:47:48 +0000 (14:47 -0700)] 
Revert "[maven-release-plugin] prepare for next development iteration"

This reverts commit e5d5ddac8a10acca031eed0221d925d54713a6ab.

2 months ago[maven-release-plugin] prepare for next development iteration
Andrey Yegorov [Tue, 26 Apr 2022 21:31:55 +0000 (14:31 -0700)] 
[maven-release-plugin] prepare for next development iteration

2 months ago[maven-release-plugin] prepare branch branch-4.15
Andrey Yegorov [Tue, 26 Apr 2022 21:31:52 +0000 (14:31 -0700)] 
[maven-release-plugin] prepare branch branch-4.15

2 months agoAdd get under replicated ledger count
Hang Chen [Tue, 26 Apr 2022 06:17:37 +0000 (14:17 +0800)] 
Add get under replicated ledger count

### Motivation
Fix #3224

### Changes
Add flag for `ListUnderreplicatedCmd` to control only display under replicated ledgers count

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Yong Zhang <zhangyong1025.zy@gmail.com>, Andrey Yegorov <None>

This closes #3228 from hangc0276/chenhang/add_get_under_replicate_ledger_count

2 months agoCorrect the method signature within annotations
iamazy [Tue, 26 Apr 2022 01:14:46 +0000 (09:14 +0800)] 
Correct the method signature within annotations

Descriptions of the changes in this PR:

### Motivation

Some interfaces have modified the method signature but the annotations have not.

### Changes

Correct the method signature within annotations and fix typos

Reviewers: Andrey Yegorov <None>, ZhangJian He <shoothzj@gmail.com>

This closes #3201 from iamazy/fix-typo

2 months agoFix bookie shell help command exception
Hang Chen [Mon, 25 Apr 2022 22:35:16 +0000 (06:35 +0800)] 
Fix bookie shell help command exception

### Motivation
When run `bin/bookkeeper shell ledgermetadata -h` command, it throw the following exception.
```
23:02:41,236 ERROR Error parsing command line arguments :
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: -h
at org.apache.commons.cli.Parser.processOption(Parser.java:363) ~[commons-cli-commons-cli-1.2.jar:1.2]
at org.apache.commons.cli.Parser.parse(Parser.java:199) ~[commons-cli-commons-cli-1.2.jar:1.2]
at org.apache.commons.cli.Parser.parse(Parser.java:85) ~[commons-cli-commons-cli-1.2.jar:1.2]
at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:240) ~[org.apache.bookkeeper-bookkeeper-server-4.16.0-SNAPSHOT.jar:4.16.0-SNAPSHOT]
at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:2372) ~[org.apache.bookkeeper-bookkeeper-server-4.16.0-SNAPSHOT.jar:4.16.0-SNAPSHOT]
at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:2463) ~[org.apache.bookkeeper-bookkeeper-server-4.16.0-SNAPSHOT.jar:4.16.0-SNAPSHOT]
ledgermetadata: Print the metadata for a ledger, or optionally dump to a file.
usage: ledgermetadata -ledgerid <ledgerid> [--dump-to-file
                      FILENAME|--restore-from-file FILENAME]
 -dumptofile <arg>        Dump metadata for ledger, to a file
 -l,--ledgerid <arg>      Ledger ID
 -restorefromfile <arg>   Restore metadata for ledger, from a file
```

### Changes
Add `-h` and `--help` support.

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, ZhangJian He <shoothzj@gmail.com>, Andrey Yegorov <None>

This closes #3227 from hangc0276/chenhang/fix_bookieshell_help_exception

2 months agoFormat log statements
Hang Chen [Mon, 25 Apr 2022 22:32:20 +0000 (06:32 +0800)] 
Format log statements

### Motivation
In PR #2455 , It convert all `System.*.print` statements to `LOG.* statements`. However, in `LOG.*` statements, it uses string concatenation to print messages, which is low performance.

### Changes
Change the string concatenation format to `LOG.*` standard statements

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Andrey Yegorov <None>, ZhangJian He <shoothzj@gmail.com>, Nicolò Boschi <boschi1997@gmail.com>, Yong Zhang <zhangyong1025.zy@gmail.com>

This closes #3213 from hangc0276/chenhang/format_log_info

2 months agoFix read empty ledger exception using bookkeeper admin
Hang Chen [Mon, 25 Apr 2022 22:28:04 +0000 (06:28 +0800)] 
Fix read empty ledger exception using bookkeeper admin

### Motivation
Fix #3222

The root cause is that BookKeeperAdmin use `asyncReadEntriesInternal` to read entries. However, the `asyncReadEntriesInternal` doesn't check the lastAddConfirm, which will lead to the exception list in issue 3222

### Changes

Add lastAddConfirm check before call `asyncReadEntriesInternal`

Reviewers: Yong Zhang <zhangyong1025.zy@gmail.com>, Andrey Yegorov <None>

This closes #3240 from hangc0276/chenhang/fix_BKBookieHandleNotAvailableException_for_empty_ledger

2 months agoBump jetty to 9.4.46.v20220331 to get rid of CVE-2021-34429
Nicolò Boschi [Mon, 25 Apr 2022 22:25:35 +0000 (00:25 +0200)] 
Bump jetty to 9.4.46.v20220331 to get rid of CVE-2021-34429

### Motivation

Current jetty version is vulnerable to https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-34429

### Changes

* Upgrade to the latest 9.x one 9.4.46.v20220331

Reviewers: Andrey Yegorov <None>, ZhangJian He <shoothzj@gmail.com>, Tian Luo <futurer@outlook.com>, Yong Zhang <zhangyong1025.zy@gmail.com>

This closes #3232 from nicoloboschi/bump-jetty-9.4.46.v20220331

2 months ago[ci] [github] upload test log when fail (#3241)
ZhangJian He [Sun, 24 Apr 2022 10:42:21 +0000 (18:42 +0800)] 
[ci] [github] upload test log when fail (#3241)

2 months agoIssue 2518:fix error link (#3217)
leiwingqueen [Sun, 24 Apr 2022 02:52:45 +0000 (10:52 +0800)] 
Issue 2518:fix error link (#3217)

### Motivation

(Explain: why you're making that change, what is the problem you're trying to solve)
[issue 2518](https://github.com/apache/bookkeeper/issues/2518)

### Changes

just try to fix the error link.

2 months ago[build] make build passes on JDK17 (#3053)
Nicolò Boschi [Fri, 22 Apr 2022 16:32:11 +0000 (18:32 +0200)] 
[build] make build passes on JDK17 (#3053)

2 months ago[minor] cleanup on InterleavedLedgerStorage (#3210)
ZhangJian He [Fri, 22 Apr 2022 16:10:59 +0000 (00:10 +0800)] 
[minor] cleanup on InterleavedLedgerStorage (#3210)

2 months agoRemove log4j from bookkeeper (#3225)
ZhangJian He [Fri, 22 Apr 2022 16:10:18 +0000 (00:10 +0800)] 
Remove log4j from bookkeeper (#3225)

2 months agobookeeper all missing stream binary (#3238)
ZhangJian He [Fri, 22 Apr 2022 16:06:17 +0000 (00:06 +0800)] 
bookeeper all missing stream binary (#3238)

2 months agofix logger number not correct (#3211)
ZhangJian He [Fri, 22 Apr 2022 01:00:01 +0000 (09:00 +0800)] 
fix logger number not correct (#3211)

### Motivation

fix logger number not correct

### Changes

fix logger number not correct

2 months ago[minor] replace deprecated api use MockitoAnnotations.initMocks (#3177)
ZhangJian He [Thu, 21 Apr 2022 11:42:58 +0000 (19:42 +0800)] 
[minor] replace deprecated api use MockitoAnnotations.initMocks (#3177)

2 months agofix broken local bookie (#3230)
ZhangJian He [Thu, 21 Apr 2022 11:42:33 +0000 (19:42 +0800)] 
fix broken local bookie (#3230)

2 months ago[ZooKeeper] [Conflict] Bump dropwizard metrics to zookeeper needed (#3229)
ZhangJian He [Thu, 21 Apr 2022 07:50:09 +0000 (15:50 +0800)] 
[ZooKeeper] [Conflict] Bump dropwizard metrics to zookeeper needed (#3229)

2 months agofix cli command doc issue (#3226)
Hang Chen [Thu, 21 Apr 2022 00:27:08 +0000 (08:27 +0800)] 
fix cli command doc issue (#3226)

### Motivation
Fix #3221

### Changes
Fix some cli command doc issue, and keep sync with code.

2 months agoBump jackson databind version to 2.13.2.2
ZhangJian He [Wed, 20 Apr 2022 14:18:20 +0000 (22:18 +0800)] 
Bump jackson databind version to 2.13.2.2

### Changes
- jackson bom will release a version when everytime its components releases, so we don't need to define variables like `jackson-data.version`
- bump jackson databind version to 2.13.2.2

Reviewers: Nicolò Boschi <boschi1997@gmail.com>, Enrico Olivelli <eolivelli@gmail.com>

This closes #3216 from Shoothzj/bump-jackson-databind-version-2-13-2-2

2 months agoBump async from 2.6.3 to 2.6.4 in /site3/website (#3219)
dependabot[bot] [Wed, 20 Apr 2022 14:15:35 +0000 (16:15 +0200)] 
Bump async from 2.6.3 to 2.6.4 in /site3/website (#3219)

Bumps [async](https://github.com/caolan/async) from 2.6.3 to 2.6.4.
- [Release notes](https://github.com/caolan/async/releases)
- [Changelog](https://github.com/caolan/async/blob/v2.6.4/CHANGELOG.md)
- [Commits](https://github.com/caolan/async/compare/v2.6.3...v2.6.4)

---
updated-dependencies:
- dependency-name: async
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2 months agoFix force GC doesn't work under forceAllowCompaction when disk is full (#3205)
Hang Chen [Wed, 20 Apr 2022 13:43:15 +0000 (21:43 +0800)] 
Fix force GC doesn't work under forceAllowCompaction when disk is full (#3205)

## Motivation
When I set `forceAllowCompaction=true` and one ledger disk reaches max usage threshold and transfer bookie to readOnly mode, I expire some pulsar topics or delete some topics to free up disk space. I found that ledger compression cannot be triggered when using `curl -XPUT http://localhost:8000/api/v1/bookie/gc` command.

The root cause is that when one ledger disk reaches max usage threshold, it will suspend minor and major compaction
https://github.com/apache/bookkeeper/blob/f7579fd13d62ce630ea26638e73f5884da505ec8/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/SingleDirectoryDbLedgerStorage.java#L1041-L1058

When we use `curl -XPUT http://localhost:8000/api/v1/bookie/gc` command to trigger compaction, it will be filtered by `suspendMajor` and `suspendMinor` flag.

https://github.com/apache/bookkeeper/blob/f7579fd13d62ce630ea26638e73f5884da505ec8/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java#L416-L444

It will lead to
- The bookie won't clean up deleted ledgers
- Ledger disk can't free up disk usage
- Bookie can't recover from readOnly state into Writeable state.

And then we can only trigger compaction by the following steps.
- Increase max disk usage threshold
- Restart the bookie
- Use command `curl -XPUT http://localhost:8000/api/v1/bookie/gc` to trigger compaction

### Changes
1. Don't take the `suspendMajor` and `suspendMinor` flag into consideration when setting `forceAllowCompaction=true` and triggered by force GC.

2 months agofix numLedgersReplicated metric not update (#3218)
Hang Chen [Tue, 19 Apr 2022 19:25:04 +0000 (03:25 +0800)] 
fix numLedgersReplicated metric not update (#3218)

2 months agoRemove log4j1 from bk test code
ZhangJian He [Tue, 19 Apr 2022 16:34:50 +0000 (00:34 +0800)] 
Remove log4j1 from bk test code

### Motivation
this is a part work of remove log4j1 from bookeeper, first remove it from our test code, unit test and integrate test

### Changes
- remove log4j1 scope test dependency
- use log4j2.xml instread of log4j.properties
- introduce testtools module to reduce duplication,  if a module needs particular configuration (very unlikely) we can add a log4j2-test.xml file which has precedence over log4j2.xml one. idea from [nicoloboschi](https://github.com/nicoloboschi)

Reviewers: Andrey Yegorov <None>, Enrico Olivelli <eolivelli@gmail.com>, Nicolò Boschi <boschi1997@gmail.com>

This closes #3208 from Shoothzj/remove-log4j1-from-bk-test-code2

2 months agoFix testCompactionPersistence do not cleanup created directory (#3207)
Hang Chen [Fri, 15 Apr 2022 06:00:44 +0000 (14:00 +0800)] 
Fix testCompactionPersistence do not cleanup created directory (#3207)

2 months ago[website] Make site3 the official website (deploy scripts - master) (#3187)
Nicolò Boschi [Thu, 14 Apr 2022 09:49:57 +0000 (11:49 +0200)] 
[website] Make site3 the official website (deploy scripts - master) (#3187)

* [website] Make site3 the official website

* fix action

2 months agoFix master broken ci due to zookeeper CVE fp (#3204)
ZhangJian He [Wed, 13 Apr 2022 12:37:08 +0000 (20:37 +0800)] 
Fix master broken ci due to zookeeper CVE fp (#3204)

2 months agosimplify throttle code in metadata scanning (#3196)
Hang Chen [Wed, 13 Apr 2022 01:49:15 +0000 (09:49 +0800)] 
simplify throttle code in metadata scanning (#3196)

### Motivation
The throttler has been checked whether throttle by bytes in `acquire` method, we doesn't need double check.

### Changes

Remove the double check of whether throttle by bytes.

2 months agoexpose netty threads num config to bk_server.conf (#3173)
Hang Chen [Wed, 13 Apr 2022 01:47:54 +0000 (09:47 +0800)] 
expose netty threads num config to bk_server.conf (#3173)

### Motivation
https://github.com/apache/bookkeeper/pull/1612 this PR make the Netty io threads number configurable, but doesn't expose to the `bk_server.conf`. This parameter will be usually changed in performance tuning especially deploy this bookie on bare metal which have many cpus. For operation engineer who is unfamiliar with the source code will be hard to do performance tuning.

### Changes
Expose this parameter configuration into `bk_server.conf` and website. And add more explanation about it.

2 months agoremove unnessary exception catch (#3169)
Hang Chen [Wed, 13 Apr 2022 01:47:20 +0000 (09:47 +0800)] 
remove unnessary exception catch (#3169)

### Motivation
When read Lac from ledger, we won't throw NoEntryException, remove this exception catch.

### Changes
Remove the NoEntryException catch in ReadLac operation.

2 months agoAllow running units test on jdk17 (#3195)
ZhangJian He [Mon, 11 Apr 2022 01:03:22 +0000 (09:03 +0800)] 
Allow running units test on jdk17 (#3195)

* Allow running units test on jdk17

* Allow running units test on jdk17

2 months agoBP-47 (task2): Add structured logger support (#3190)
Hang Chen [Sun, 10 Apr 2022 23:50:18 +0000 (07:50 +0800)] 
BP-47 (task2): Add structured logger support (#3190)

* Add slogger support

* update pom.xml

* fix check style

2 months agoBP-47 (task1): Add native IO library support (#3189)
Hang Chen [Sun, 10 Apr 2022 23:49:39 +0000 (07:49 +0800)] 
BP-47 (task1): Add native IO library support (#3189)

* Add native io library support

* fix jni compile issue

* fix findbugs failed

2 months agoUpgrade to log4j 2.17.2. (#3188)
Raúl Gracia [Thu, 7 Apr 2022 06:56:36 +0000 (08:56 +0200)] 
Upgrade to log4j 2.17.2. (#3188)

Signed-off-by: Raúl Gracia <raul.gracia@emc.com>