[BK-SERVER] Clean up over-replicated ledgers owned by different bookies
authorRajan Dhabalia <rdhabalia@apache.org>
Fri, 12 Jul 2019 08:03:10 +0000 (01:03 -0700)
committerEnrico Olivelli <eolivelli@gmail.com>
Fri, 12 Jul 2019 08:03:10 +0000 (10:03 +0200)
commit6b892a45f02a6d8d0b1d33235d4134d91e3ea111
tree347878cde63f07ebb10d683a83bd4c28db7af8f5
parent762a61365408d81a20193ef4ae1d44394746c105
[BK-SERVER] Clean up over-replicated ledgers owned by different bookies

### Motivation
As described at: https://github.com/apache/pulsar/issues/4632
- Sometimes due to overreplication, bookie contains ledgers which are not owned by that bookie anymore and that bookie is not part of the ensemble-list of those ledgers. In this case, GC finds out those overreplicated ledgers and
- deletes their index from dbStorage (rocksDB) and
- tries to delete them from entrylog files.

However, bookie doesn't delete them from entry-log files due to change made in [#870](https://github.com/apache/bookkeeper/issues/870) where bookie avoids deleting ledger if znode of that ledger exists.

Because of that bookie ends up storing large number entrylog files with ledgers which are owned by different bookies. It also cause OOM when GC tries to deal with large number of entry log files.

### Modification

Delete the ledgers if bookie is not part of ensemble list of over-replicated ledgers

Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Sijie Guo <sijie@apache.org>

This closes #2119 from rdhabalia/overRepl
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/ScanAndCompareGarbageCollector.java
bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/GcLedgersTest.java