Sijie Guo [Tue, 10 Jul 2018 08:45:44 +0000 (01:45 -0700)] 
Upgrade nokogiri to version 1.8.2 (#246)
*Motivation*
Version <1.8.2 contain a security vulnerability.
Sijie Guo [Wed, 28 Mar 2018 05:55:23 +0000 (22:55 -0700)]
Update README status about distributedlog project (#245)
Descriptions of the changes in this PR:
As distributedlog modules are merged into Apache BookKeeper, update the README status to reflect the project status.
See BP-26 for current status.
Sijie Guo [Thu, 30 Nov 2017 07:33:14 +0000 (23:33 -0800)] 
Include `nc` command in distributedlog docker image
Descriptions of the changes in this PR:
`nc` is usually used for checking if zookeeper is up and also verifying connectivity. It is good to have `nc` installed in the docker image by default. so a k8s helm script can use that for checking connectivities.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #241 from sijie/sijie/docker_image_nc
Sijie Guo [Thu, 30 Nov 2017 07:30:32 +0000 (23:30 -0800)]
Make distributedlog compiled with latest bookkeeper version
Descriptions of the changes in this PR:
There are code changes on PendingReadOp for new bookkeeper api in current master. DistributedLog uses PendingReadOp for some administration tools. So the current master doesn't compile with the latest bookkeeper version. This code change is to fix that.
The change here includes:
- bump bk to 4.7.0-SNAPSHOT (will switch to 4.6.0 after it is released)
- change to use the latest CompletableFuture in latest PendingReadOp. (this change doesn't target at making distributedlog work with new API)
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #240 from sijie/sijie/use_new_ledger_api
Arvin [Tue, 21 Nov 2017 04:22:35 +0000 (20:22 -0800)] 
issue #236: shaded jar lose necessary bk jar to execute
Descriptions of the changes in this PR:
after bump to bk4.6, add bk-proto bk-http dependency to shaded jar
Author: Arvin <arvindevel@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #237 from ArvinDevel/addBkDependencyInShade, closes #236
Jia Zhai [Wed, 1 Nov 2017 08:39:24 +0000 (01:39 -0700)] 
Update exception handling in constructing bookkeeper client
Sijie Guo [Tue, 31 Oct 2017 08:31:51 +0000 (01:31 -0700)]
Fix DistributedSchedule.WriteSet compilation issue
DisributedSchedule writeset structure is changed in recent 4.6.0-SNAPSHOT version
Sijie Guo [Mon, 30 Oct 2017 04:17:01 +0000 (12:17 +0800)]
DLFS - A FileSystem API wrapper over dlog API
Descriptions of the changes in this PR:
- FileSystem API wrapper built over dlog API
(This is based on initial implementation from gerritsundaram at #43)
Features supported:
- create and append files
- open files for reading
- input stream and output stream for reading and writing data
- list files
- get file status
- rename
- mkdir
Features aren't supported:
- truncate
- currently there is no clear distinguish between file and dir
- only support delete recursive
(This change includes small changes for #224 #225 #226 ).
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #227 from sijie/fix_create_log
Sijie Guo [Tue, 24 Oct 2017 11:56:42 +0000 (19:56 +0800)]
ISSUE #226: ByteBuf.release() was not called before it's garbage-collected
Descriptions of the changes in this PR:
the problem is a new entry buffer was allocated when closing log segment writer. the entry buffer is never used and also never recycled. It causes an annoying logging.
the fix is to assign a dummy entry writer which basically rejects writes, when closing a log segment writer. it would prevent leaking bytebuf.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #230 from sijie/fix_bytebuf_release_pr, closes #226
Sijie Guo [Tue, 24 Oct 2017 11:49:19 +0000 (19:49 +0800)]
ISSUE #224: listing logs should exclude <default>
Descriptions of the changes in this PR:
exclude `<default>` from listing logs
(the tests are covered by #227)
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #229 from sijie/fix_listing_log_pr, closes #224
Sijie Guo [Tue, 24 Oct 2017 11:35:36 +0000 (19:35 +0800)]
Issue 225: Create log should create missing path components
Descriptions of the changes in this PR:
reuse the methods used by `rename` to create missing path components.
(the test is covered by #227)
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #228 from sijie/fix_create_log_pr
Sijie Guo [Mon, 23 Oct 2017 11:01:47 +0000 (19:01 +0800)]
ISSUE #222: post-commit ci job is broken
Descriptions of the changes in this PR:
The reason that the post commit ci job is broken is because `git remote show apache` is used in `publish-website.sh` script, but there isn't `apache` branch in CI cloned workspace. we need to change this from `apache` to `origin`.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #223 from sijie/debug_postcommit_job, closes #222
Sijie Guo [Mon, 23 Oct 2017 09:14:02 +0000 (17:14 +0800)]
ISSUE #209: Support rename log
Descriptions of the changes in this PR:
- add rename operation in `Namespace`
- add rename operation in `LogStreamMetadataStore`
- implement the rename operation use zookeeper `multi` operation
Author: Sijie Guo <sijie@apache.org>
Author: Shoukun Huai <shoukunhuai@gmail.com>
Author: Arvin <arvindevel@gmail.com>
Reviewers: Jia Zhai <None>
This closes #210 from sijie/4_support_rename_pr, closes #209
Jia Zhai [Mon, 23 Oct 2017 08:03:25 +0000 (01:03 -0700)]
ISSUE #219: Exclude files under target directory from apache-rat check
Descriptions of the changes in this PR:
The problem is distributedlog-client and distributedlog-service have been renamed to distributedlog-proxy-client and distributedlog-proxy-server. so those files don't belong to any active modules, so they are not excluded automatically. exclude them manually.
Author: Jia Zhai <zhaijia@apache.org>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #220 from zhaijack/exclude_files_under_targets, closes #219
Jia Zhai [Mon, 23 Oct 2017 07:47:40 +0000 (00:47 -0700)]
ISSUE #217: Enable apache-rat check to output result to console
Descriptions of the changes in this PR:
- bump `apache-rat` to 0.12. since `outputConsole` is only supported since 0.12
- enable `outputConsole` to dump the files that have unapproved licences.
Author: Jia Zhai <zhaijia@apache.org>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #218 from zhaijack/output_ratcheck_results_to_console, closes #217
Arvin [Mon, 23 Oct 2017 06:45:16 +0000 (23:45 -0700)]
ISSUE164: Move findbug settings files to distributedlog-build-tools
Descriptions of the changes in this PR:
Move findbug settings files to distributedlog-build-tools,
and a little change to bk-shade: add bk-common dependency and unshade netty.
Author: Arvin <arvindevel@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #216 from ArvinDevel/issue164
Sijie Guo [Wed, 18 Oct 2017 22:14:16 +0000 (15:14 -0700)]
ISSUE #211: Support listing logs by prefix
Descriptions of the changes in this PR:
- extend `getLogs` to `getLogs(prefix)`, so it provides a filesystem `listFiles`-like semantic.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #212 from sijie/5_support_list_logs_by_prefix_pr, closes #211
Sijie Guo [Wed, 18 Oct 2017 20:09:34 +0000 (13:09 -0700)]
ISSUE #207: Support getFirstLogRecord
Descriptions of the changes in this PR:
- support getFirstLogRecord
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #208 from sijie/3_get_firstrecord_pr, closes #207
Sijie Guo [Wed, 18 Oct 2017 20:07:54 +0000 (13:07 -0700)]
Issue 205: Avoid copying bytebuf for constructing log record to write
Descriptions of the changes in this PR:
- avoid copying memory for writer facing constructor for log records
- construct a ByteBuf by copying ByteBuffer for reader facing constructors
This closes #205, #206
Sijie Guo [Wed, 18 Oct 2017 20:06:39 +0000 (13:06 -0700)]
Issue 201: Fix the flaky test TestBKDistributedLogManager.deleteDuringRead
This closes #204, #201
Arvin [Wed, 18 Oct 2017 19:16:58 +0000 (12:16 -0700)]
ISSUE173: Implement AutoCloseable in BKDistributedLogNamespace
Descriptions of the changes in this PR:
Implement AutoCloseable in BKDistributedLogNamespace
Author: Arvin <arvindevel@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #214 from ArvinDevel/issue173
Shoukun Huai [Wed, 18 Oct 2017 19:14:58 +0000 (12:14 -0700)] 
ISSUE #213: Correct logger name in BKLogWriteHandler
correct logger name in BKLogWriteHandler
Descriptions of the changes in this PR:
(PR description content here)...
Author: Shoukun Huai <shoukunhuai@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #215 from shoukunhuai/issue-213, closes #213
Sijie Guo [Wed, 4 Oct 2017 09:48:04 +0000 (02:48 -0700)]
Avoid copying bytebuf for constructing log record to write
Sijie Guo [Wed, 4 Oct 2017 10:07:05 +0000 (03:07 -0700)]
Fix the flaky test TestBKDistributedLogManager.deleteDuringRead
Sijie Guo [Wed, 11 Oct 2017 06:02:38 +0000 (14:02 +0800)]
ISSUE #197: Provide a guide for running on k8s
Descriptions of the changes in this PR:
add one page for how to deploy on k8s. it is based on http://bookkeeper.apache.org/docs/latest/deployment/kubernetes/ and use distributedlog image and add instructions on how to create distributedlog namespaces and run benchmark.
This change is based on #196
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #198 from sijie/add_docker, closes #197
Arvin [Wed, 4 Oct 2017 09:04:57 +0000 (02:04 -0700)]
ISSUE #193: Shade Bookkeeper and unshade ZooKeeper in core module
Descriptions of the changes in this PR:
Shade Bookkeeper and unshade ZooKeeper in core module
Author: Arvin <arvindevel@gmail.com>
Reviewers: Jia Zhai <None>, Enrico Olivelli <eolivelli@gmail.com>, Sijie Guo <sijie@apache.org>
This closes #194 from ArvinDevel/shade_bk, closes #193
Arvin [Wed, 4 Oct 2017 08:59:45 +0000 (01:59 -0700)]
ISSUE #165: Enable checkstyle in core module
Descriptions of the changes in this PR:
Fix checkstyle error in core module
Author: Arvin <arvindevel@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #192 from ArvinDevel/enable_check, closes #165
Arvin [Wed, 4 Oct 2017 08:48:15 +0000 (01:48 -0700)]
ISSUE #181: Enable LogRecord ByteBuf based constructor public
Descriptions of the changes in this PR:
change LogRecord ByteBuf based constructor to public
Author: Arvin <arvindevel@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #182 from ArvinDevel/issue_181, closes #181
Sijie Guo [Sat, 16 Sep 2017 10:03:52 +0000 (18:03 +0800)]
Update ci.sh script to build website
Descriptions of the changes in this PR:
The ci.sh was copied from bookkeeper project but was updated correctly. This issue is to address the problem.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #191 from sijie/fix_ci_script
Sijie Guo [Sat, 16 Sep 2017 09:55:28 +0000 (17:55 +0800)]
ISSUE #186: distributedlog.incubator.apache.org should have same content as bookkeeper.apache.org/distributedlog
Descriptions of the changes in this PR:
- move `docs` to `website/docs/latest`
- change `header.html` to support different logo url, to support both bookkeeper.apache.org/distributedlog and distributedlog.incubator.apache.org
- add docker scripts to build website and javadoc
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #190 from sijie/automate_building_website, closes #186
Jia Zhai [Sat, 16 Sep 2017 04:40:36 +0000 (21:40 -0700)]
ISSUE #171: Website and documentation updates for release 0.5.0
Descriptions of the changes in this PR:
Website and documentation updates for release 0.5.0
Since the release is in-progress, please hold on to merge this PR.
Author: Jia Zhai <zhaijia@apache.org>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #179 from zhaijack/release050_website_updates, closes #171
Jia Zhai [Sat, 16 Sep 2017 04:34:26 +0000 (21:34 -0700)]
ISSUE #168: Release Notes for 0.5.0
Descriptions of the changes in this PR:
Release Notes for release0.5.0
Since the release is in-progress, please hold on to merge this PR.
Author: Jia Zhai <zhaijia@apache.org>
Reviewers: Khurrum Nasim <None>, Sijie Guo <sijie@apache.org>
This closes #180 from zhaijack/releasenotes_050, closes #168
Sijie Guo [Tue, 12 Sep 2017 07:16:49 +0000 (15:16 +0800)]
ISSUE #185: Thrift version conflicts and use heap bytebuffer for thrift serialization
Descriptions of the changes in this PR:
- it seems that shade plugin doesn't work well with sub-modules. even the libthrift-9 was shaded, it is still imported/included in the sub-modules, causing the version conflict. Explicitly exclude the libthrift from distributedlog-core.
- thrift serialization is using heap bytebuffer. so if a bytebuffer is direct, the serialization will fail. add a change to check if bytebuffer is heap, if bytebuffer is not heap, copy the content into a heap buffer. this is not good for performance, but we don't have any other choices because the limitation comes from libthrift.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Jia Zhai <None>, Leigh Stewart <None>
This closes #187 from sijie/use_heapbytebuffer, closes #185
Jia Zhai [Thu, 7 Sep 2017 07:37:47 +0000 (15:37 +0800)]
[maven-release-plugin] prepare for next development iteration
Jia Zhai [Thu, 7 Sep 2017 07:37:17 +0000 (15:37 +0800)]
[maven-release-plugin] prepare branch release-0.5.0
Sijie Guo [Thu, 7 Sep 2017 07:28:04 +0000 (15:28 +0800)]
ISSUE #177: mvn release failed due to wrong scm url
Descriptions of the changes in this PR:
Reason:
we are using `gitbox` url as the scm connection. However gitbox is a readonly mirror of github repo.
The committers don't have permissions to commit directly to gitbox repo.
Solution:
update the scm connection to github url
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #178 from sijie/fix_scm_url, closes #177
Sijie Guo [Thu, 7 Sep 2017 06:33:28 +0000 (23:33 -0700)]
ISSUE #160: Introduce backward compatibility testing
Descriptions of the changes in this PR:
The core testing logic is in `tests/bin/docker-backward-test.sh`.
- introduce an all module for packaging
- add docker images for 0.3.51, 0.4.0-incubating and latest
- add two backward compat testing module
- using docker images for backward compatibility testing
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>, Leigh Stewart <None>
This closes #175 from sijie/docker_image, closes #160
Sijie Guo [Wed, 6 Sep 2017 03:24:42 +0000 (11:24 +0800)]
ISSUE #166: Code cleanup for 0.5.0 release
Descriptions of the changes in this PR:
- add InterfaceAudience and InterfaceStability for public API to inform people what changes would be expected for those interface.
- avoid using guava classes in public API since we will provide a shaded jar for distributedlog-core
- enable ImportOrder checkstyle rule in some modules
- move `org.apache.distributedlog.io` to `distributedlog-common` module
- rename `setReadyToFlush` to `flush` and rename `flushAndSync` to `commit` for the new API
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>, Leigh Stewart <None>
This closes #172 from sijie/finalize_api, closes #166
Sijie Guo [Thu, 31 Aug 2017 06:50:53 +0000 (23:50 -0700)]
DL-45: DL should allow ByteBuf based API to avoid copying bytes array
Descriptions of the changes in this PR:
This change leverages the `ByteBuf` api introduced in bookkeeper 4.5.0. It will avoid copying bytes array between LogRecord and LogRecordSet/Entry, and avoid copying bytes from DL to BK client.
This change also bump the lz4 library to `1.3.0` to leverage the `ByteBuffer` binding.
Author: Sijie Guo <sijie@apache.org>
Author: Jia Zhai <zhaijia03@gmail.com>
Author: Jia Zhai <zhaijia@apache.org>
Reviewers: Jia Zhai <None>, Enrico Olivelli <eolivelli@gmail.com>, Leigh Stewart <None>
This closes #151 from sijie/zero_copy
Jia Zhai [Thu, 31 Aug 2017 06:49:33 +0000 (23:49 -0700)]
ISSUE #154: Update mailing list page
Descriptions of the changes in this PR:
Update mailing lists with `distributedlog-issuesbookkeeper.apache.org`.
Author: Jia Zhai <zhaijia@apache.org>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #163 from zhaijack/update_mailing_lists, closes #154
Jia Zhai [Thu, 31 Aug 2017 06:48:34 +0000 (23:48 -0700)]
ISSUE #155: Update home page banner
Descriptions of the changes in this PR:
Update home page banner
Author: Jia Zhai <zhaijia@apache.org>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #162 from zhaijack/update_banner, closes #155
Jia Zhai [Thu, 31 Aug 2017 06:47:45 +0000 (23:47 -0700)]
ISSUE #156: Provide a shading jar for distributedlog-core
Descriptions of the changes in this PR:
- provide a shade-all jar (shading all the dependencies) for distributedlog-core. the `shade-all` jar is classified as `shaded`.
Author: Jia Zhai <zhaijia@apache.org>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #161 from zhaijack/shade_libs, closes #156
Jia Zhai [Wed, 30 Aug 2017 04:49:36 +0000 (21:49 -0700)]
ISSUE #157: Introduce JMH for micro benchmarking
Descriptions of the changes in this PR:
- add a `tests/jmh` and `tests/jmh-0.4` module for micro-benchmarking
- `tests/jmh` is to benchmark latest master
- `tests/jmh-0.4` is to benchmark `0.4.0-incubating`
- add benchmark for compression
Author: Jia Zhai <zhaijia@apache.org>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #158 from zhaijack/tests, closes #157
Jia Zhai [Wed, 30 Aug 2017 03:40:50 +0000 (11:40 +0800)]
ISSUE #142: remove word incubator in coveralls.io in README
Descriptions of the changes in this PR:
[INFRA-14937](https://issues.apache.org/jira/browse/INFRA-14937) ticket has been solved, so update the coveralls link here.
Author: Jia Zhai <zhaijia@apache.org>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #159 from zhaijack/issue_142, closes #142
Yiming Zang [Thu, 24 Aug 2017 05:25:40 +0000 (22:25 -0700)] 
DL-206: Delete the log should also delete the underline ledgers
Problem:
We're not deleting the ledgers when we delete the whole log stream using dlm.delete() API. This would cause a lot of garbage/orphan ledgers in Bookkeeper.
The fix is to delete the ledger when we delete the log stream. Also added a test to validate.
Author: Yiming Zang <yzang@twitter.com>
Reviewers: Sijie Guo <sijie@apache.org>
This closes #152 from yzang/yzang/DL-206
Jia Zhai [Thu, 17 Aug 2017 23:51:53 +0000 (16:51 -0700)] 
DL-2: DistributedLog should work with the official apache bookkeeper
This change is to upgrade bookkeeper version to BK 4.5.0.
- upgrade bookkeeper version to 4.5.0-SNAPSHOT (still waiting a few pull requests to merge apache/bookkeeper#297 apache/bookkeeper#287
- change registerSuccessEvent for StatsLogger to add TimeUnit
- use netty4 eventloop
- move twitter repository dependencies to proxy related module only. core library will not depend on scala dependency anymore.
This change is a collaboration change with sijie
We will provide a performance comparison between 0.4.0 (using Twitter BK) and 0.5.0 (using BK 4.5.) in a separate pull request or email.
Author: Jia Zhai <zhaijia03@gmail.com>
Author: Jia Zhai <zhaijia@apache.org>
Author: Sijie Guo <sijie@apache.org>
Reviewers: Leigh Stewart <lstewart@apache.org>
This closes #135 from zhaijack/bump_dl_version
Sijie Guo [Wed, 16 Aug 2017 01:25:00 +0000 (18:25 -0700)]
DL-209: Update build script to build distributedlog website under `bookkeeper.apache.org/distributedlog`
*Problem*:
Currently `bookkeeper.apache.org/distributedlog` is using content at the root of branch `asf-site`.
`distributedlog.incubator.apache.org` is using content at `content` dir of branch `asf-site`.
The website build script should support both.
*Solution*:
- add `_config-apache.yml` for building website at baseurl `/distributedlog`
- update `website/build.sh` to build apache to build different configurations
- add a `publish-website.sh` to be able to publish website by CI
Also:
- add `_config-staging.yml` for building website at staging
- add a `staging-website.sh` to publish staging site.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
This closes #136 from sijie/fix_production_url
Sijie Guo [Wed, 16 Aug 2017 01:22:27 +0000 (18:22 -0700)]
ISSUE #146: Update merge script to use bk merge script
Descriptions of the changes in this PR:
- copy the `dev/bk-merge-pr.py` to distributedlog
- update the repo from `bookkeeper` to `distributedlog`
Author: Sijie Guo <sijie@apache.org>
Author: Jia Zhai <zhaijia03@gmail.com>
Reviewers: Jia Zhai <None>
This closes #149 from sijie/handle_labels, closes #146
Jia Zhai [Tue, 15 Aug 2017 22:41:51 +0000 (15:41 -0700)]
ISSUE #144: add template for github issue and pull request
This is part of the work to adopt github issue for issue management.
Use similar template as bookkeeper.
Author: Jia Zhai <zhaijia03@gmail.com>
Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Sijie Guo <sijie@apache.org>
This closes #147 from zhaijack/dl_144, closes #144
Jia Zhai [Tue, 15 Aug 2017 01:13:08 +0000 (18:13 -0700)]
Issue 141: graduation changes, remove/change incubate related things
- remove the "-incubating" from the code base.
- change website link from http://distributedlog.incubator.apache.org into https://bookkeeper.apache.org/distributedlog
- change repo from https://github.com/apache/incubator-distributedlog into https://github.com/apache/distributedlog
- git from https://git-wip-us.apache.org/repos/asf/incubator-distributedlog.git into https://gitbox.apache.org/repos/asf/distributedlog.git
- remove website foot, whch contains incubate infos in websiteindex.md
- remove Disclaimer file and its references
Author: Jia Zhai <zhaijia03@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #143 from zhaijack/issue_141
Jia Zhai [Thu, 10 Aug 2017 08:03:28 +0000 (01:03 -0700)]
DL-208: Update release notes about the download location
- fix DL 208: update download location
- update dev repository in release_guide
Author: Jia Zhai <zhaijia03@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #138 from zhaijack/DL-208
Jia Zhai [Thu, 10 Aug 2017 08:02:29 +0000 (01:02 -0700)]
DL-210: Update DL mailing lists
Mail list changed from
{user,dev,commits}distributedlog.incubator.apache.org
to
distributedlog-{user,dev,commits}bookkeeper.apache.org
Author: Jia Zhai <zhaijia03@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #139 from zhaijack/DL-210
Jia Zhai [Thu, 10 Aug 2017 08:00:12 +0000 (01:00 -0700)]
Issue 137: fix Tutorial link is broken
change docs direction from main/java/com/twitter/distributedlog to main/java/org/apache/distributedlog"
Author: Jia Zhai <zhaijia03@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #140 from zhaijack/issue_137
Sijie Guo [Wed, 21 Jun 2017 17:21:46 +0000 (10:21 -0700)]
DL-81: Build the distributedlog release procedure
Borrow from http://beam.incubator.apache.org/contribute/release-guide/
Author: Sijie Guo <sijie@apache.org>
Reviewers: Franck Cuny <franck.cuny@gmail.com>
Closes #51 from sijie/sijie/release_guid
Sijie Guo [Wed, 21 Jun 2017 17:20:15 +0000 (10:20 -0700)]
DL-124: Use Java8 Future rather than twitter Future
Switch to use Java8 CompletableFuture, to reduce dependencies introduced by twitter future and make it more friendly to users (users don't think of using which version of scala).
This change is based on #132 . Gitsha
ce0686e is the change to review.
The changes:
- Change Future to CompletableFuture
- Map to thenApply
- flatMap to thenCompose
- Added a FutureEventListener, and switch addEvenListener to whenComplete (or whenCompleteAsync)
- setValue to complete
- setException to completeExceptionally
- add rescue, ignore, ensure to FutureUtils as util functions.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>, Leigh Stewart <lstewart@apache.org>
Closes #133 from sijie/change_twitter_future_to_java_future
Sijie Guo [Wed, 21 Jun 2017 17:17:46 +0000 (10:17 -0700)]
Fix merge script issues
Problem:
Currently the merge script is broken when the reviewer's id is not in reviewers list.
Solution:
- Change "reviewers[reviewer_id]" to "reviewer_id in reviewers"
- Bump the default version to 0.5.0.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>
Closes #134 from sijie/fix_merge_script_issues
Sijie Guo [Mon, 12 Jun 2017 21:12:13 +0000 (14:12 -0700)]
DL-204: Bump libthrift to latest version for distributedlog-core
Currently finagle heavily depends on an out-of-dated version - libthrift 5.0. Proxy modules (client, server) depend on this version, however the core library doesn't really depend on libthrift.
This change is to change libthrift to 0.9.* in distributedlog-core and shade it to avoid it conflict with the version used by finagle.
This change is based on #131 . The main change is at gitsha [
6e58786](https://github.com/apache/incubator-distributedlog/commit/
6e587869f87cdce50ae93ba3d52767719d1ab5a6)
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>, Leigh Stewart <lstewart@apache.org>
Closes #132 from sijie/change_thrift_for_core_module
Sijie Guo [Mon, 12 Jun 2017 19:55:41 +0000 (12:55 -0700)]
DL-199: Be able to support filesystem-path like name
In order to support hierarchical namespace, we need to be able to support filesystem path like log name.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>, Leigh Stewart <lstewart@apache.org>
Closes #130 from sijie/DL_199
Sijie Guo [Mon, 12 Jun 2017 15:45:01 +0000 (08:45 -0700)]
DL-205: Remove StatusCode dependency on DLException
- Remove StatusCode from exceptions. Use integer as exception codes.
- Also re-organize the modules:
- [ ] distributedlog-protocol (for core structures) and distributedlog-core (for core library).
- [ ] proxy: distributedlog-proxy-protocol (new module for thrift generated protocol), distributedlog-proxy-client (proxy client) and distributedlog-proxy-server (proxy server)
- [ ] benchmark & tutorials.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <None>, Leigh Stewart <lstewart@apache.org>
Closes #131 from sijie/new_layout
Sijie Guo [Wed, 24 May 2017 17:08:43 +0000 (01:08 +0800)]
Remove package-info.java under com.twitter.distributedlog.subscription
There is still one file left under com.twitter.distributedlog after repackage. This change is to delete this file.
https://github.com/apache/incubator-distributedlog/tree/master/distributedlog-core/src/main/java/com/twitter/distributedlog/subscription
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <zhaijia03@gmail.com>
Closes #129 from sijie/sijie/remove_subscription_files
Sijie Guo [Wed, 26 Apr 2017 18:41:12 +0000 (11:41 -0700)]
Release 0.4.0-incubating
Changes for 0.4.0-incubating
- change the stable doc to 0.4.0-incubating
- change latest version to 0.5.0-incubating-snapshot
- add 0.4.0-incubating to the header
- copied current docs to website/docs/0.4.0-incubating as the documents for release 0.4.0-incubating.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <zhaijia03@gmail.com>, Jia Zhai <zhaijia03@gmail.com>, Leigh Stewart <lstewart@apache.org>
Closes #109 from sijie/sijie/release_0.4.0
adamtracymartin [Thu, 13 Apr 2017 04:38:32 +0000 (21:38 -0700)] 
DL-174: added getParent method to Utils to replace usage of File.getParent which changes / to \ on Windows
…ich changes / to \ in Windows.
Author: adamtracymartin <atmartin@yahoo.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #104 from adamtracymartin/DL-174
adamtracymartin [Thu, 13 Apr 2017 04:36:19 +0000 (21:36 -0700)]
DL-173: changed FileUtils.deleteDirectory to FileUtils.forceDeleteOnExit so d…
I am doing a new pull request because I deleted the branch in the remote.
Actions done to fix the conflict.
1. git checkout DL-173
2. git checkout master distributedlog-protocol/src/main/thrift/service.thrift
3. git fetch
4. git add *
5. git commit with message
The actual fix for this branch is changing FileUtils.deleteDirectory to FileUtils.forceDeleteOnExit.
Author: adamtracymartin <atmartin@yahoo.com>
Reviewers: Jia Zhai <zhaijia03@gmail.com>, Sijie Guo <sijie@apache.org>, Leigh Stewart <lstewart@apache.org>
Closes #116 from adamtracymartin/DL-173
Sijie Guo [Thu, 13 Apr 2017 03:57:45 +0000 (20:57 -0700)]
DL-198: Combining the NOTICE, LICENSE, and DISCLAIMER files
Author: Sijie Guo <sijie@apache.org>
Reviewers: Henry Saputra <henry.saputra@gmail.com>, Jia Zhai <zhaijia03@gmail.com>
Closes #128 from sijie/combine_asf_files
Sijie Guo [Thu, 13 Apr 2017 03:55:07 +0000 (20:55 -0700)]
Change the NOTICE year for Twitter Copyright to 2016
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <zhaijia03@gmail.com>
Closes #127 from sijie/change_notice
Sijie Guo [Thu, 13 Apr 2017 03:54:11 +0000 (20:54 -0700)]
DL-196: Remove 'Copyright 2017 The Apache Software Foundation' from license header
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <zhaijia03@gmail.com>
Closes #126 from sijie/sijie/fix_license_headers
Sijie Guo [Thu, 13 Apr 2017 03:52:56 +0000 (20:52 -0700)]
DL-197: Add DISCLAIMER file to the project
DISCLAIMER is missing for source release.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <zhaijia03@gmail.com>
Closes #125 from sijie/sijie/include_DISCLAIMER_file
jiazhai [Mon, 3 Apr 2017 16:52:04 +0000 (09:52 -0700)]
DL-195: exclude DISCLAIMER.bin.txt for Apache Rat Check Failures
"mvn apache-rat:check package findbugs:check -DskipTests" would fail with report:
```
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.7:check (default-cli) on project distributedlog_2.10: Too many unapproved licenses: 1 -> [Help 1]
```
Check the rat report, seems we need to exclude file DISCLAIMER.bin.txt from rat checking.
After exclude 1 file for rat check, by adding this line at line232 of pom.xml, we could execute above command successfully,
```
<exclude>src/main/resources/DISCLAIMER.bin.txt</exclude>
```
build result
```
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache DistributedLog :: Parent .................... SUCCESS [ 2.625 s]
[INFO] Apache DistributedLog :: Build Tools ............... SUCCESS [ 0.809 s]
[INFO] Apache DistributedLog :: Protocol .................. SUCCESS [ 21.284 s]
[INFO] Apache DistributedLog :: Core Library .............. SUCCESS [ 24.496 s]
[INFO] Apache DistributedLog :: Proxy Client .............. SUCCESS [ 11.254 s]
[INFO] Apache DistributedLog :: Proxy Service ............. SUCCESS [ 21.825 s]
[INFO] Apache DistributedLog :: Benchmark ................. SUCCESS [ 14.015 s]
[INFO] Apache DistributedLog :: Tutorials :: Basics ....... SUCCESS [ 18.098 s]
[INFO] Apache DistributedLog :: Tutorials :: Messaging Tutorial SUCCESS [ 32.122 s]
[INFO] Apache DistributedLog :: Tutorials :: Kafka Tutorial SUCCESS [ 51.698 s]
[INFO] Apache DistributedLog :: Tutorials :: MapReduce Tutorial SUCCESS [09:17 min]
[INFO] Apache DistributedLog :: Tutorials ................. SUCCESS [ 0.050 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
```
Author: jiazhai <zhaijia03@gmail.com>
Author: jiazhai <jiazhai@users.noreply.github.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #124 from jiazhai/DL-195 and squashes the following commits:
6c642c3 [jiazhai] exclude 1 file to pass rat check
f30f454 [jiazhai] Merge remote-tracking branch 'us/master'
a40f92f [jiazhai] Merge pull request #2 from apache/master
f2cb5f4 [jiazhai] Merge pull request #1 from apache/master
xieliang [Tue, 28 Mar 2017 07:09:47 +0000 (00:09 -0700)] 
DL-184: reduce server_graceful_shutdown_period_ms from 35s to 20s, which is smaller than the default daemon stop timeout value
Author: xieliang <xieliang007@gmail.com>
Reviewers: Leigh Stewart <lstewart@apache.org>, Sijie Guo <sijie@apache.org>
Closes #115 from xieliang/DL-184-graceful
Sijie Guo [Thu, 23 Mar 2017 03:28:55 +0000 (23:28 -0400)]
DL-190: Add DISCLAIMER to the release packages
per apache release policy, we need to DISCLAIMER for releases.
http://incubator.apache.org/incubation/Incubation_Policy.html#Releases
Author: Sijie Guo <sijie@apache.org>
Reviewers: Leigh Stewart <lstewart@apache.org>
Closes #120 from sijie/sijie/add_disclaimer
Sijie Guo [Thu, 23 Mar 2017 03:27:19 +0000 (23:27 -0400)]
DL-191: Fix license header issues
the copyright year is wrong.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Leigh Stewart <lstewart@apache.org>, Flavio Junqueira <fpj@apache.org>
Closes #121 from sijie/sijie/fix_license_header
arvindkandhare [Tue, 21 Mar 2017 21:50:59 +0000 (14:50 -0700)] 
DL-193: Bumped up the Guava version
Moving the Guava version from 16.0 to 20.0.
Author: arvindkandhare <arvind.kandhare@emc.com>
Reviewers: Henry Saputra <henry.saputra@gmail.com>, Sijie Guo <sijie@apache.org>
Closes #123 from arvindkandhare/guava_version
Sijie Guo [Wed, 8 Feb 2017 07:16:13 +0000 (23:16 -0800)]
DL-192: year needs updating in NOTICE
The year was 2016. but we are going to release at 2017. so updating the files to 2017.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Leigh Stewart <lstewart@apache.org>
Closes #119 from sijie/sijie/fix_notice_file
Sijie Guo [Wed, 8 Feb 2017 07:10:33 +0000 (23:10 -0800)]
Fix log4j binding issue on binary package
Author: Sijie Guo <sijie@apache.org>
Reviewers: Leigh Stewart <lstewart@apache.org>
Closes #122 from sijie/sijie/fix_log4j_issue
xieliang [Tue, 31 Jan 2017 01:52:29 +0000 (17:52 -0800)]
DL-186: fix common.sh grep bug; kill MultiReader java process as expected
Author: xieliang <xieliang007@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #114 from xieliang/DL-186-smoketest
xieliang [Tue, 31 Jan 2017 01:51:08 +0000 (17:51 -0800)]
DL-187: load test doc typo fix
Author: xieliang <xieliang007@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #117 from xieliang/DL-187-doc_typo
xieliang [Tue, 31 Jan 2017 01:49:58 +0000 (17:49 -0800)]
DL-189: remove the extra NOTICE change
Author: xieliang <xieliang007@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #118 from xieliang/DL-189
Sijie Guo [Wed, 11 Jan 2017 06:59:27 +0000 (22:59 -0800)]
DL-183: Configure DL to generate source jar
also move the plugin version definition to the properties
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <zhaijia@apache.org>
Closes #113 from sijie/sijie/generate_source and squashes the following commits:
6fa1ce5 [Sijie Guo] Merge branch 'master' into sijie/generate_source
d355d6a [Sijie Guo] Generate source jar during jar phase
Sijie Guo [Wed, 11 Jan 2017 06:56:11 +0000 (22:56 -0800)]
Javdoc build should use "org.apache.distributedlog"
The packages are now under "org.apache.distributedlog", instead of "com.twitter.distributedlog".
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <zhaijia@apache.org>
Closes #108 from sijie/sijie/fix_javadoc_build
Sijie Guo [Wed, 11 Jan 2017 05:45:42 +0000 (21:45 -0800)]
Fix the script to make sure binary package can execute scripts correctly
- fix the release jar path
- copy the common script to each modules otherwise the built package won't be able to execute
- change the type of runner and copy the runner script to each tutorial module
Author: Sijie Guo <sijie@apache.org>
Reviewers: Jia Zhai <zhaijia@apache.org>
Closes #111 from sijie/sijie/copy_all_scripts_to_modules
xieliang [Wed, 11 Jan 2017 05:00:58 +0000 (21:00 -0800)]
DL-182: assembled package should include README.md
Author: xieliang <xieliang007@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #112 from xieliang/DL-182-assembled
xieliang [Tue, 10 Jan 2017 06:47:09 +0000 (22:47 -0800)]
DL-181: avoid starting failure due to call Optional.get() on an absent value
see original JIRA for the detailed error msg
Author: xieliang <xieliang007@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #110 from xieliang/DL-181-startingProxyFailed
Sijie Guo [Sun, 8 Jan 2017 22:46:53 +0000 (14:46 -0800)] 
[maven-release-plugin] prepare for next development iteration
Sijie Guo [Sun, 8 Jan 2017 22:46:49 +0000 (14:46 -0800)]
[maven-release-plugin] prepare branch release-0.4.0-incubating
Sijie Guo [Sun, 8 Jan 2017 22:46:02 +0000 (14:46 -0800)]
Remove Xlint compiler arguments for building release
Gerrit Sundaram [Sun, 8 Jan 2017 22:10:37 +0000 (14:10 -0800)] 
DL-170: All dependency versions should be defined in the parent pom
make all dependencies' version be defined in the parent pom file.
Author: Gerrit Sundaram <gerritsundaram@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #97 from gerritsundaram/DL-170
adamtracymartin [Sun, 8 Jan 2017 22:09:06 +0000 (14:09 -0800)]
DL-180: added script to point to ../../../scripts/runner instead of symlink
Author: adamtracymartin <atmartin@yahoo.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #107 from adamtracymartin/DL-180
adamtracymartin [Sun, 8 Jan 2017 22:06:38 +0000 (14:06 -0800)]
DL-152: removed symlink distributedlog-service/conf and copied files from distributedlog-core/conf to distributed-service/conf
…tributedlog-core/conf to distributedlog-service/conf
Author: adamtracymartin <atmartin@yahoo.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #105 from adamtracymartin/DL-152
adamtracymartin [Sun, 8 Jan 2017 22:03:13 +0000 (14:03 -0800)]
DL-28: changed // comments to /* */
Author: adamtracymartin <atmartin@yahoo.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #100 from adamtracymartin/DL-28
Sijie Guo [Sun, 8 Jan 2017 22:01:35 +0000 (14:01 -0800)]
DL-83: Add javadoc to website
- update the javadoc plugin to aggregate api by 'core library' and 'proxy service'. (turn some java class to package private and delete unused files. so they won't appear in java doc)
- improve the website build script to include javadoc
- update some readme files to include user mail list and slack channel
Author: Sijie Guo <sijieg@twitter.com>
Author: Sijie Guo <sijie@apache.org>
Reviewers: Leigh Stewart <lstewart@apache.org>
Closes #96 from sijie/sijie/merge_website_script
Sijie Guo [Sat, 7 Jan 2017 00:51:06 +0000 (16:51 -0800)]
DL-176: Rename the DL artifact from com.twitter to org.apache.distributedlog
Author: Sijie Guo <sijieg@twitter.com>
Reviewers: Dave Rusek <drusek@apache.org>, Leigh Stewart <lstewart@apache.org>
Closes #102 from sijie/sijie/fix_pom_file_layout
xieliang [Sat, 7 Jan 2017 00:30:32 +0000 (16:30 -0800)]
DL-171: adding a short sleep to let the WriteCompleteListener have time to run before the final position be requested
once the "writer.write" is done, if "writer.position()" be invoked easier than the WriteCompleteListener onSuccess callback, due to the "synchronized", the position result will be 0, not the expected 33. we can just add a short sleep to avoid this test issue.
Author: xieliang <xieliang007@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #98 from xieliang/DL-171-Fix-TestAppendOnlyStreamWriter
xieliang [Thu, 5 Jan 2017 06:55:51 +0000 (22:55 -0800)]
DL-167: fix testCalculateUnequalWeight computing corner case
Author: xieliang <xieliang007@gmail.com>
Reviewers: Leigh Stewart <lstewart@apache.org>
Closes #94 from xieliang/DL-167-fix-testCalculateUnequalWeight
Sijie Guo [Thu, 5 Jan 2017 00:44:15 +0000 (16:44 -0800)]
DL-4: Repackage the source under apache namespace
Khurrum Nasim [Thu, 5 Jan 2017 00:12:52 +0000 (16:12 -0800)] 
DL-4: Add repackaging script
this pull request only contains the script to repackage files.
run it at the root directory of the project - **./scripts/dev/repackage.sh**
once the script is done, run "**mvn clean apache-rat:check package windbags:check**" to verify everything work.
then you can commit the changes with the modified files.
/cc leighst sijie
Author: Khurrum Nasim <khurrumnasimm@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #95 from khurrumnasimm/kn/repackaging_script
xieliang [Thu, 5 Jan 2017 00:09:01 +0000 (16:09 -0800)]
DL-165: Add TestTimedOutTestsListener to dump timed out cases thread dump
Author: xieliang <xieliang007@gmail.com>
Reviewers: Leigh Stewart <lstewart@apache.org>
Closes #91 from xieliang/DL-165-TimedOutTestsListener
Xi Liu [Wed, 4 Jan 2017 08:43:56 +0000 (00:43 -0800)] 
DL-132: Enable check style for distributedlog service module.
Author: Xi Liu <xiliuant@gmail.com>
Reviewers: Sijie Guo <sijie@apache.org>
Closes #89 from xiliuant/xi/checkstyle_service
xieliang [Wed, 4 Jan 2017 08:41:56 +0000 (00:41 -0800)]
DL-140: Fix distributedlog-core findbug inconsistent synchronization warings
Author: xieliang <xieliang007@gmail.com>
Reviewers: Leigh Stewart <lstewart@apache.org>
Closes #71 from xieliang/DL-140
Sijie Guo [Wed, 4 Jan 2017 08:40:28 +0000 (00:40 -0800)]
Improve merge pull request scripts
- check if a pull request is approved or not
- reject merging if there is no approval
- handle cases that reviewers don't have name and email configured.
Author: Sijie Guo <sijie@apache.org>
Reviewers: Franck Cuny <fcuny@apache.org>
Closes #93 from sijie/sijie/improve_merge_scripts
Sijie Guo [Thu, 29 Dec 2016 23:16:20 +0000 (15:16 -0800)]
Address a few flaky test cases
projects / distributedlog.git / log
