incubator-crail.git
3 years ago[Documentation] clarify where to find tarball 49/head
Jonas Pfefferle [Tue, 23 Oct 2018 14:43:46 +0000 (16:43 +0200)] 
[Documentation] clarify where to find tarball

Clarify build instructions where to find tarball after a
successful build.

https://jira.apache.org/jira/browse/CRAIL-67

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
3 years ago[Documentation] fix path to crail script 47/head
Jonas Pfefferle [Tue, 23 Oct 2018 14:18:10 +0000 (16:18 +0200)] 
[Documentation] fix path to crail script

Fix path to crail script. The correct path is $CRAIL_HOME/bin/crail

https://jira.apache.org/jira/browse/CRAIL-66

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
3 years ago[maven-release-plugin] prepare for next development iteration
Jonas Pfefferle [Mon, 22 Oct 2018 12:54:09 +0000 (14:54 +0200)] 
[maven-release-plugin] prepare for next development iteration

3 years ago[maven-release-plugin] prepare release v1.1-rc2 v1.1-rc2
Jonas Pfefferle [Mon, 22 Oct 2018 12:53:33 +0000 (14:53 +0200)] 
[maven-release-plugin] prepare release v1.1-rc2

3 years ago[History]: rc2 fixes and improvements
Jonas Pfefferle [Mon, 22 Oct 2018 12:48:13 +0000 (14:48 +0200)] 
[History]: rc2 fixes and improvements

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
3 years ago[Documentation]: fixing JIRA-61 46/head
Animesh Trivedi [Mon, 22 Oct 2018 12:31:19 +0000 (14:31 +0200)] 
[Documentation]: fixing JIRA-61

write up about the release process.

Signed-off-by: Animesh Trivedi <atrivedi@apache.org>
3 years agoDocker: fix base image url/tag and DiSNI version 45/head
Jonas Pfefferle [Mon, 22 Oct 2018 08:42:21 +0000 (10:42 +0200)] 
Docker: fix base image url/tag and DiSNI version

Fix base image url/tag and update DiSNI version to match version
in pom file.

https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-58

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
3 years ago[maven-release-plugin] prepare for next development iteration
Jonas Pfefferle [Thu, 18 Oct 2018 14:50:16 +0000 (16:50 +0200)] 
[maven-release-plugin] prepare for next development iteration

3 years ago[maven-release-plugin] prepare release v1.1-rc1 v1.1-rc1
Jonas Pfefferle [Thu, 18 Oct 2018 14:49:41 +0000 (16:49 +0200)] 
[maven-release-plugin] prepare release v1.1-rc1

3 years agoDocumentation: Add programming documentation 44/head
Jonas Pfefferle [Thu, 18 Oct 2018 14:30:42 +0000 (16:30 +0200)] 
Documentation: Add programming documentation

Add documentation how to program against Crail.

https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-55

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
3 years agoHISTORY: bug fixes and improvements of v1.1
Jonas Pfefferle [Thu, 18 Oct 2018 13:42:20 +0000 (15:42 +0200)] 
HISTORY: bug fixes and improvements of v1.1

Add history of bug fixes and improvements of v1.1
release.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
3 years agoREADME: Top-level features 38/head
Jonas Pfefferle [Fri, 7 Sep 2018 14:32:28 +0000 (16:32 +0200)] 
README: Top-level features

Keep list of top level features. Change Contact -> Community.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
3 years agoREADME: remove content, add disclaimer
Jonas Pfefferle [Thu, 6 Sep 2018 11:20:08 +0000 (13:20 +0200)] 
README: remove content, add disclaimer

Remove content which is provided in readthedocs to avoid
keeping track of multiple location when updating the documentation.
Add a disclaimer that we are a incubator project.
Add badges for license and docs build status.

https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-36
https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-24

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
3 years ago[maven-release-plugin] prepare for next development iteration
Jonas Pfefferle [Thu, 18 Oct 2018 13:05:35 +0000 (15:05 +0200)] 
[maven-release-plugin] prepare for next development iteration

3 years ago[maven-release-plugin] prepare release v1.1-rc0 v1.1-rc0
Jonas Pfefferle [Thu, 18 Oct 2018 13:04:09 +0000 (15:04 +0200)] 
[maven-release-plugin] prepare release v1.1-rc0

3 years agoFix checkstyle xml header 43/head
Jonas Pfefferle [Thu, 18 Oct 2018 12:14:50 +0000 (14:14 +0200)] 
Fix checkstyle xml header

Move xml version to the top of the file to avoid illformed xml file.

https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-53

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
3 years agoPOM: use tag HEAD in scm configuration 42/head
Jonas Pfefferle [Thu, 18 Oct 2018 12:10:24 +0000 (14:10 +0200)] 
POM: use tag HEAD in scm configuration

Use tag HEAD in scm configuration to avoid version increments by
release plugin for rc candidates.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
3 years agoGenerate source tar files instead of zip
Jonas Pfefferle [Thu, 18 Oct 2018 11:20:03 +0000 (13:20 +0200)] 
Generate source tar files instead of zip

Generate source tar.gz files instead of zip files to be
consistent with the binary release.

https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-54

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
3 years agoAdd missing license headers 41/head
Jonas Pfefferle [Thu, 18 Oct 2018 10:45:51 +0000 (12:45 +0200)] 
Add missing license headers

Add missing license headers to docker files and exclude CREDITS file
from rat check.

https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-53

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoThis commit fixes two minor issues for the docker build.
Adrian Schuepbach [Wed, 26 Sep 2018 14:11:34 +0000 (14:11 +0000)] 
This commit fixes two minor issues for the docker build.

The first issue is that core-site.xml can be 0 length after
substituting variables, see:
https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-51

The second issue is that CRAIL_HOME points to a hardcoded version, see:
https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-52

To fix these issues, the commit makes the following changes:

- Read Crail version from toplevel pom.xml
- Move Crail build directory to /home/crail
- Point CRAIL_HOME to /home/crail
- Use template core-site.xml.env file and subsitute environment variables
  to a new core-site.xml file

This closes #40

Signed-off-by: Adrian Schuepbach <asq@apache.org>
4 years agoCheckstyle: rename style file 13/head
Jonas Pfefferle [Fri, 7 Sep 2018 14:06:37 +0000 (16:06 +0200)] 
Checkstyle: rename style file

Rename file style.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoCheckstyle plugin
Jonas Pfefferle [Fri, 7 Sep 2018 12:49:22 +0000 (14:49 +0200)] 
Checkstyle plugin

Move checkstyle plugin to parent pom.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoCheckstyle plugin with google style
Jonas Pfefferle [Wed, 28 Mar 2018 20:54:19 +0000 (22:54 +0200)] 
Checkstyle plugin with google style

Add checkstyle plugin but do not check on compilation yet.
Checkstyle can be run with 'mvn checkstyle:check'

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-19

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoApache parent pom 37/head
Jonas Pfefferle [Tue, 21 Aug 2018 09:36:53 +0000 (11:36 +0200)] 
Apache parent pom

Use Apache parent pom. It already includes all necessary pieces
to do source and binary releases and push to maven central.

https://jira.apache.org/jira/projects/CRAIL/issues/CRAIL-34

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoDocumentation: remove logo 34/head
Jonas Pfefferle [Tue, 4 Sep 2018 09:01:56 +0000 (11:01 +0200)] 
Documentation: remove logo

Remove Crail logo from documentation.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoDocumentation: Spark-IO
Jonas Pfefferle [Wed, 15 Aug 2018 09:15:36 +0000 (11:15 +0200)] 
Documentation: Spark-IO

Add Spark-IO documentation: building and configuration.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoDocumentation: HDFS Adapter
Jonas Pfefferle [Wed, 15 Aug 2018 08:45:54 +0000 (10:45 +0200)] 
Documentation: HDFS Adapter

Add Spark HDFS Adapater documentation.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoDocumentation: iobench & fsck
Jonas Pfefferle [Tue, 14 Aug 2018 09:26:08 +0000 (11:26 +0200)] 
Documentation: iobench & fsck

Add iobench and fsck documentation.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoDocumentation: readthedocs Documentation
Jonas Pfefferle [Fri, 10 Aug 2018 08:25:01 +0000 (10:25 +0200)] 
Documentation: readthedocs Documentation

New Documentation using readthedocs based on github README.
Extended and adapted to latest Crail version.
Will be hosted at http://incubator-crail.readthedocs.io

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-24

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoAdjusting to latest DiSNI/DaRPC (v1.7) 36/head
Patrick Stuedi [Tue, 4 Sep 2018 14:32:54 +0000 (16:32 +0200)] 
Adjusting to latest DiSNI/DaRPC (v1.7)

https://issues.apache.org/jira/browse/CRAIL-50

Signed-off-by: Patrick Stuedi <pstuedi@apache.com>
4 years agoDocker: fix cache hugepage requirement 33/head
Jonas Pfefferle [Fri, 3 Aug 2018 08:36:57 +0000 (10:36 +0200)] 
Docker: fix cache hugepage requirement

Allow users to use the Crail shell without providing
hugepages via a volume mount to Docker.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoDocker build: TCP and RDMA tier
Jonas Pfefferle [Thu, 26 Jul 2018 11:35:25 +0000 (13:35 +0200)] 
Docker build: TCP and RDMA tier

Docker build files for the TCP and RDMA tier.
User can extend configuration to use NVMf or other tiers.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-49

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNVMf: new jNVMf version 1.4 32/head
Jonas Pfefferle [Thu, 5 Jul 2018 11:43:52 +0000 (13:43 +0200)] 
NVMf: new jNVMf version 1.4

Use new jNVMf to fix dependency issue with DiSNI 1.6.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoThrow more meaningful exception if datapath is invalid or has no write 31/head
Patrick Stuedi [Tue, 26 Jun 2018 08:43:04 +0000 (10:43 +0200)] 
Throw more meaningful exception if datapath is invalid or has no write
permissions.

Fixes the issue of invalid datapath or missing write permissions on the
datapath for both the RDMA and the TCP storage tier.

https://issues.apache.org/jira/browse/CRAIL-45

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
4 years agoMake sure statistics are updated across all classes in the Crail HDFS 30/head
Patrick Stuedi [Mon, 25 Jun 2018 11:46:45 +0000 (13:46 +0200)] 
Make sure statistics are updated across all classes in the Crail HDFS
module

Previously statistics where only captured in HDFSOutputStream and parts
of the CrailHadoopFileSystem.

https://issues.apache.org/jira/browse/CRAIL-26

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
4 years agoDelegate implementation of CrailHDFS to CrailHadoopFileSystem
Patrick Stuedi [Mon, 25 Jun 2018 11:16:01 +0000 (13:16 +0200)] 
Delegate implementation of CrailHDFS to CrailHadoopFileSystem

Implement AbstractFileSystem by delegating function calls to
CrailHadoopFileSystm. The same can be done also by using
DelegateToFileSystem from hadoop-common, but this results in additional
operations (e.g., exists(path) and checkPath(path)).

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-27

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
4 years agoAdding extra config parameters to control backlog and connnection 28/head
Patrick Stuedi [Fri, 22 Jun 2018 09:37:51 +0000 (11:37 +0200)] 
Adding extra config parameters to control backlog and connnection
timeouts for RDMA-based RPC and storage backends

The backlog parameter controlls the backlog of incoming connection
requests for rdma storage and rdma RPC. The connectiontimeout puts a
limit on the time to wait for the connection establishement event.

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
4 years agoAdjustments to comply with DiSNI v1.6 and DaRPC v1.6
Patrick Stuedi [Thu, 21 Jun 2018 14:40:26 +0000 (16:40 +0200)] 
Adjustments to comply with DiSNI v1.6 and DaRPC v1.6

The connect/bind API of DiSNI version 1.6 has changed, affecting
rpc-darpc and storage-rdma.

https://issues.apache.org/jira/browse/CRAIL-41

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
4 years agoScripts: fix log/pid file for multiple datanodes 27/head
Jonas Pfefferle [Wed, 20 Jun 2018 13:45:47 +0000 (15:45 +0200)] 
Scripts: fix log/pid file for multiple datanodes

Fix using the same filename when starting multiple storage
tiers on the same node. Append arguments from slave file
to make path unique.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-29

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoScripts: add crail-env.sh 26/head
Jonas Pfefferle [Wed, 20 Jun 2018 14:52:46 +0000 (16:52 +0200)] 
Scripts: add crail-env.sh

Add crail-env.sh file to allow setting parameters for
the start scripts like extra java options.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoStart script: remove heap size argument
Jonas Pfefferle [Wed, 20 Jun 2018 11:36:11 +0000 (13:36 +0200)] 
Start script: remove heap size argument

Remove Xmx and MaxDirectMemorySize argument from start script
since it does not allow to run Crail on machines with less
than 64G.

https://issues.apache.org/jira/browse/CRAIL-40

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoCheck for already existing cache directory with the same ID 25/head
Patrick Stuedi [Wed, 20 Jun 2018 13:06:02 +0000 (15:06 +0200)] 
Check for already existing cache directory with the same ID

Throw an exception if a cache directory for the same ID
(reflecting a particular instance of CrailStore) already exists

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
4 years agoIntroducing StorageUtils as a common ground for storage utility
Patrick Stuedi [Wed, 20 Jun 2018 12:11:04 +0000 (14:11 +0200)] 
Introducing StorageUtils as a common ground for storage utility
functions

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
4 years agoMake default configuration easier to run
Patrick Stuedi [Tue, 19 Jun 2018 13:21:07 +0000 (15:21 +0200)] 
Make default configuration easier to run

Better default values for cachepath and tcp storage path. Throw an
exception if paths are invalid. Throw an exception if interface is
invalid. Trim down crail-site.conf.template to only include the
necessary parameters.

https://issues.apache.org/jira/browse/CRAIL-15

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
4 years agoMove release date in HISTORY 24/head v1.0
Jonas Pfefferle [Tue, 15 May 2018 11:04:26 +0000 (13:04 +0200)] 
Move release date in HISTORY

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoApache incubator disclaimer
Jonas Pfefferle [Mon, 14 May 2018 11:02:57 +0000 (13:02 +0200)] 
Apache incubator disclaimer

Add Aapache incubator disclaimer.

https://issues.apache.org/jira/browse/CRAIL-33

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoMove IBM copyright NOTICE to CREDITS file
Jonas Pfefferle [Mon, 14 May 2018 10:59:28 +0000 (12:59 +0200)] 
Move IBM copyright NOTICE to CREDITS file

Move the IBM copyright notice from the NOTICE file to the CREDITS file.

https://issues.apache.org/jira/browse/CRAIL-33

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNVMf: decrease keep alive interval 22/head
Jonas Pfefferle [Thu, 3 May 2018 11:37:44 +0000 (13:37 +0200)] 
NVMf: decrease keep alive interval

Set keep alive interval to lower value to not run into the
situation where we missed the keep alive deadline and the connection
is shutdown by the controller.

https://issues.apache.org/jira/browse/CRAIL-32

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNVMf: fix staging buffer cache get race condition 23/head
Jonas Pfefferle [Mon, 7 May 2018 07:16:22 +0000 (09:16 +0200)] 
NVMf: fix staging buffer cache get race condition

Fixes NVMf staging buffer cache get. Do not throw an exception
if we can't free the previous buffer in the hashmap. If we can't
free it it was already freed by allocateFreeBuffers.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNVMf: staging buffer cache clear buffer
Jonas Pfefferle [Thu, 3 May 2018 11:38:00 +0000 (13:38 +0200)] 
NVMf: staging buffer cache clear buffer

Clear buffer position and limit when getting the buffer of the
free queue since position and limit might have been modified before.

https://issues.apache.org/jira/browse/CRAIL-31

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNVMf: staging buffer cache existing entry bug
Jonas Pfefferle [Wed, 2 May 2018 10:53:47 +0000 (12:53 +0200)] 
NVMf: staging buffer cache existing entry bug

A buffer might already exist at a certain remote address (resp. LBA) if
a file was deleted and a block was reassigned to a different file. Do not
throw an exception but free the buffer. If we can't free it something
went wrong as there should never be two writers writing to the same
remote address.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-30

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoREADME: give all three options to download src 21/head
Jonas Pfefferle [Mon, 23 Apr 2018 10:54:26 +0000 (12:54 +0200)] 
README: give all three options to download src

Include all three options to download the Crail source:
Apache, github and Apache source distro.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-24

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoMove HISTORY release date
Jonas Pfefferle [Mon, 23 Apr 2018 09:21:00 +0000 (11:21 +0200)] 
Move HISTORY release date

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoREADME: build from source distro
Jonas Pfefferle [Mon, 23 Apr 2018 09:19:27 +0000 (11:19 +0200)] 
README: build from source distro

Add url for source distro and build instructions.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-24

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoCopyright NOTICE: change to use actual year
Jonas Pfefferle [Mon, 23 Apr 2018 09:11:41 +0000 (11:11 +0200)] 
Copyright NOTICE: change to use actual year

Change copyright notice for the Apache Software Foundation
to use the actual year instead of "onwards".

https://issues.apache.org/jira/browse/CRAIL-28

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoMove release date in HISTORY 20/head
Jonas Pfefferle [Wed, 18 Apr 2018 07:45:15 +0000 (09:45 +0200)] 
Move release date in HISTORY

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoAdd license header to md files
Jonas Pfefferle [Mon, 16 Apr 2018 15:37:16 +0000 (17:37 +0200)] 
Add license header to md files

Add license header to md files and remove exclude from rat.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoMove copyright notice to NOTICE file.
Jonas Pfefferle [Mon, 16 Apr 2018 15:31:10 +0000 (17:31 +0200)] 
Move copyright notice to NOTICE file.

Remove copyright notice from license headers and move it
to NOTICE file.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoFix LICENSE file
Jonas Pfefferle [Fri, 13 Apr 2018 08:56:04 +0000 (10:56 +0200)] 
Fix LICENSE file

LICENSE file should only include licenses that we use.
Now only includes Apache License 2.0

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoCheck licenses with apache-rat
Jonas Pfefferle [Fri, 13 Apr 2018 08:53:59 +0000 (10:53 +0200)] 
Check licenses with apache-rat

Add apache-rat maven plugin to allow to check licenses with
mvn apache-rat:check

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoAdd missing license headers
Jonas Pfefferle [Fri, 13 Apr 2018 08:52:48 +0000 (10:52 +0200)] 
Add missing license headers

https://issues.apache.org/jira/browse/CRAIL-25

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agov1.0 release history
Jonas Pfefferle [Thu, 12 Apr 2018 08:58:13 +0000 (10:58 +0200)] 
v1.0 release history

Close #19

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoREADME: update NVMf tier constants
Jonas Pfefferle [Tue, 10 Apr 2018 13:06:19 +0000 (15:06 +0200)] 
README: update NVMf tier constants

Update NVMf tier constants in README.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-24

Close #18

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNew jNVMf version 1.3 17/head
Jonas Pfefferle [Tue, 10 Apr 2018 12:12:05 +0000 (14:12 +0200)] 
New jNVMf version 1.3

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNew DiSNI and DaRPC version
Jonas Pfefferle [Tue, 10 Apr 2018 12:03:32 +0000 (14:03 +0200)] 
New DiSNI and DaRPC version

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoCrail internal version to 3100
Jonas Pfefferle [Tue, 10 Apr 2018 11:08:59 +0000 (13:08 +0200)] 
Crail internal version to 3100

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfUnalignedWriteFuture: refactoring
Jonas Pfefferle [Tue, 10 Apr 2018 09:08:50 +0000 (11:08 +0200)] 
NvmfUnalignedWriteFuture: refactoring

Use offsetInSector in floorToSector and provide a new
floorToSector overload method for ints.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfStorageEndpoint: Number of logical blocks fix type
Jonas Pfefferle [Tue, 10 Apr 2018 09:03:24 +0000 (11:03 +0200)] 
NvmfStorageEndpoint: Number of logical blocks fix type

jNVMf API has been changed to take int instead of long, fix code
accordingly.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfStagingBufferCache: enhance excpetion message
Jonas Pfefferle [Tue, 10 Apr 2018 09:00:06 +0000 (11:00 +0200)] 
NvmfStagingBufferCache: enhance excpetion message

Add values of illegal arguments to exception message to make
it easier for the user to debug.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfStorageConstants: Allocation size integer
Jonas Pfefferle [Tue, 10 Apr 2018 08:56:37 +0000 (10:56 +0200)] 
NvmfStorageConstants: Allocation size integer

Change allocationsize constant to be an integer. Since we do not
support allocation sizes > MAXINT.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfStorageClient: isValid -> isAlive
Jonas Pfefferle [Tue, 10 Apr 2018 08:51:08 +0000 (10:51 +0200)] 
NvmfStorageClient: isValid -> isAlive

Change method name from isValid to isAlive. isValid can be
misinterpreted by the users that he misconfigured the system.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfStorageClient: getEndpointGroup private
Jonas Pfefferle [Tue, 10 Apr 2018 08:48:23 +0000 (10:48 +0200)] 
NvmfStorageClient: getEndpointGroup private

Make getEndpointGroup private, it should not be used outside of
this class and does not handle concurrent execution.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmf Tier: keep alive timer constant
Jonas Pfefferle [Tue, 10 Apr 2018 08:46:03 +0000 (10:46 +0200)] 
Nvmf Tier: keep alive timer constant

Introduce new keep alive timer constant. For now we don't make
this configurable since jNVMf does not allow to change it yet, i.e.
it tells the target to use the default 2 minutes keep alive timer.
We set it to 110 seconds to not miss the 2 minutes deadline.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfStorageClient: number of LBAs
Jonas Pfefferle [Mon, 9 Apr 2018 13:24:41 +0000 (15:24 +0200)] 
NvmfStorageClient: number of LBAs

Number of LBAs in new jNVMf API is no longer 0-based.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfRegisteredBufferCache: simplify
Jonas Pfefferle [Mon, 9 Apr 2018 13:22:41 +0000 (15:22 +0200)] 
NvmfRegisteredBufferCache: simplify

Simplify buffer cache by do not hold references to CrailBuffers.
We only return the corresponding key now instead of a wrapped
KeyedNativeBuffer.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfStorageEndpoint: too many outstanding requests
Jonas Pfefferle [Mon, 9 Apr 2018 13:20:38 +0000 (15:20 +0200)] 
NvmfStorageEndpoint: too many outstanding requests

Fix bug were there could be too many outstanding requests posted
on the NVMf SQ.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfFuture: fix concurrency bug
Jonas Pfefferle [Mon, 9 Apr 2018 13:18:40 +0000 (15:18 +0200)] 
NvmfFuture: fix concurrency bug

Fix bug when completed counter was incremented in parallel
and never reached 2.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfStorageClient: keep alive thread
Jonas Pfefferle [Mon, 9 Apr 2018 13:16:49 +0000 (15:16 +0200)] 
NvmfStorageClient: keep alive thread

Start keep alive thread which sends keep alive message to
each controller every 110s (timeout 120s). Otherwise thread sleeps.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfStagingBufferCache: fix allocation
Jonas Pfefferle [Mon, 9 Apr 2018 13:13:57 +0000 (15:13 +0200)] 
NvmfStagingBufferCache: fix allocation

Fix allocation of Crail buffers to slice correct amount
of buffers for staging.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNvmfStorageTier: fix to work with new jNVMf API
Jonas Pfefferle [Wed, 4 Apr 2018 14:32:01 +0000 (16:32 +0200)] 
NvmfStorageTier: fix to work with new jNVMf API

Minor changes to work with latest jNVMf version.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNew NVMf storage tier: use jNVMf library
Jonas Pfefferle [Thu, 1 Mar 2018 16:29:34 +0000 (17:29 +0100)] 
New NVMf storage tier: use jNVMf library

New NVMf storage tier implementation which uses jNVMf library instead
of SPDK. We do not implement unaligned reads anymore since we believe
the semantics of the underlying storage system should not be hidden like
this. We guarantee good performance when using buffered streams since
they now try to align accesses whenever possible.

https://issues.apache.org/jira/browse/CRAIL-22

Close #16

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoCrailStore: refactor alloc/free buffer interface
Jonas Pfefferle [Wed, 4 Apr 2018 08:52:46 +0000 (10:52 +0200)] 
CrailStore: refactor alloc/free buffer interface

Do not implement CrailBufferCache but instead use
abstract method to define allocate/free buffer interface.
Pass BufferCache instead of CrailStore to StorageClient init.

Close #14

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoBufferCache: use new interface in buffer cache
Jonas Pfefferle [Tue, 3 Apr 2018 13:50:59 +0000 (15:50 +0200)] 
BufferCache: use new interface in buffer cache

Use newly introduced buffer cache interface in buffer cache
implementation.

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoStorageClient: provide statistics and buffer cache
Jonas Pfefferle [Thu, 29 Mar 2018 12:27:37 +0000 (14:27 +0200)] 
StorageClient: provide statistics and buffer cache

Initialize the storage client with statistics and the buffer cache
to be used in the client.

https://issues.apache.org/jira/projects/CRAIL/issues/CRAIL-20

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoBufferedOutputStream: align CoreStream position
Jonas Pfefferle [Tue, 27 Mar 2018 11:53:37 +0000 (13:53 +0200)] 
BufferedOutputStream: align CoreStream position

After reopening an BufferedOutputStream the underlying CoreStream
might be at an arbitrary position since the stream is purged on close.
To avoid unnecessary operations on the underlying stream,
e.g. because of slices crossing the block boundary we adjust
the length of the first slice to align the position to slice size
such that after the first write all following writes on the underlying
stream will be aligned to slice size again. This also helps to avoid
unaligned writes in block storage tiers.

https://issues.apache.org/jira/browse/CRAIL-17

Close #11

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoInclude test-jars and dependencies in assembly
Jonas Pfefferle [Wed, 28 Mar 2018 12:25:35 +0000 (14:25 +0200)] 
Include test-jars and dependencies in assembly

Add test-jars and test dependencies to jars directory.
Add test option to bin/crail to allow run unit tests from
command line.

https://issues.apache.org/jira/browse/CRAIL-18

Close #12

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoConfiguration: environment variable expansion
Radu Stoica [Tue, 27 Feb 2018 14:09:45 +0000 (15:09 +0100)] 
Configuration: environment variable expansion

Enable environment variable expansion in crail-site.conf.

Close #7

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoFixes issue with local short circuit (memcpy) path in the RDMA storage
Patrick Stuedi [Wed, 28 Feb 2018 08:41:47 +0000 (09:41 +0100)] 
Fixes issue with local short circuit (memcpy) path in the RDMA storage
tier
- mmaped buffers were previously indexed with wrong LBA in
  RdmaStorageLocalEndpoint

This fixes the CRAIL-3 JIRA ticket
https://issues.apache.org/jira/browse/CRAIL-3

Close #8

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
Signed-off-by: Adrian Schuepbach <asq@apache.org>
4 years agoApplications can now choose on a per node basis (files, directories,
Patrick Stuedi [Fri, 23 Feb 2018 14:44:49 +0000 (15:44 +0100)] 
Applications can now choose on a per node basis (files, directories,
tables, key/value pairs, etc.) whether the node should be enumerable or
not. If a node is enumerable it will be included in an enumeration
operation of the parent node.

https://issues.apache.org/jira/browse/CRAIL-10

Close #6

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoNarpc: enable multiple core dispatcher
Patrick Stuedi [Fri, 23 Feb 2018 12:59:02 +0000 (13:59 +0100)] 
Narpc: enable multiple core dispatcher

Enable multi core dispatcher if configured to use NaRPC (RPC and
Storage).

Fixes CRAIL-9 JIRA ticket

Close #5

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years agoDataNodeInfo.key() was not updated when the object got updated, this is
Patrick Stuedi [Thu, 22 Feb 2018 15:59:08 +0000 (16:59 +0100)] 
DataNodeInfo.key() was not updated when the object got updated, this is
fixed now.

https://issues.apache.org/jira/browse/CRAIL-8

Close #4

Signed-off-by: Jonas Pfefferle <pepperjo@apache.org>
4 years ago[BUILD] Moving the default build to hadoop 2.7.3
Animesh Trivedi [Wed, 21 Feb 2018 12:56:00 +0000 (13:56 +0100)] 
[BUILD] Moving the default build to hadoop 2.7.3

Close #3

Signed-off-by: Animesh Trivedi <atrivedi@apache.org>
4 years agoUpdating README.md, how to contribute to Crail
Animesh Trivedi [Wed, 21 Feb 2018 12:46:27 +0000 (13:46 +0100)] 
Updating README.md, how to contribute to Crail

The website is also updated.

This closes #2

Signed-off-by: Animesh Trivedi <atrivedi@apache.org>
4 years agoFixing CRAIL-4 JIRA ticket
Animesh Trivedi [Wed, 7 Feb 2018 12:34:00 +0000 (13:34 +0100)] 
Fixing CRAIL-4 JIRA ticket

readShort and readLong functions by did not return
the expected types. Now that is fixed in the function
signature.

This closes #1

Signed-off-by: Patrick Stuedi <pstuedi@apache.org>
4 years agoAdjusting README (github links, source code links)
Patrick Stuedi [Mon, 22 Jan 2018 13:59:46 +0000 (14:59 +0100)] 
Adjusting README (github links, source code links)

4 years agoFixes bug when using the TCP RPC binding together with multiple namenodes
Patrick Stuedi [Fri, 19 Jan 2018 12:06:05 +0000 (13:06 +0100)] 
Fixes bug when using the TCP RPC binding together with multiple namenodes

4 years agoMoving to DaRPC and DiSNI v1.4
Patrick Stuedi [Thu, 18 Jan 2018 15:35:00 +0000 (16:35 +0100)] 
Moving to DaRPC and DiSNI v1.4

4 years agoSupport configuration with cachelimit=0
Patrick Stuedi [Thu, 18 Jan 2018 13:40:03 +0000 (14:40 +0100)] 
Support configuration with cachelimit=0

4 years agoAdjusting license and configuration (Apache)
Patrick Stuedi [Thu, 18 Jan 2018 13:27:40 +0000 (14:27 +0100)] 
Adjusting license and configuration (Apache)

4 years agoRenaming artifacts (Apache)
Patrick Stuedi [Thu, 18 Jan 2018 12:01:50 +0000 (13:01 +0100)] 
Renaming artifacts (Apache)