2 years agoAllow setting some default transaction options on the db handle
Nick Vatamaniuc [Thu, 23 Apr 2020 04:48:48 +0000 (00:48 -0400)] 
Allow setting some default transaction options on the db handle

Since FDB version 6.1 [1] it is possible to set default transaction
options on database handles, and any transactions created from that
handle will inherit those options.


2 years agoAdd .asf.yaml file for commits and notifications
Nick Vatamaniuc [Wed, 22 Apr 2020 18:54:10 +0000 (14:54 -0400)] 
Add .asf.yaml file for commits and notifications

Instructions from [Apache Infra](

2 years agoDo not commit read-only transactions
Nick Vatamaniuc [Wed, 22 Apr 2020 16:25:31 +0000 (12:25 -0400)] 
Do not commit read-only transactions

This might save a round-trip to the network thread [1]. It also follows the
recommendation in the C api docs [2].


However, it turns out in order for the watches to fire the read-only
transaction still has to commit, so avoid this optimization if has_watches(Tx)
is true.

2 years agoImplement has_watches(Tx) transaction flag
Nick Vatamaniuc [Wed, 22 Apr 2020 16:24:42 +0000 (12:24 -0400)] 
Implement has_watches(Tx) transaction flag

Returns true if transaction has created any watches. If writes are disallowed,
watches are also cannot be created.

2 years agoMerge pull request #1 from apache/dont-create-new-tx-object-on-retries
Nick Vatamaniuc [Sat, 18 Apr 2020 20:49:13 +0000 (16:49 -0400)] 
Merge pull request #1 from apache/dont-create-new-tx-object-on-retries

Do not create a new transaction object during retries

2 years agoDo not create a new transaction object during retries 1/head
Nick Vatamaniuc [Sat, 18 Apr 2020 19:16:23 +0000 (15:16 -0400)] 
Do not create a new transaction object during retries

`on_error` function automatically resets the transaction object but
preserves retry and backoff-related information in the object. Avoid
creating a new transaction object which would lose that context in
case transactions retry more than 2 times.


2 years agoMerge pull request #17 from cloudant-labs/encode-lists-like-binaries v1.0.0
Kyle Snavely [Fri, 27 Mar 2020 16:47:01 +0000 (12:47 -0400)] 
Merge pull request #17 from cloudant-labs/encode-lists-like-binaries

erlfdb_tuple:encode/2 should handle lists through binary conversion

2 years agoerlfdb_tuple:encode/2 should provide a clear error message when provided
Kyle Snavely [Thu, 26 Mar 2020 02:48:37 +0000 (20:48 -0600)] 
erlfdb_tuple:encode/2 should provide a clear error message when provided
with a bad term to encode

2 years agoAdd transaction option to disallow writes
Nick Vatamaniuc [Sat, 25 Jan 2020 04:03:28 +0000 (23:03 -0500)] 
Add transaction option to disallow writes

Add `disallow_writes | allow_writes` transaction options.

If the transaction hasn't done any writes yet, it is possible to set the
`disallow_writes` option, and then every write attempt will throw a
`writes_not_allowed` error.

2 years agoBump API level to 620
Nick Vatamaniuc [Mon, 6 Jan 2020 20:56:29 +0000 (15:56 -0500)] 
Bump API level to 620

This comes with a few changes:

 * `fdb_future_get_version` was renamed to `fdb_future_get_int64`
 * a new API: `fdb_transaction_get_approximate_size`
 * `size_limit` transaction option


2 years agoMerge pull request #9 from cloudant-labs/add-get-error-string
Nick Vatamaniuc [Mon, 25 Nov 2019 17:05:41 +0000 (12:05 -0500)] 
Merge pull request #9 from cloudant-labs/add-get-error-string

Expose error string description function in the API

2 years agoExpose error string description function in the API
Nick Vatamaniuc [Mon, 25 Nov 2019 17:00:34 +0000 (12:00 -0500)] 
Expose error string description function in the API

2 years agoAdd instructions on running the fdb bindingtester
Paul J. Davis [Tue, 29 Oct 2019 21:40:36 +0000 (16:40 -0500)] 
Add instructions on running the fdb bindingtester

2 years agoMerge pull request #6 from cloudant-labs/update-api-level-to-610
Nick Vatamaniuc [Tue, 29 Oct 2019 21:33:37 +0000 (17:33 -0400)] 
Merge pull request #6 from cloudant-labs/update-api-level-to-610

Switch API level to 610

2 years agoSwitch API level to 610
Nick Vatamaniuc [Mon, 28 Oct 2019 23:03:37 +0000 (19:03 -0400)] 
Switch API level to 610

There were some changes that had to be accounted for:

 * Cluster is gone, so cluster futures, resources, etc were removed

 * Create database returns a database object not future, so database future
   type is removed.

 * It's possible to use json tracing format on the client, so added that
   netowrk option.

2 years agoAdd a debug utility function
Paul J. Davis [Tue, 29 Oct 2019 21:14:04 +0000 (16:14 -0500)] 
Add a debug utility function

2 years agoEnsure intialization of offset
Paul J. Davis [Tue, 29 Oct 2019 21:12:47 +0000 (16:12 -0500)] 
Ensure intialization of offset

If we don't initialize offset here its possible that a key selector
could apply the provided offset to an uninitialized value passed in from
the calling function.

3 years agoAdding a better README
Paul J. Davis [Wed, 25 Sep 2019 15:44:20 +0000 (10:44 -0500)] 
Adding a better README

3 years agoReplace get_range_raw with fold_range_future/wait
Paul J. Davis [Tue, 6 Aug 2019 16:47:40 +0000 (11:47 -0500)] 
Replace get_range_raw with fold_range_future/wait

This has been bugging me for awhile in that we're not properly handling
the return value from the range future. The `more` return value in
particular was ignored.

Rather than force people to have to repeatedly reimplement this I've
changed `get_range_raw` with a pair of functions `fold_range_future` and
`fold_range_wait`. This allows us the benefit of the async approach
while also ensuring we're not dealing with partial ranges being returned
to users.

3 years agoMerge pull request #5 from cloudant-labs/improve-message-flushing
Nick Vatamaniuc [Mon, 1 Jul 2019 17:17:05 +0000 (13:17 -0400)] 
Merge pull request #5 from cloudant-labs/improve-message-flushing

Improve message flushing

3 years agoImprove message flushing when waiting for a future
Nick Vatamaniuc [Thu, 20 Jun 2019 21:50:50 +0000 (17:50 -0400)] 
Improve message flushing when waiting for a future

Re-use the same function as cancel/2. Also, flush after getting the result to
decreases the change of the message being missed.

3 years agoAdd cancel/2 function, which takes an options argument
Nick Vatamaniuc [Thu, 20 Jun 2019 21:43:46 +0000 (17:43 -0400)] 
Add cancel/2 function, which takes an options argument

When futures are canceled, they may have a `ready` message already in the
mailbox. There is now a way to flush that message with the `[flush]` option.
That is currently the only option.

3 years agoRemember to free the msg_env on future destruction
Paul J. Davis [Thu, 20 Jun 2019 20:59:34 +0000 (15:59 -0500)] 
Remember to free the msg_env on future destruction

3 years agoMark futures as cancelled
Paul J. Davis [Thu, 20 Jun 2019 15:16:11 +0000 (10:16 -0500)] 
Mark futures as cancelled

This allows callers to cancel futures without having to worry about
receiving a `ready` message if the future completes before the
cancellation is applied.

Its possible that a message for this already in the callers message
queue. In which case you can call `cancel(Future, [flush])` to remove
that message.

3 years agoFix reference counting of ErlFDBFuture resources
Paul J. Davis [Thu, 20 Jun 2019 14:45:40 +0000 (09:45 -0500)] 
Fix reference counting of ErlFDBFuture resources

I forgot to account for the fact that we need to keep a reference to the
ErlFDBFuture resources while they are being processed by the network
thread. This change accounts for that fact.

3 years agoMake erlfdb_get_key_selector more user friendly
Paul J. Davis [Fri, 14 Jun 2019 19:16:33 +0000 (14:16 -0500)] 
Make erlfdb_get_key_selector more user friendly

Previously it was not possible to combine the `lt`/`lteq`/`gt`/`gteq`
behaviors with an offset without duplicating all of the logic in
`erlfdb_get_key_selector`. This changes that so a skip can be applied
onto any of those atom based selectors.

3 years agoAdd get_next_tx_id API
Paul J. Davis [Thu, 2 May 2019 17:36:14 +0000 (12:36 -0500)] 
Add get_next_tx_id API

This API provides a transaction ID that can be used in the 12 byte
version stamps to distinguish between version stamps created in a single
transaction. There's a limit of 65,535 transaction ids afterwhich this
function starts throwing badarg exceptions.

3 years agoExpose get_raw_range
Paul J. Davis [Thu, 2 May 2019 17:21:17 +0000 (12:21 -0500)] 
Expose get_raw_range

This allows clients to access the raw futures for range requests.
Clients of this API are then required to manager their own iteration
state and continuation logic.

3 years agoFix get_last_error/1
Paul J. Davis [Fri, 26 Apr 2019 18:30:50 +0000 (13:30 -0500)] 
Fix get_last_error/1

Make sure to use the correct `get/1` function.

3 years agoAdd utilities for creating test clusters
Paul J. Davis [Fri, 26 Apr 2019 17:55:46 +0000 (12:55 -0500)] 
Add utilities for creating test clusters

These utilities allow us to setup a test cluster for use during unit
tests. By default this will setup a cluster once per VM and also re-use
state from old test runs if it exists. Tests should ensure an empty
cluster when necessary by specifying the `empty` option when calling

3 years agoFlush `ready` message if already sent
Nick Vatamaniuc [Mon, 22 Apr 2019 20:22:44 +0000 (16:22 -0400)] 
Flush `ready` message if already sent

3 years agoRelease future resources properly
Paul J. Davis [Fri, 19 Apr 2019 16:01:39 +0000 (11:01 -0500)] 
Release future resources properly

This prevented all future objects from being properly garbage collected.

3 years agoAdd an API for getting the previous error
Paul J. Davis [Thu, 18 Apr 2019 14:21:56 +0000 (09:21 -0500)] 
Add an API for getting the previous error

3 years agoExpose is_read_only/1 API
Paul J. Davis [Tue, 16 Apr 2019 20:51:47 +0000 (15:51 -0500)] 
Expose is_read_only/1 API

3 years agoStore the error code for clients
Paul J. Davis [Fri, 12 Apr 2019 18:15:27 +0000 (13:15 -0500)] 
Store the error code for clients

Clients of the transactional API may need to refer to which error
occurred in order to react appropriately on transaction retries.

3 years agoAdd tracking of read only transactions
Paul J. Davis [Fri, 12 Apr 2019 17:53:08 +0000 (12:53 -0500)] 
Add tracking of read only transactions

3 years agoOnly unpack the tail after the provided prefix
Paul J. Davis [Tue, 26 Mar 2019 22:08:43 +0000 (17:08 -0500)] 
Only unpack the tail after the provided prefix

3 years agoExport unpack/2
Paul J. Davis [Tue, 26 Mar 2019 22:08:06 +0000 (17:08 -0500)] 
Export unpack/2

3 years agoAllow specifying a prefix for unpack and range
Paul J. Davis [Tue, 26 Mar 2019 20:33:06 +0000 (15:33 -0500)] 
Allow specifying a prefix for unpack and range

This pretty much removes any usefulness from `erlfdb_subspace` since
Erlang doesn't support operator overrides or other similar syntactic
sugar that makes subspaces useful in other languages.

3 years agoUpdate
Kyle Snavely [Fri, 22 Mar 2019 15:02:07 +0000 (11:02 -0400)] 

Just small typos I observed on reviewing the notes.

3 years agoMatch Python catching all exceptions
Paul J. Davis [Mon, 25 Feb 2019 16:52:49 +0000 (10:52 -0600)] 
Match Python catching all exceptions

3 years agoImplement the DirectoryLayer
Paul J. Davis [Fri, 22 Feb 2019 17:19:36 +0000 (11:19 -0600)] 
Implement the DirectoryLayer

This implements a DirectoryLayer intends to behave identically to the
Python version. Soak tests have so far run as long as an hour without
finding issues. However I plan on trying to get that number to upwards
of 24 hours or more.

3 years agoFix wait_for_empty calls in concurrent testing
Paul J. Davis [Fri, 22 Feb 2019 17:13:52 +0000 (11:13 -0600)] 
Fix wait_for_empty calls in concurrent testing

3 years agoUse the new snapshot object for API tests
Paul J. Davis [Fri, 22 Feb 2019 17:12:21 +0000 (11:12 -0600)] 
Use the new snapshot object for API tests

3 years agoFix test runs with concurrency
Paul J. Davis [Fri, 22 Feb 2019 17:10:17 +0000 (11:10 -0600)] 
Fix test runs with concurrency

3 years agoSimplify clearing the stack
Paul J. Davis [Fri, 22 Feb 2019 17:09:01 +0000 (11:09 -0600)] 
Simplify clearing the stack

Killing the process made for some dirty log output since the exit wasn't
handled cleanly. Rather than trying to fix that we just add a message to
reset the stack state inside the same pid.

3 years agoMove py_repr to erlfdb_util:repr
Paul J. Davis [Fri, 22 Feb 2019 17:04:11 +0000 (11:04 -0600)] 
Move py_repr to erlfdb_util:repr

This is a useful function to have when using the FoundationDB binding
tester in comparison to the Python implementation.

3 years agoFix the `contains` check for subspaces
Paul J. Davis [Fri, 22 Feb 2019 17:00:04 +0000 (11:00 -0600)] 
Fix the `contains` check for subspaces

Silly mistake reordering which value was tested for a prefix.

3 years agoAdd compatibility shim
Paul J. Davis [Fri, 22 Feb 2019 16:59:20 +0000 (10:59 -0600)] 
Add compatibility shim

This is just a compatibility shim to make the directory layer tests
slightly easier to implement.

3 years agoFix HCA module
Paul J. Davis [Fri, 22 Feb 2019 16:58:06 +0000 (10:58 -0600)] 
Fix HCA module

A few minor bug fixes found with testing the directory layer.

3 years agoAdd support for snapshots as a transaction object
Paul J. Davis [Fri, 22 Feb 2019 16:49:43 +0000 (10:49 -0600)] 
Add support for snapshots as a transaction object

The way that FoundationDB tests are written in OO languages more or less
requires that we have this concept so that we maintain the same read
semantics without having to add plumbing for options during testing.

3 years agoImprove handling of futures
Paul J. Davis [Fri, 22 Feb 2019 16:47:09 +0000 (10:47 -0600)] 
Improve handling of futures

This covers two cases where a future is either immediately available and
similarly when a future has already been waited on. The first case
appears to be mostly when reading a key that was either previously read
or set in the same transaction. The second case can happen when passing
the future around. Since we've already consumed the ready message we
just add a check for `is_ready`.

3 years agoFix tuple mutation macros
Paul J. Davis [Fri, 22 Feb 2019 16:45:25 +0000 (10:45 -0600)] 
Fix tuple mutation macros

This improves the consistency of the macro argument ordering and general
usage after having worked with it in the directory layer.

3 years agoReplace subspace usage with macros
Paul J. Davis [Fri, 15 Feb 2019 17:15:13 +0000 (11:15 -0600)] 
Replace subspace usage with macros

The subspace layer is really not super helpful in Erlang. In Pyhton the
subspace layer helps significantly by using magic functions and
inheritance. However, Erlang doesn't have similar syntactical niceties
so it ends up being more burdensome than just adding macros for tuple

If someone has a better idea I'm all ears on this as the macros do take
a decent level of knowledge on tuple packing to make use of.

3 years agoAdd a utility module for reusable functions
Paul J. Davis [Fri, 15 Feb 2019 17:14:33 +0000 (11:14 -0600)] 
Add a utility module for reusable functions

3 years agoImplement a high contention allocator
Paul J. Davis [Tue, 12 Feb 2019 17:40:19 +0000 (11:40 -0600)] 
Implement a high contention allocator

This allocator is used to assign short binary strings that are
guaranteed to never be duplicated and also remain relatively short
(i.e., 1-4 byte range). This is currently primarily used by the
DirectoryLayer for creating nodes in the directory tree.

3 years agoLock transactions to the process that created them
Paul J. Davis [Tue, 12 Feb 2019 17:38:44 +0000 (11:38 -0600)] 
Lock transactions to the process that created them

Rather than attempt to manage concurrent access to transactions we've
just removed the necessity by locking transaction resources to a given
thread. Perhaps in the future a use case for sharing a transaction
between threads may arrise and we'll have to revisit this approach.
However, given the five second limit on transactions, this seems fairly
unlikely at this point in time.

3 years agoFix typo in function head for future_is_ready
Paul J. Davis [Tue, 12 Feb 2019 17:14:21 +0000 (11:14 -0600)] 
Fix typo in function head for future_is_ready

Fixes #1

Co-Authored-By: Robert Newson <>
3 years agoSupport 64bit integers in atomic operations
Paul J. Davis [Mon, 11 Feb 2019 18:28:11 +0000 (12:28 -0600)] 
Support 64bit integers in atomic operations

3 years agoImplement the subspace layer
Paul J. Davis [Mon, 11 Feb 2019 16:12:31 +0000 (10:12 -0600)] 
Implement the subspace layer

3 years agoRename erlfdb_layer_tuple to erlfdb_tuple
Paul J. Davis [Sun, 10 Feb 2019 17:19:57 +0000 (11:19 -0600)] 
Rename erlfdb_layer_tuple to erlfdb_tuple

Originally I thought I'd have these stacked in some sort of record to
force all operations to go through the various layers but that does not
appear to be how FoundationDB bindings generally work.

3 years agoImplement driver for fdb bindingstester
Paul J. Davis [Thu, 7 Feb 2019 23:14:14 +0000 (17:14 -0600)] 
Implement driver for fdb bindingstester

3 years agoSupport packing versionstamps in the tuple layer
Paul J. Davis [Sat, 9 Feb 2019 22:46:10 +0000 (16:46 -0600)] 
Support packing versionstamps in the tuple layer

3 years agoFix bugs found using FoundationDBs bindings tester
Paul J. Davis [Sat, 9 Feb 2019 22:43:32 +0000 (16:43 -0600)] 
Fix bugs found using FoundationDBs bindings tester

There were a handful of subtle bugs related to how various corner cases
were handled. The bindings tester is fairly impressivly thorough.

3 years agoFix iterating over key/value arrays
Paul J. Davis [Sat, 9 Feb 2019 22:31:47 +0000 (16:31 -0600)] 
Fix iterating over key/value arrays

Classic off by one error starting one element past the end of the array.

3 years agoAdd missing void future type
Paul J. Davis [Thu, 7 Feb 2019 23:48:39 +0000 (17:48 -0600)] 
Add missing void future type

I initially misread the docs and thought some futures just returned
empty values instead of no value whatsoever. This accounts for that
new reality.

3 years agoTidy up API implementation
Paul J. Davis [Thu, 7 Feb 2019 23:13:38 +0000 (17:13 -0600)] 
Tidy up API implementation

3 years agoAdd string and keyvalue future callbacks
Paul J. Davis [Thu, 7 Feb 2019 23:13:12 +0000 (17:13 -0600)] 
Add string and keyvalue future callbacks

3 years agoFix bug when creating futures
Paul J. Davis [Thu, 7 Feb 2019 21:48:46 +0000 (15:48 -0600)] 
Fix bug when creating futures

We accidentally set the wrong future variable for the callback data
which caused a segfault when attempting to invoke the callback.

3 years agoImplement Erlang API
Paul J. Davis [Wed, 6 Feb 2019 14:21:54 +0000 (08:21 -0600)] 
Implement Erlang API

This is the first draft API over the NIF bindings to FoundationDB. Once
this gets some more use we may want to make a couple tweaks for things
that are are discovered to be awkward or repetitive.

3 years agoInitial tuple comparison implementation
Paul J. Davis [Fri, 1 Feb 2019 17:07:11 +0000 (11:07 -0600)] 
Initial tuple comparison implementation

3 years agoImplement tuple layer encoding
Paul J. Davis [Thu, 31 Jan 2019 17:29:50 +0000 (11:29 -0600)] 
Implement tuple layer encoding

3 years agoImplement set_option APIs
Paul J. Davis [Wed, 30 Jan 2019 14:10:41 +0000 (08:10 -0600)] 
Implement set_option APIs

3 years agoImplement network_set_option
Paul J. Davis [Wed, 30 Jan 2019 13:43:32 +0000 (07:43 -0600)] 
Implement network_set_option

3 years agoAdd IS_ATOM helper
Paul J. Davis [Wed, 30 Jan 2019 13:31:55 +0000 (07:31 -0600)] 
Add IS_ATOM helper

3 years agoList option names for specs
Paul J. Davis [Wed, 30 Jan 2019 13:26:17 +0000 (07:26 -0600)] 
List option names for specs

3 years agoImplemented the rest of the API
Paul J. Davis [Wed, 30 Jan 2019 12:57:33 +0000 (06:57 -0600)] 
Implemented the rest of the API

3 years agoImplmeented transaction commit APIs
Paul J. Davis [Tue, 29 Jan 2019 22:48:53 +0000 (16:48 -0600)] 
Implmeented transaction commit APIs

3 years agoImplemented transaction side effect APIs
Paul J. Davis [Tue, 29 Jan 2019 22:28:28 +0000 (16:28 -0600)] 
Implemented transaction side effect APIs

3 years agoMore transaction APIs implemented
Paul J. Davis [Tue, 29 Jan 2019 21:15:45 +0000 (15:15 -0600)] 
More transaction APIs implemented

3 years agoInitial work on transaction APIs
Paul J. Davis [Tue, 29 Jan 2019 18:36:14 +0000 (12:36 -0600)] 
Initial work on transaction APIs

3 years agoImplement cluster and database creation
Paul J. Davis [Tue, 29 Jan 2019 17:55:37 +0000 (11:55 -0600)] 
Implement cluster and database creation

This includes initial work on being able to resolve futures.

3 years agoAdd create_cluster callback
Paul J. Davis [Tue, 29 Jan 2019 16:20:52 +0000 (10:20 -0600)] 
Add create_cluster callback

Completely untested for now. This is the initial outline for how
features will work. Next up I'll add the shape for erlfdb_future_get to
return the various value types from different futures.

3 years agoAdd base future functions
Paul J. Davis [Tue, 29 Jan 2019 15:30:06 +0000 (09:30 -0600)] 
Add base future functions

3 years agoAdd network thread creation and destruction
Paul J. Davis [Tue, 29 Jan 2019 15:07:49 +0000 (09:07 -0600)] 
Add network thread creation and destruction

3 years agoAdd network configuration outline
Paul J. Davis [Tue, 29 Jan 2019 13:46:03 +0000 (07:46 -0600)] 
Add network configuration outline

3 years agoInitial Import
Paul J. Davis [Tue, 29 Jan 2019 12:37:30 +0000 (06:37 -0600)] 
Initial Import