couchdb.git
2 days agoFix elixir tests and add back to make check (#1918) master
garren smith [Fri, 15 Feb 2019 13:42:57 +0000 (15:42 +0200)] 
Fix elixir tests and add back to make check (#1918)

* Add back elixir tests to make check

* Fix reliability of the flaky elixir tests

3 days agoMerge pull request #1642 from cloudant/91984-set-io_priority-for-couch-index-pids
iilyak [Thu, 14 Feb 2019 13:42:17 +0000 (05:42 -0800)] 
Merge pull request #1642 from cloudant/91984-set-io_priority-for-couch-index-pids

Set io_priority for couch_index pids

3 days agoSet io_priority for couch_index pids 1642/head
ILYA Khlopotov [Sat, 29 Sep 2018 00:06:08 +0000 (17:06 -0700)] 
Set io_priority for couch_index pids

4 days agoMerge pull request #1803 from cloudant/configurable-auth-salt
Jay Doane [Tue, 12 Feb 2019 22:57:37 +0000 (14:57 -0800)] 
Merge pull request #1803 from cloudant/configurable-auth-salt

Sync admin passwords at cluster setup finish

5 days agoSync admin password hashes at cluster setup finish 1803/head
Jay Doane [Tue, 12 Feb 2019 19:27:13 +0000 (11:27 -0800)] 
Sync admin password hashes at cluster setup finish

This ensures that admin password hashes are the same on all nodes when
passwords are set directly on each node rather than through the
coordinator node.

10 days agoMerge pull request #1910 from apache/import-cloudant-ken
Robert Newson [Thu, 7 Feb 2019 21:48:19 +0000 (21:48 +0000)] 
Merge pull request #1910 from apache/import-cloudant-ken

Import ken from cloudant

10 days agoImport ken 1910/head
Robert Newson [Wed, 6 Feb 2019 20:03:39 +0000 (20:03 +0000)] 
Import ken

10 days agoAdd check for repeated `partition` definitions
Garren Smith [Tue, 5 Feb 2019 14:47:39 +0000 (16:47 +0200)] 
Add check for repeated `partition` definitions

This is a usability improvement. If someone specifies the a `partition`
value in the query string that is different than the `partition` value
in the URL path it is not clear which value would be used. This allows
specifying it in both places as long as the query string matches the URL
path and throws a 400 Bad Request error otherwise.

Co-Authored-By: Garren Smith <garren.smith@gmail.com>
10 days agofix couchup for python3 (#1905)
Clemens Stolle [Thu, 7 Feb 2019 05:54:19 +0000 (06:54 +0100)] 
fix couchup for python3 (#1905)

11 days agoMerge pull request #1904 from apache/import-smoosh
Robert Newson [Wed, 6 Feb 2019 13:27:23 +0000 (13:27 +0000)] 
Merge pull request #1904 from apache/import-smoosh

Import smoosh from Cloudant

11 days agoremove elixir tests from 'make check' until they are reliable 1904/head
Robert Newson [Wed, 6 Feb 2019 12:46:45 +0000 (12:46 +0000)] 
remove elixir tests from 'make check' until they are reliable

11 days agorun formatting check before time-consuming tests
Robert Newson [Wed, 6 Feb 2019 12:43:37 +0000 (12:43 +0000)] 
run formatting check before time-consuming tests

11 days agoImport smoosh from Cloudant
Robert Newson [Tue, 5 Feb 2019 17:04:54 +0000 (17:04 +0000)] 
Import smoosh from Cloudant

Remove couch_compaction_daemon and related tests too.

12 days agoForce mix rebar/hex/deps get on make elixir (#1894)
Joan Touzet [Tue, 5 Feb 2019 18:28:05 +0000 (13:28 -0500)] 
Force mix rebar/hex/deps get on make elixir (#1894)

12 days agoMerge pull request #1901 from apache/fix-doc-update-invalid-rev-crash
Eric Avdey [Tue, 5 Feb 2019 14:39:27 +0000 (10:39 -0400)] 
Merge pull request #1901 from apache/fix-doc-update-invalid-rev-crash

Fix `badarg` crash on an invalid revision for individual doc update

12 days agoFix badarg crash on invalid rev for individual doc update 1901/head
Eric Avdey [Tue, 5 Feb 2019 02:51:18 +0000 (22:51 -0400)] 
Fix badarg crash on invalid rev for individual doc update

12 days agoFix from_json_obj_validate crash when provided rev isn't a valid hex
Eric Avdey [Fri, 1 Feb 2019 16:41:54 +0000 (12:41 -0400)] 
Fix from_json_obj_validate crash when provided rev isn't a valid hex

12 days agoMake from_json_error_cases tests idiomatic
Eric Avdey [Fri, 1 Feb 2019 16:39:42 +0000 (12:39 -0400)] 
Make from_json_error_cases tests idiomatic

2 weeks agoMerge pull request #1889 from apache/import-cloudant-ioq
Robert Newson [Fri, 1 Feb 2019 18:36:58 +0000 (18:36 +0000)] 
Merge pull request #1889 from apache/import-cloudant-ioq

Import IOQ from Cloudant

2 weeks agoWe don't need to verify that erlang:garbage_collect() works 1889/head
Robert Newson [Fri, 1 Feb 2019 17:49:10 +0000 (17:49 +0000)] 
We don't need to verify that erlang:garbage_collect() works

2 weeks agoincrease timeout on test
Robert Newson [Fri, 1 Feb 2019 17:23:26 +0000 (17:23 +0000)] 
increase timeout on test

2 weeks agoRun each apps test in a separate process
Robert Newson [Fri, 1 Feb 2019 10:31:31 +0000 (10:31 +0000)] 
Run each apps test in a separate process

This reduces non-causal test failures between apps.

2 weeks agoImport IOQ from Cloudant
Robert Newson [Wed, 30 Jan 2019 17:01:18 +0000 (17:01 +0000)] 
Import IOQ from Cloudant

2 weeks agoadd w:3 for lots of docs test (#1893)
garren smith [Fri, 1 Feb 2019 08:46:38 +0000 (10:46 +0200)] 
add w:3 for lots of docs test (#1893)

2 weeks agoformat
Garren Smith [Thu, 31 Jan 2019 13:22:30 +0000 (15:22 +0200)] 
format

2 weeks agoAdd junit formatter
Garren Smith [Wed, 30 Jan 2019 13:52:04 +0000 (15:52 +0200)] 
Add junit formatter

Add the junit formatter so that Jenkins can read the elixir tests

2 weeks agoChange elixir test failing on travis
Garren Smith [Wed, 30 Jan 2019 13:16:17 +0000 (15:16 +0200)] 
Change elixir test failing on travis

2 weeks agoadd elixir tests to travis
Garren Smith [Tue, 29 Jan 2019 13:04:38 +0000 (15:04 +0200)] 
add elixir tests to travis

Add the elixir test to `make check` and change the build language
in travis from erlang to elixir.

2 weeks agoChange enviromental variables not to clash with travis
Garren Smith [Wed, 30 Jan 2019 09:53:28 +0000 (11:53 +0200)] 
Change enviromental variables not to clash with travis

Creates EX_USERNAME, EX_PASSWORD and EX_COUCH_URL to manage the
elixir test setup.

2 weeks agofix credo and formatting issues of elixir tests
Garren Smith [Tue, 29 Jan 2019 13:04:23 +0000 (15:04 +0200)] 
fix credo and formatting issues of elixir tests

2 weeks agoCompaction: Add snooze_period_ms for finer tuning (#1880)
Adrien VergĂ© [Thu, 31 Jan 2019 00:06:21 +0000 (01:06 +0100)] 
Compaction: Add snooze_period_ms for finer tuning (#1880)

This commit introduces a new option `snooze_period_ms` (measured in
milliseconds), and deprecates `snooze_period` while still supporting it
for obvious legacy reasons.

2 weeks agoMerge pull request #1886 from apache/fix-pq-db-header-version
Peng Hui Jiang [Wed, 30 Jan 2019 12:56:22 +0000 (20:56 +0800)] 
Merge pull request #1886 from apache/fix-pq-db-header-version

Fix partitioned query database header version

2 weeks agoFix engine header upgrades fix-pq-db-header-version 1886/head
Paul J. Davis [Mon, 28 Jan 2019 21:04:15 +0000 (15:04 -0600)] 
Fix engine header upgrades

2 weeks agoMake it possible to override elixir couchdb settings via environmental variables
Garren Smith [Wed, 30 Jan 2019 08:16:37 +0000 (10:16 +0200)] 
Make it possible to override elixir couchdb settings via environmental variables

2 weeks agoMerge pull request #1877 from apache/restrict-purge-admin
Peng Hui Jiang [Tue, 29 Jan 2019 00:47:44 +0000 (08:47 +0800)] 
Merge pull request #1877 from apache/restrict-purge-admin

restrict _purge to server admin

2 weeks agorestrict _purge to server admin restrict-purge-admin 1877/head
jiangph [Mon, 28 Jan 2019 02:43:34 +0000 (10:43 +0800)] 
restrict _purge to server admin

This restrict _purge and _purged_infos_limit to server admin
in terms of the security level required to run them.

Fixes #1799

2 weeks agorename partitions feature to partitioned feature
Garren Smith [Mon, 28 Jan 2019 08:13:44 +0000 (10:13 +0200)] 
rename partitions feature to partitioned feature

3 weeks agoUpdate config dependency to 2.1.5
Nick Vatamaniuc [Thu, 24 Jan 2019 16:59:49 +0000 (11:59 -0500)] 
Update config dependency to 2.1.5

This fixes inability to set keys with regex symbols in them

3 weeks agofix test url to fix test
Garren Smith [Thu, 24 Jan 2019 09:35:23 +0000 (11:35 +0200)] 
fix test url to fix test

3 weeks agoShift to new GH multi-template issue workflow (#1872)
Joan Touzet [Wed, 23 Jan 2019 19:32:06 +0000 (14:32 -0500)] 
Shift to new GH multi-template issue workflow (#1872)

3 weeks agoImplement customizable chttpd statistics
Paul J. Davis [Thu, 29 Nov 2018 19:11:38 +0000 (13:11 -0600)] 
Implement customizable chttpd statistics

3 weeks agoMake _all_docs and view query limits configurable
Garren Smith [Thu, 6 Dec 2018 11:03:18 +0000 (13:03 +0200)] 
Make _all_docs and view query limits configurable

This allows us to set a maximun allowed number of documents
to be returned for a global or a partitioned view query and
_all_docs query.

Co-authored-by: Paul J. Davis <paul.joseph.davis@gmail.com>
3 weeks agoBlacklist known bad Erlang releases, fixes #1857 (#1871)
Joan Touzet [Wed, 23 Jan 2019 00:49:29 +0000 (19:49 -0500)] 
Blacklist known bad Erlang releases, fixes #1857 (#1871)

3 weeks agoUpdate to mochiweb 2.19.0
Nick Vatamaniuc [Tue, 22 Jan 2019 16:02:34 +0000 (11:02 -0500)] 
Update to mochiweb 2.19.0

It has a fix to revert user socket buffer size to 8192 and also
allow setting this buffer values directly (not necessarily
via {recbuf, ...}).

Fixes #1810

Warning:

2.19.0 blacklists a series of OTP releases: 21.2, 21.2.1, 21.2.2
This is done via a runtime check of the ssl application version.

The blacklist seems valid as there is a bug which prevents data from
being delivered on TSL sockets. That could affect either CouchDB
server side (chttpd) or replication client side (ibrowse).

3 weeks agodefault config for partition feature flag
Garren Smith [Tue, 15 Jan 2019 13:12:41 +0000 (15:12 +0200)] 
default config for partition feature flag

3 weeks agoAdd ability to feature flag partitioned db's
Garren Smith [Tue, 15 Jan 2019 13:12:24 +0000 (15:12 +0200)] 
Add ability to feature flag partitioned db's

3 weeks agoAdd a test for a doc read with etag
Eric Avdey [Mon, 21 Jan 2019 20:42:10 +0000 (16:42 -0400)] 
Add a test for a doc read with etag

3 weeks agoDon't crash test on a response without Content-Type
Eric Avdey [Mon, 21 Jan 2019 20:36:30 +0000 (16:36 -0400)] 
Don't crash test on a response without Content-Type

Some GET responses (e.g. 304 Not Modified) should not generate
representation metadata (e.g. Content-Type) as per rfc7232.

This fix allows to process such requests without crashing process response
on a regexp over nil.

3 weeks agoadd default fabric request timeouts
Garren Smith [Thu, 29 Nov 2018 06:57:50 +0000 (08:57 +0200)] 
add default fabric request timeouts

3 weeks agoAdd Global timeout for map/reduce queries
Garren Smith [Tue, 27 Nov 2018 14:40:02 +0000 (16:40 +0200)] 
Add Global timeout for map/reduce queries

This makes the global timeout for a map/reduce and all_docs request
configurable via the config. It separates the config into global queries
and partition queries so that it is possible to make the global timeout
is for partitioned queries

3 weeks agoFix python2 compatibility for couchup (#1868)
Paulo Eduardo Althoff [Mon, 21 Jan 2019 02:32:31 +0000 (00:32 -0200)] 
Fix python2 compatibility for couchup (#1868)

Closes #1053

4 weeks agoAdd new /{db}/_sync_shards endpoint (admin-only) (#1811)
Joan Touzet [Fri, 18 Jan 2019 20:16:19 +0000 (15:16 -0500)] 
Add new /{db}/_sync_shards endpoint (admin-only) (#1811)

This server admin-only endpoint forces an n-way sync of all shards
across all nodes on which they are hosted.

This can be useful for an administrator adding a new node to the
cluster, after updating _dbs so that the new node hosts an existing db
with content, to force the new node to sync all of that db's shards.

Users may want to bump their `[mem3] sync_concurrency` value to a
larger figure for the duration of the shards sync.

Closes #1807

4 weeks agoEnforce partition size limits
Paul J. Davis [Fri, 14 Dec 2018 17:06:03 +0000 (11:06 -0600)] 
Enforce partition size limits

This limit helps prevent users from inadvertently misusing partitions by
refusing to add documents when the size of a partition exceeds 10GiB.

Co-authored-by: Robert Newson <rnewson@apache.org>
4 weeks agoUse an accumulator when merging revision trees
Paul J. Davis [Fri, 14 Dec 2018 16:31:02 +0000 (10:31 -0600)] 
Use an accumulator when merging revision trees

This cleans up the `couch_db_updater:merge_rev_trees/7` to instead use
an accumulator argument.

4 weeks agoAdd Elixir tests for database partitions
Paul J. Davis [Wed, 28 Nov 2018 16:58:42 +0000 (10:58 -0600)] 
Add Elixir tests for database partitions

Co-authored-by: Garren Smith <garren.smith@gmail.com>
Co-authored-by: Robert Newson <rnewson@apache.org>
4 weeks agoSupport partitioned queries in Mango
Paul J. Davis [Mon, 29 Oct 2018 19:08:05 +0000 (14:08 -0500)] 
Support partitioned queries in Mango

Co-authored-by: Garren Smith <garren.smith@gmail.com>
Co-authored-by: Robert Newson <rnewson@apache.org>
4 weeks agoUse index names when testing index selection
Paul J. Davis [Wed, 19 Dec 2018 17:48:45 +0000 (11:48 -0600)] 
Use index names when testing index selection

Using the internal hash values for indexes was a brittle approach to
ensuring that a specific index was or was not picked. By naming the
index and design docs we can more concretely ensure that the chosen
indexes match the intent of the test while also not breaking each time
mango internals change.

4 weeks agoOptimize offset/limit for partition queries
Paul J. Davis [Thu, 25 Oct 2018 19:27:32 +0000 (14:27 -0500)] 
Optimize offset/limit for partition queries

Now that a single shard handles the entire response we can optimize work
normally done in the coordinator by moving it to the RPC worker which
then removes the need to send an extra `skip` number of rows to the
coordinator.

Co-authored-by: Robert Newson <rnewson@apache.org>
4 weeks agoOptimize all_docs queries in a single partition
Paul J. Davis [Thu, 25 Oct 2018 19:26:25 +0000 (14:26 -0500)] 
Optimize all_docs queries in a single partition

If a user specifies document ids that scope the query to a single
partition key we can automatically determine that we only need to
consuly a single shard range.

Co-authored-by: Robert Newson <rnewson@apache.org>
4 weeks agoImplement partitioned views
Paul J. Davis [Thu, 25 Oct 2018 19:19:07 +0000 (14:19 -0500)] 
Implement partitioned views

The benefit of using partitioned databases is that views can then be
scoped to a single shard range. This allows for views to scale nearly as
linearly as document lookups.

Co-authored-by: Garren Smith <garren.smith@gmail.com>
Co-authored-by: Robert Newson <rnewson@apache.org>
4 weeks agoImplement `couch_db:get_partition_info/2`
Paul J. Davis [Tue, 23 Oct 2018 19:18:35 +0000 (14:18 -0500)] 
Implement `couch_db:get_partition_info/2`

This feature allows us to fetch statistics for a given partition key
which will allow for users to find bloated partitions and such forth.

Co-authored-by: Garren Smith <garren.smith@gmail.com>
Co-authored-by: Robert Newson <rnewson@apache.org>
4 weeks agoImplement partitioned dbs
Paul J. Davis [Thu, 25 Oct 2018 22:03:00 +0000 (17:03 -0500)] 
Implement partitioned dbs

This change introduces the ability for users to place a group of
documents in a single shard range by specifying a "partition key" in the
document id. A partition key is denoted by everything preceding a colon
':' in the document id.

Every document id (except for design documents) in a partitioned
database is required to have a partition key.

Co-authored-by: Garren Smith <garren.smith@gmail.com>
Co-authored-by: Robert Newson <rnewson@apache.org>
4 weeks agoImplement configurable hash functions
Paul J. Davis [Thu, 25 Oct 2018 21:58:48 +0000 (16:58 -0500)] 
Implement configurable hash functions

This provides the capability for features to specify alternative hash
functions for placing documents in a given shard range. While the
functionality exists with this implementation it is not yet actually
used.

4 weeks agoValidate design document options more strictly
Paul J. Davis [Wed, 24 Oct 2018 16:13:29 +0000 (11:13 -0500)] 
Validate design document options more strictly

This adds specific datatype requirements to the list of allowable design
document options.

Co-authored-by: Garren Smith <garren.smith@gmail.com>
Co-authored-by: Robert Newson <rnewson@apache.org>
4 weeks agoPass the DB record to index validation functions
Paul J. Davis [Wed, 24 Oct 2018 16:12:27 +0000 (11:12 -0500)] 
Pass the DB record to index validation functions

Allow index validation to be parameterized by the database without
having to reopen its own copy.

4 weeks agoImplement `fabric_util:open_cluster_db`
Paul J. Davis [Thu, 25 Oct 2018 21:13:48 +0000 (16:13 -0500)] 
Implement `fabric_util:open_cluster_db`

This allows for more fine grained use of couch_db:clustered_db as well
as chagnes the name to something more appropriate than `fake_db`.

4 weeks agoImprove `couch_db:clustered_db` flexibility
Paul J. Davis [Thu, 25 Oct 2018 21:09:38 +0000 (16:09 -0500)] 
Improve `couch_db:clustered_db` flexibility

This allows for setting any combintaion of supported settings using a
proplist appraoch.

4 weeks agoAdd PSE API to store opaque properties
Garren Smith [Thu, 21 Jun 2018 14:30:35 +0000 (16:30 +0200)] 
Add PSE API to store opaque properties

This allows us to implement features outside of the PSE API without
requiring changes to the API for each bit of data we may want to end up
storing. The use of this opaque object should only be used for features
that don't require a beahvior change from the storage engine API.

Co-authored-by: Garren Smith <garren.smith@gmail.com>
Co-authored-by: Robert Newson <rnewson@apache.org>
4 weeks agoFix timeout in chttpd_purge_tests
Paul J. Davis [Fri, 18 Jan 2019 16:52:24 +0000 (10:52 -0600)] 
Fix timeout in chttpd_purge_tests

4 weeks agoMove Jenkins to use Erlang 19 for initial build step (#1866)
Joan Touzet [Fri, 18 Jan 2019 16:53:55 +0000 (11:53 -0500)] 
Move Jenkins to use Erlang 19 for initial build step (#1866)

4 weeks agoMerge pull request #1865 from apache/purge_request_with_101_docid
Peng Hui Jiang [Fri, 18 Jan 2019 00:31:25 +0000 (08:31 +0800)] 
Merge pull request #1865 from apache/purge_request_with_101_docid

Support one purge request with more than 100 docid

4 weeks agoSupport one purge request with more than 100 docid 1865/head
jiangph [Thu, 17 Jan 2019 09:50:30 +0000 (17:50 +0800)] 
Support one purge request with more than 100 docid

COUCHDB-3226

4 weeks agoFix fabric_open_doc_revs
Paul J. Davis [Tue, 15 Jan 2019 00:32:34 +0000 (18:32 -0600)] 
Fix fabric_open_doc_revs

There was a subtle bug when opening specific revisions in
fabric_doc_open_revs due to a race condition between updates being
applied across a cluster.

The underlying cause here was due to the stemming after a document had
been updated more than revs_limit number of times along with concurrent
reads to a node that had not yet made the update. To illustrate lets
consider a document A which has a revision history from `{N, RevN}` to
`{N+1000, RevN+1000}` (assuming revs_limit is the default 1000). If we
consider a single node perspective when an update comes in we added the
new revision and stem the oldest revision. The docs the revisions on the
node would be `{N+1, RevN+1}` to `{N+1001, RevN+1001}`.

The bug exists when we attempt to open revisions on a different node
that has yet to apply the new update. In this case when
fabric_doc_open_revs could be called with `{N+1000, RevN+1000}`. This
results in a response from fabric_doc_open_revs that includes two
different `{ok, Doc}` results instead of the expected one instance. The
reason for this is that one document has revisions `{N+1, RevN+1}` to
`{N+1000, RevN+1000}` from the node that has applied the update, while
the node without the update responds with revisions `{N, RevN}` to
{N+1000, RevN+1000}`.

To rephrase that, a node that has applied an update can end up returning
a revision path that contains `revs_limit - 1` revisions while a node
wihtout the update returns all `revs_limit` revisions. This slight
change in the path prevented the responses from being properly combined
into a single response.

This bug has existed for many years. However, read repair effectively
prevents it from being a significant issue by immediately fixing the
revision history discrepancy. This was discovered due to the recent bug
in read repair during a mixed cluster upgrade to a release including
clustered purge. In this situation we end up crashing the design
document cache which then leads to all of the design document requests
being direct reads which can end up causing cluster nodes to OOM and
die. The conditions require a significant number of design document
edits coupled with already significant load to those modified design
documents. The most direct example observed was a clustered that had a
significant number of filtered replications in and out of the cluster.

4 weeks agoImprove vm.args template comments (#1861)
Joan Touzet [Tue, 15 Jan 2019 18:36:38 +0000 (13:36 -0500)] 
Improve vm.args template comments (#1861)

4 weeks agoFix read repair in a mixed cluster environment
Paul J. Davis [Mon, 14 Jan 2019 22:33:13 +0000 (16:33 -0600)] 
Fix read repair in a mixed cluster environment

This enables backwards compatbility with nodes still running the old
version of fabric_rpc when a cluster is upgraded to master. This has no
effect once all nodes are upgraded to the latest version.

5 weeks agoFix end_time field in /_replicate response
Nick Vatamaniuc [Tue, 8 Jan 2019 20:17:06 +0000 (15:17 -0500)] 
Fix end_time field in /_replicate response

Previously `end_time` was generated converting the start_time to universal,
then passing that to `httpd_util:rfc1123_date/1`. However, `rfc1123_date/1`
also transates its argument from local to UTC time, that is it accepts input to
be in local time format.

Fixes #1841

6 weeks agoMerge pull request #1808 from apache/before_doc_update
Peng Hui Jiang [Sat, 5 Jan 2019 07:40:32 +0000 (15:40 +0800)] 
Merge pull request #1808 from apache/before_doc_update

Update before_doc_update/2 to before_doc_update/3

6 weeks agoUpdate before_doc_update/2 to before_doc_update/3 1808/head
jiangph [Thu, 13 Dec 2018 06:34:47 +0000 (14:34 +0800)] 
Update before_doc_update/2 to before_doc_update/3

  - Pass UpdateType to before_doc_update/3

6 weeks agoMerge pull request #1831 from apache/intro-cpse_test_purge_seqs
Peng Hui Jiang [Thu, 3 Jan 2019 02:48:07 +0000 (10:48 +0800)] 
Merge pull request #1831 from apache/intro-cpse_test_purge_seqs

Re-Introduce cpse_test_purge_seqs

6 weeks agoIntroduce cpse_test_purge_seqs again 1831/head
jiangph [Tue, 25 Dec 2018 10:23:00 +0000 (18:23 +0800)] 
Introduce cpse_test_purge_seqs again

  - Re-introduce cpse_test_purge_seqs after fixing issue on
  cpse_test_purge_seqs:cpse_increment_purge_seq_on_partial_purge/1
  with undef issue

6 weeks agohappy new year (#1838)
Jan Lehnardt [Mon, 31 Dec 2018 18:23:17 +0000 (19:23 +0100)] 
happy new year (#1838)

7 weeks agoMerge pull request #1833 from cloudant/minimum-erlang-otp-19
iilyak [Fri, 28 Dec 2018 19:05:06 +0000 (11:05 -0800)] 
Merge pull request #1833 from cloudant/minimum-erlang-otp-19

Change minimum supported Erlang version to OTP 19

7 weeks agoChange minimum supported Erlang version to OTP 19 1833/head
Jay Doane [Fri, 28 Dec 2018 18:18:06 +0000 (10:18 -0800)] 
Change minimum supported Erlang version to OTP 19

7 weeks agoRemove obsolete travis files
Jay Doane [Sun, 23 Dec 2018 08:52:26 +0000 (00:52 -0800)] 
Remove obsolete travis files

These files were used when their apps had separate repositories, but are
obsolete in the "mono repo" since their apps are built together using
the top level .travis.yml now.

7 weeks agoRemove explicit modules list from .app.src files
Jay Doane [Sun, 23 Dec 2018 09:12:55 +0000 (01:12 -0800)] 
Remove explicit modules list from .app.src files

The modules lists in .app files are automatically generated by rebar
from .app.src files, so these explicit lists are unnecessary and prone
to being out of date.

7 weeks agoMerge pull request #1798 from cloudant/suppress-compiler-warnings
iilyak [Thu, 27 Dec 2018 21:34:36 +0000 (13:34 -0800)] 
Merge pull request #1798 from cloudant/suppress-compiler-warnings

Suppress compiler warnings

7 weeks agoSuppress export-related compiler warnings 1798/head
Jay Doane [Thu, 27 Dec 2018 20:49:47 +0000 (12:49 -0800)] 
Suppress export-related compiler warnings

- For export_all warnings: either replace with explicit exports, add
nowarn_export_all compiler directives when appropriate, or in the case
of couch_epi_sup, move the test to dedicated test file and export the
function needed for testing.

- For "function already exported" warning in couch_key_tree_prop_tests,
remove include_lib attribute for eunit.hrl since it already gets
imported in triq.hrl

7 weeks agoReduce number of behaviour undefined compiler warnings
Jay Doane [Thu, 13 Dec 2018 06:00:48 +0000 (22:00 -0800)] 
Reduce number of behaviour undefined compiler warnings

Move couch_event and mem3 modules earlier in the list of SubDirs to
suppress behaviour undefined warnings.

This has the side effect of running the tests in the new order, which
induces failures in couch_index tests. Those failures are related to
quorum, and can be traced to mem3 seeds tests leaving a _nodes db
containing several node docs in the tmp/data directory, ultimately
resulting in badmatch errors e.g. when a test expects 'ok' but gets
'accepted' instead.

To prevent test failures, a cleanup function is implemented which
deletes any existing "nodes_db" left after test completion.

7 weeks agoSuppress misc compiler warnings
Jay Doane [Thu, 13 Dec 2018 04:41:27 +0000 (20:41 -0800)] 
Suppress misc compiler warnings

- couch_util_tests.erl:90:
    Warning: the result of the expression is ignored

- couch_mrview_index_changes_tests.erl:189,196:
    Warning: a term is constructed, but never used

- couch_replicator_connection_tests.erl:76:
    Warning: this expression will fail with a 'badarith' exception

7 weeks agoSuppress unused function compiler warnings
Jay Doane [Thu, 13 Dec 2018 03:49:01 +0000 (19:49 -0800)] 
Suppress unused function compiler warnings

- Add unused test cases to test fixture
- Eliminate unreferenced code
- Comment out code that is referenced in commented code only

7 weeks agoSuppress crypto and random compiler warnings
Jay Doane [Sun, 23 Dec 2018 18:07:02 +0000 (10:07 -0800)] 
Suppress crypto and random compiler warnings

Replace deprecated crypto:rand_uniform/2 and 'random' module functions
with equivalent couch_rand:uniform/1 calls, or eliminate the offending
code entirely if unused.

Note that crypto:rand_uniform/2 takes two parameters which have
different semantics than the single argument couch_rand:uniform/1.

Tests in mem3 are also provided to validate that the random rotation of
node lists was converted correctly.

7 weeks agoSuppress unused variable and type compiler warnings
Jay Doane [Thu, 13 Dec 2018 03:08:09 +0000 (19:08 -0800)] 
Suppress unused variable and type compiler warnings

7 weeks agoSuppress variable exported from 'case' compiler warnings
Jay Doane [Thu, 13 Dec 2018 03:03:16 +0000 (19:03 -0800)] 
Suppress variable exported from 'case' compiler warnings

7 weeks agoMerge pull request #1829 from cloudant/elixir-test-improvements
iilyak [Thu, 27 Dec 2018 21:04:12 +0000 (13:04 -0800)] 
Merge pull request #1829 from cloudant/elixir-test-improvements

Elixir test improvements

8 weeks agoDo not automatically fail tests if quorum conditions unmet 1829/head
Jay Doane [Sun, 23 Dec 2018 06:12:50 +0000 (22:12 -0800)] 
Do not automatically fail tests if quorum conditions unmet

8 weeks agoImprove all_docs_test robustness
Jay Doane [Sun, 23 Dec 2018 05:46:11 +0000 (21:46 -0800)] 
Improve all_docs_test robustness

Wrap deleted element assertions in retry_until to prevent timing related
failures like:

AllDocsTest
  * test All Docs tests (331.1ms)

  1) test All Docs tests (AllDocsTest)
     test/all_docs_test.exs:15
     Assertion with == failed
     code:  assert length(deleted) == 1
     left:  0
     right: 1
     stacktrace:
       test/all_docs_test.exs:72: (test)

8 weeks agoFix elixir test formatting
Jay Doane [Sun, 23 Dec 2018 05:42:31 +0000 (21:42 -0800)] 
Fix elixir test formatting

Prior to this, `make elixir` was failing with these errors:

** (Mix) mix format failed due to --check-formatted.
The following files were not formatted:

  * test/security_validation_test.exs
  * test/rewrite_test.exs
  * test/cluster_with_quorum_test.exs
  * test/cluster_without_quorum_test.exs
  * test/all_docs_test.exs

8 weeks agoRemove shim couch_replicator_manager module
Kyle Snavely [Thu, 20 Dec 2018 21:37:13 +0000 (16:37 -0500)] 
Remove shim couch_replicator_manager module

8 weeks agoClean rexi stream workers when coordinator process is killed
Nick Vatamaniuc [Thu, 20 Dec 2018 17:19:01 +0000 (12:19 -0500)] 
Clean rexi stream workers when coordinator process is killed

Sometimes fabric coordinators end up getting brutally terminated [1], and in that
case they might never process their `after` clause where their remote rexi
workers are killed. Those workers are left lingering around keeping databases
active for up to 5 minutes at a time.

To prevent that from happening, let coordinators which use streams spawn an
auxiliary cleaner process. This process will monitor the main coordinator and
if it dies will ensure remote workers are killed, freeing resources
immediately. In order not to send 2x the number of kill messages during the
normal exit, fabric_util:cleanup() will stop the auxiliary process before
continuing.

[1] One instance is when the ddoc cache is refreshed:
 https://github.com/apache/couchdb/blob/master/src/ddoc_cache/src/ddoc_cache_entry.erl#L236

8 weeks agoMove fabric streams to a fabric_streams module
Nick Vatamaniuc [Thu, 20 Dec 2018 17:11:10 +0000 (12:11 -0500)] 
Move fabric streams to a fabric_streams module

Streams functionality is fairly isolated from the rest of the utils module so
move it to its own. This is mostly in preparation to add a streams workers
cleaner process.