couchdb.git
14 hours agoMerge pull request #1370 from apache/COUCHDB-3326-clustered-purge-pr5-implementation master
Peng Hui Jiang [Tue, 21 Aug 2018 17:31:35 +0000 (01:31 +0800)] 
Merge pull request #1370 from apache/COUCHDB-3326-clustered-purge-pr5-implementation

[5/5] Clustered Purge Implementation

14 hours ago[10/10] Clustered Purge: Clustered HTTP API COUCHDB-3326-clustered-purge-pr5-implementation 1370/head
Paul J. Davis [Tue, 24 Apr 2018 17:27:58 +0000 (12:27 -0500)] 
[10/10] Clustered Purge: Clustered HTTP API

The HTTP API for clustered purge is fairly straightforward. It is
designed to match the general shape of the single node API. The only
major caveat here is that the purge sequence is now hardcoded as null
since the purge sequence would now otherwise be an opaque blob similar
to the update_seq blobs.

Its important to note that there is as yet no API invented for
traversing the history of purge requests in any shape or form as that
would mostly invalidate the entire purpose of using purge to remove any
trace of a document from a database at the HTTP level. Although there
will still be traces in individual shard files until all database
components have processed the purge and compaction has run (while
allowing for up to purge_infos_limit requests to remain available in
perpetuity).

COUCHDB-3326

Co-authored-by: Mayya Sharipova <mayyas@ca.ibm.com>
Co-authored-by: jiangphcn <jiangph@cn.ibm.com>
14 hours ago[09/10] Clustered Purge: Fabric API
Paul J. Davis [Tue, 24 Apr 2018 17:27:43 +0000 (12:27 -0500)] 
[09/10] Clustered Purge: Fabric API

This commit implements the clustered API for performing purge requests.
This change should be a fairly straightforward change for anyone already
familiar with the general implementation of a fabric coordinator given
that the purge API is fairly simple.

COUCHDB-3326

Co-authored-by: Mayya Sharipova <mayyas@ca.ibm.com>
Co-authored-by: jiangphcn <jiangph@cn.ibm.com>
14 hours ago[08/10] Clustered Purge: Update read-repair
Paul J. Davis [Wed, 30 May 2018 22:26:02 +0000 (17:26 -0500)] 
[08/10] Clustered Purge: Update read-repair

Read-repair needs to know which nodes have requested an update to a
local doc so that it can determine if the update is applied. The basic
idea here is that we may have gotten an update from a remote node that
has yet to apply a purge request. If the local node were to apply this
update it would effectively undo a succesful purge request.

COUCHDB-3326

Co-authored-by: Mayya Sharipova <mayyas@ca.ibm.com>
Co-authored-by: jiangphcn <jiangph@cn.ibm.com>
14 hours ago[07/10] Clustered Purge: Internal replication
Paul J. Davis [Tue, 24 Apr 2018 17:27:14 +0000 (12:27 -0500)] 
[07/10] Clustered Purge: Internal replication

This commit implements the internal replication of purge requests. This
part of the anit-entropy process is important for ensuring that shard
copies continue to be eventually consistent even if updates happen to
shards independently due to a network split or other event that prevents
the successful purge request to a given copy.

The main addition to internal replication is that we both pull and push
purge requests between the source and target shards. The push direction
is obvious given that internal replication is in the push direction
already. Pull isn't quite as obvious but is required so that we don't
push an update that was already purged on the target.

Of note is that internal replication also has to maintain _local doc
checkpoints to prevent compaction removing old purge requests or else
shard copies could end up missing purge requests which would prevent the
shard copies from ever reaching a consistent state.

COUCHDB-3326

Co-authored-by: Mayya Sharipova <mayyas@ca.ibm.com>
Co-authored-by: jiangphcn <jiangph@cn.ibm.com>
14 hours ago[06/10] Clustered Purge: Update mrview indexes
Paul J. Davis [Tue, 24 Apr 2018 17:26:42 +0000 (12:26 -0500)] 
[06/10] Clustered Purge: Update mrview indexes

This commit updates the mrview secondary index to properly process the
new history of purge requests as well as to store the _local purge
checkpoint doc.

The importance of the _local checkpoint doc is to ensure that compaction
of a database does not remove any purge requests that have not yet been
processed by this secondary index.

COUCHDB-3326

Co-authored-by: Mayya Sharipova <mayyas@ca.ibm.com>
Co-authored-by: jiangphcn <jiangph@cn.ibm.com>
14 hours ago[05/10] Clustered Purge: Add upgrade tests
jiangph [Tue, 10 Jul 2018 06:58:49 +0000 (14:58 +0800)] 
[05/10] Clustered Purge: Add upgrade tests

These test that we can successfully upgrade old databases that have
various configurations of purge requests in the legacy format.

COUCHDB-3326

Co-authored-by: Mayya Sharipova <mayyas@ca.ibm.com>
Co-authored-by: jiangphcn <jiangph@cn.ibm.com>
14 hours ago[04/10] Clustered Purge: Update couch_pse_tests
Paul J. Davis [Wed, 30 May 2018 21:13:08 +0000 (16:13 -0500)] 
[04/10] Clustered Purge: Update couch_pse_tests

This updates the couch_pse_tests to account for the new purge APIs as
well as introduces a bunch of new tests for covering the new APIs.

COUCHDB-3326

Co-authored-by: Mayya Sharipova <mayyas@ca.ibm.com>
Co-authored-by: jiangphcn <jiangph@cn.ibm.com>
14 hours ago[03/10] Clustered Purge: Update couch_bt_engine
Paul J. Davis [Tue, 24 Apr 2018 17:25:43 +0000 (12:25 -0500)] 
[03/10] Clustered Purge: Update couch_bt_engine

This commit updates the couch_bt_engine storage engine implementation to
satisfy the newly defined single-node purge APIs. This is accomplished
by storing two new database btrees.

The purge_seq_tree orders purge requests by their purge_seq. This tree
is used to satisfy the fold_purge_infos API for database components to
enumerate the list of purge requests in a defined order.

The second index is the purge_tree which orders purge requests by their
UUID to make for an efficient lookup when filtering replicated purge
requests.

COUCHDB-3326

Co-authored-by: Mayya Sharipova <mayyas@ca.ibm.com>
Co-authored-by: jiangphcn <jiangph@cn.ibm.com>
14 hours ago[02/10] Clustered Purge: Update single node APIs
Paul J. Davis [Tue, 24 Apr 2018 17:24:10 +0000 (12:24 -0500)] 
[02/10] Clustered Purge: Update single node APIs

This patch updates the single node API implementations for use with the
new clustered purge API. At the single node level the major change is to
store a history of purge requests that can then be consumed by various
other parts of the database system.

The simpler of the major areas to use this new functionality will be any
secondary indices. Rather than checking that only a single purge request
has occurred each secondary index will store a _local document
referencing its oldest processed purge request. During index updates
each secondary index implementation will process any new purge requests
and update its local doc checkpoint. In this way secondary indexes will
no longer be sensitive to reset when multiple purge requests are issued
against the database.

The two other major areas that will make use of the newly stored purge
request history are both of the anit-entropy mechanisms: read-repair and
internal replication.

Read-repair will use the purge request history to know when a node
should discard updates that have come from a node that has not yet
processed a purge request during internal replication. Otherwise
read-repair would effectively undo any purge replication that happened
"recently".

Internal replication will use the purge request history to be able to
mend any differences between shards. For instance, if a shard is down
when a purge request is issue against a cluster this process will pull
the purge request and apply it during internal replication. And
similarly any local purge requests will be applied on the target before
normal internal replication.

COUCHDB-3326

Co-authored-by: Mayya Sharipova <mayyas@ca.ibm.com>
Co-authored-by: jiangphcn <jiangph@cn.ibm.com>
14 hours ago[01/10] Clustered Purge: Define new purge API
Paul J. Davis [Tue, 24 Apr 2018 17:23:16 +0000 (12:23 -0500)] 
[01/10] Clustered Purge: Define new purge API

This is the first of a series of commits to implement the new clustered
purge API. Currently purge is a single-node only API that allows for
removing document revisions (and by extension entire documents)
completely from a database. However, given our anti-entropy measures
this API is extremely difficult to use in a cluster and requires
significant operator intervention.

Along with the operator intervention, this API is inherently unsafe with
regards to accidentally triggering the rebuild of secondary indices. As
such this patch set is aimed at creating a cluster aware API that is
both easier to use and less likely to cause application downtime while
secondary indices are rebuilt.

There are four major areas that will be covered by this patch set:

  1. Single node APIs and behavior changes
  2. Cluster aware APIs
  3. Anti-entropy updates
  4. Cluster HTTP implementation

This patch set is split up into a series of commits to aid in the review
by other commiters that will hopefully allow for a logical and intuitive
progression of implementation rather than landing as a single opaque
commit covering a huge swath of the code base.

COUCHDB-3326

Co-authored-by: Mayya Sharipova <mayyas@ca.ibm.com>
Co-authored-by: jiangphcn <jiangph@cn.ibm.com>
14 hours agoMerge pull request #1369 from apache/COUCHDB-3326-clustered-purge-pr4-on-compact...
Peng Hui Jiang [Tue, 21 Aug 2018 16:53:20 +0000 (00:53 +0800)] 
Merge pull request #1369 from apache/COUCHDB-3326-clustered-purge-pr4-on-compact-plugin

[4/5] Clustered Purge: Add on_compact EPI hook

15 hours agoCreate new on_compact trigger COUCHDB-3326-clustered-purge-pr4-on-compact-plugin 1369/head
Paul J. Davis [Wed, 30 May 2018 19:34:44 +0000 (14:34 -0500)] 
Create new on_compact trigger

This trigger allows any storage engine that makes use of compaction to
notify that compaction is starting. This is preparatory work for
clustered indexes so that existing indexes are allowed to ensure they
have a clustered purge local doc before compaction runs.

COUCHDB-3326

Co-Authored-By: jiangphcn <jiangph@cn.ibm.com>
15 hours agoMerge pull request #1368 from apache/COUCHDB-3326-clustered-purge-pr3-refactor-pse...
Peng Hui Jiang [Tue, 21 Aug 2018 16:04:43 +0000 (00:04 +0800)] 
Merge pull request #1368 from apache/COUCHDB-3326-clustered-purge-pr3-refactor-pse-tests

[3/5] Clustered Purge - Rewrite pluggable storage engine tests

16 hours agoEnhance PSE tests with setup/teardown functions COUCHDB-3326-clustered-purge-pr3-refactor-pse-tests 1368/head
Paul J. Davis [Wed, 23 May 2018 20:31:08 +0000 (15:31 -0500)] 
Enhance PSE tests with setup/teardown functions

16 hours agoUpdate to use new couch_pse_tests app
Paul J. Davis [Wed, 23 May 2018 19:31:26 +0000 (14:31 -0500)] 
Update to use new couch_pse_tests app

16 hours agoUpdate PSE test definitions for new util module
Paul J. Davis [Wed, 23 May 2018 19:28:32 +0000 (14:28 -0500)] 
Update PSE test definitions for new util module

16 hours agoRename PSE test modules
Paul J. Davis [Wed, 23 May 2018 19:23:07 +0000 (14:23 -0500)] 
Rename PSE test modules

16 hours agoMove PSE tests to their own app
Paul J. Davis [Fri, 4 May 2018 13:55:33 +0000 (08:55 -0500)] 
Move PSE tests to their own app

16 hours agoRewrite the PSE test suite to use couch_server
Paul J. Davis [Thu, 26 Apr 2018 16:41:01 +0000 (11:41 -0500)] 
Rewrite the PSE test suite to use couch_server

It turns out that if any storage engine has to open itself during
a callback it would end up violating the guarantee of a single writer.
This change in the test suite changes things to use couch_server so that
storage engines are now free to do as they want reopening themselves.

16 hours agoMerge pull request #1570 from apache/COUCHDB-3326-clustered-purge-pr2-simplify-mem3-rep
Peng Hui Jiang [Tue, 21 Aug 2018 15:28:15 +0000 (23:28 +0800)] 
Merge pull request #1570 from apache/COUCHDB-3326-clustered-purge-pr2-simplify-mem3-rep

Simplify logic in mem3_rep

16 hours agoSimplify logic in mem3_rep COUCHDB-3326-clustered-purge-pr2-simplify-mem3-rep 1570/head
Paul J. Davis [Wed, 21 Mar 2018 17:23:47 +0000 (12:23 -0500)] 
Simplify logic in mem3_rep

Previously there were two separate database references and it was not
clear which was used where. This simplifies things by reducing it to a
single instance so that the logic is simpler.

17 hours agoMerge pull request #1366 from apache/COUCHDB-3326-clustered-purge-pr1-misc-cleanup
Peng Hui Jiang [Tue, 21 Aug 2018 14:16:50 +0000 (22:16 +0800)] 
Merge pull request #1366 from apache/COUCHDB-3326-clustered-purge-pr1-misc-cleanup

[1/5] Clustered Purge - Misc Cleanup

30 hours agoUpdate fabric_doc_open eunit tests 1366/head
Paul J. Davis [Wed, 23 May 2018 17:32:48 +0000 (12:32 -0500)] 
Update fabric_doc_open eunit tests

Modernize and fix the eunit tests in fabric_doc_open.erl

30 hours agoFix race on couch_db:reopen/1
Paul J. Davis [Thu, 26 Apr 2018 16:39:58 +0000 (11:39 -0500)] 
Fix race on couch_db:reopen/1

This fixes a minor race by opening the database before closing it. This
was never found to be an issue in production and was just caught while
contemplating the PSE test suite.

30 hours agoFix default security object handling
Paul J. Davis [Thu, 26 Apr 2018 16:38:24 +0000 (11:38 -0500)] 
Fix default security object handling

There's a race where if a database is opened with a default_security set
and it crashes before first compact, and is then reopened after the
default_security option has changed that it will pick the second
security option. This change fixes the relatively obscure bug
that was only found during testing.

30 hours agoFix bug during purge
Paul J. Davis [Thu, 26 Apr 2018 16:36:49 +0000 (11:36 -0500)] 
Fix bug during purge

30 hours agoFix typos in couch_db_engine.erl
Paul J. Davis [Fri, 5 Feb 2016 17:49:34 +0000 (11:49 -0600)] 
Fix typos in couch_db_engine.erl

5 days agoMerge pull request #1543 from cloudant/implement-node-restart
iilyak [Thu, 16 Aug 2018 16:56:18 +0000 (09:56 -0700)] 
Merge pull request #1543 from cloudant/implement-node-restart

Add `POST /_node/$node/_restart` endpoint

6 days agoRemove no longer needed handle_restart_req handler 1543/head
ILYA Khlopotov [Wed, 15 Aug 2018 14:33:22 +0000 (07:33 -0700)] 
Remove no longer needed handle_restart_req handler

6 days agoRemove _restart endpoint from non-clustered interface
ILYA Khlopotov [Wed, 15 Aug 2018 13:49:25 +0000 (06:49 -0700)] 
Remove _restart endpoint from non-clustered interface

6 days agoRemove special handling of 'restart' from 'test/javascript/run'
ILYA Khlopotov [Wed, 15 Aug 2018 11:10:31 +0000 (04:10 -0700)] 
Remove special handling of 'restart' from 'test/javascript/run'

6 days agoUse "/_node/<node>/_restart" from JavaScript tests
ILYA Khlopotov [Wed, 15 Aug 2018 10:56:42 +0000 (03:56 -0700)] 
Use "/_node/<node>/_restart" from JavaScript tests

6 days agoCalculate uptime since application start instead of a beam start
ILYA Khlopotov [Wed, 15 Aug 2018 10:55:06 +0000 (03:55 -0700)] 
Calculate uptime since application start instead of a beam start

6 days agoAdd `POST /_node/$node/_restart` endpoint
ILYA Khlopotov [Tue, 7 Aug 2018 19:01:31 +0000 (12:01 -0700)] 
Add `POST /_node/$node/_restart` endpoint

We need to be able to restart CouchDB from integration test suite.
We used to have this feature, but it was removed.
This PR brings this functionality back under `/_node/$node/_restart`.

6 days agoReduce size of #leaf.atts keys
Nick Vatamaniuc [Wed, 15 Aug 2018 19:23:52 +0000 (15:23 -0400)] 
Reduce size of #leaf.atts keys

`#leaf.atts` data structure is a `[{Position, AttachmentLength}, ...]` proplist
which keeps track of attachment lengths and it is used when calculating
external data size of documents. `Position` is supposed to uniquely identify an
attachment in a file stream. Initially it was just an integer file offset. Then,
after some refactoring work it became a list of `{Position, Size}` tuples.

During the PSE work streams were abstracted such that each engine can supply
its own stream implementation. The position in the stream then became a tuple
that looks like `{couch_bt_engine_stream,{<0.1922.0>,[{4267,21}]}}`. This was
written to the file the `#leaf.atts` data structure. While still correct, it is
unnecessarily verbose wasting around 100 bytes per attachment, per leaf.

To fix it use the disk serialized version of the stream position as returned
from `couch_stream:to_disk_term`. In case of the default CouchDB engine
implementation, this should avoid writing the module name and the pid value for
each attachment entry.

6 days agoMerge pull request #1486 from cloudant/fix-doc-update-case-clause
iilyak [Wed, 15 Aug 2018 19:23:58 +0000 (12:23 -0700)] 
Merge pull request #1486 from cloudant/fix-doc-update-case-clause

Expose document update errors to client

8 days agoExpose document update errors to client 1486/head
Jay Doane [Sat, 4 Aug 2018 00:01:32 +0000 (17:01 -0700)] 
Expose document update errors to client

Currently, errors resulting from race conditions during document updates
don't get handled correctly, result in a case error, and sending a 500
to the client.

This change instead allows errors, which occur in the race to sync
a doc update across the cluster, to be exposed to the client.

8 days agoFix session based replicator auth when endpoints have require_valid_user set
Nick Vatamaniuc [Thu, 9 Aug 2018 20:05:04 +0000 (16:05 -0400)] 
Fix session based replicator auth when endpoints have require_valid_user set

If _session response is 401 and WWW-Authentication header is set assume
endpoint has require_valid_user set. Remember that in the state and
retry to reinitialize again. If it succeeds, keep sending basic auth
creds with every subsequent _session request.

Since session uses the replicator worker pool, it needs to handle worker
cleanup properly just like couch_replicator_httpc module does. If response
headers indicate the connection will be closed, don't recycle it back to the
pool, otherwise during an immediate retry there will be a connection_closing
error, instead follow what the server indicated and stop the worker then
release it to the pool. The pool already knows how to handle dead worker
processes. This is needed with this commit, because we now have a pattern of an
immediate retry after an auth failure.

Fixes #1550

11 days agoMerge pull request #1553 from apache/ref-match-1544
Robert Newson [Fri, 10 Aug 2018 16:05:40 +0000 (17:05 +0100)] 
Merge pull request #1553 from apache/ref-match-1544

Ensure we only receive the correct DOWN message

11 days agoEnsure we only receive the correct DOWN message 1553/head
Robert Newson [Fri, 10 Aug 2018 12:28:58 +0000 (13:28 +0100)] 
Ensure we only receive the correct DOWN message

relates to issue #1544.

13 days agoMove mango selector matching to the shard level
Garren Smith [Wed, 8 Aug 2018 07:56:10 +0000 (09:56 +0200)] 
Move mango selector matching to the shard level

This moves the Mango selector matching down to the shard level.
this would mean that the document is retrieved from the index and
matched against the selector before being sent to the coordinator node.
This reduces the network traffic for a mango query

Co-authored-by: Paul J. Davis <paul.joseph.davis@gmail.com>
Co-authored-by: Garren Smith <garren.smith@gmail.com>
13 days agoAdd rexi ping message
Garren Smith [Wed, 8 Aug 2018 07:53:47 +0000 (09:53 +0200)] 
Add rexi ping message

Add a ping message to rexi to avoid any long running operations from
timing out. Long running operations at the node level can exceed the
fabric timeout and be cancelled. Sending a ping message back will
stop that from happening.

13 days agoMerge pull request #1432 from cloudant/support-callback-module-data-provider
iilyak [Wed, 8 Aug 2018 14:27:37 +0000 (07:27 -0700)] 
Merge pull request #1432 from cloudant/support-callback-module-data-provider

Support callback module data provider

13 days agoUpdate documentation 1432/head
ILYA Khlopotov [Wed, 11 Jul 2018 16:20:10 +0000 (09:20 -0700)] 
Update documentation

2 weeks agoAdd 'callback_module' provider
ILYA Khlopotov [Wed, 11 Jul 2018 16:19:45 +0000 (09:19 -0700)] 
Add 'callback_module' provider

2 weeks agoRename 'module' data provider to 'static_module'
ILYA Khlopotov [Wed, 11 Jul 2018 16:05:01 +0000 (09:05 -0700)] 
Rename 'module' data provider to 'static_module'

2 weeks agoBump fauxton
Joan Touzet [Tue, 7 Aug 2018 18:29:16 +0000 (14:29 -0400)] 
Bump fauxton

2 weeks agoUse new mochiweb recbuf|sndbuf undefined setting
Joan Touzet [Mon, 25 Jun 2018 16:29:48 +0000 (12:29 -0400)] 
Use new mochiweb recbuf|sndbuf undefined setting

Fixes #1409

2 weeks agoadded missing dependencies for Debian-based systems
Cerem Cem ASLAN [Sun, 5 Aug 2018 04:17:25 +0000 (07:17 +0300)] 
added missing dependencies for Debian-based systems

2 weeks agoFixed use of find ... -printf "%f" on Mac OS by replacing it with -exec basename
Sergey Shevelev [Sun, 5 Aug 2018 01:27:08 +0000 (18:27 -0700)] 
Fixed use of find ... -printf "%f" on Mac OS by replacing it with -exec basename

Fixes issue #1028

2 weeks agoMerge pull request #1484 from cloudant/fix-stats-reducer-for-array
iilyak [Wed, 1 Aug 2018 21:02:22 +0000 (14:02 -0700)] 
Merge pull request #1484 from cloudant/fix-stats-reducer-for-array

Fix _stats reducer when map function emits array

2 weeks agoFix _stats reducer when map function emits array 1484/head
ILYA Khlopotov [Wed, 1 Aug 2018 19:31:40 +0000 (12:31 -0700)] 
Fix _stats reducer when map function emits array

3 weeks agoremove obsolete update_notification feature
Jan Lehnardt [Tue, 31 Jul 2018 14:56:43 +0000 (16:56 +0200)] 
remove obsolete update_notification feature

3 weeks agobump fauxton
Joan Touzet [Tue, 31 Jul 2018 13:10:29 +0000 (09:10 -0400)] 
bump fauxton

3 weeks agoImprove logging from test_util
ILYA Khlopotov [Thu, 26 Jul 2018 13:08:46 +0000 (06:08 -0700)] 
Improve logging from test_util

Sometimes it is hard to guess what went wrong when application
started via `test_util:start_applications` or `test_util:start_couch`
is unable to start. Since the traceback was truncated.
This change would print the reason in addition to the traceback.

4 weeks agobump docs dependency 2.2.0-RC1
Joan Touzet [Tue, 24 Jul 2018 11:40:11 +0000 (07:40 -0400)] 
bump docs dependency

4 weeks ago[travis] 20.1 -> 20.3
Joan Touzet [Tue, 24 Jul 2018 07:27:58 +0000 (03:27 -0400)] 
[travis] 20.1 -> 20.3

4 weeks agoBump fauxton dependency
Joan Touzet [Tue, 24 Jul 2018 05:32:33 +0000 (01:32 -0400)] 
Bump fauxton dependency

4 weeks agoEnable replication client _session auth by default
Joan Touzet [Mon, 23 Jul 2018 23:49:00 +0000 (19:49 -0400)] 
Enable replication client _session auth by default

Relates to #1153 and #1176

4 weeks agoRevert "Add bcrypt hashing option"
Joan Touzet [Wed, 18 Jul 2018 16:31:42 +0000 (12:31 -0400)] 
Revert "Add bcrypt hashing option"

This reverts commit 817b2b6f5f0883092df60c1ec8ec7ec6d6094a23.

4 weeks agoRevert "Replace resource expensive bcrypt test with shorter version (#1231)"
Joan Touzet [Wed, 18 Jul 2018 16:23:49 +0000 (12:23 -0400)] 
Revert "Replace resource expensive bcrypt test with shorter version (#1231)"

This reverts commit 89a727b625e74f40fcf612bda18421b8fc21eead.

4 weeks agoRevert "Make loginUser wait for successful authentication"
Joan Touzet [Wed, 18 Jul 2018 16:23:36 +0000 (12:23 -0400)] 
Revert "Make loginUser wait for successful authentication"

This reverts commit 6ffe0421ac524138a6b48fd010f2540631b83d16.

4 weeks agobump hyper dependency, fix Windows build
Joan Touzet [Wed, 18 Jul 2018 16:15:22 +0000 (12:15 -0400)] 
bump hyper dependency, fix Windows build

5 weeks agoSwitch fabric attachment receiver to use messages
Nick Vatamaniuc [Tue, 17 Jul 2018 18:53:26 +0000 (14:53 -0400)] 
Switch fabric attachment receiver to use messages

Function closures are fragile and attachment uploads would break if
fabric_doc_attachments is different on any of the nodes in a cluster, like it
might happen during a roling cluster upgrade.

Previously fe53e437ca5ec9d23aa1b55d7934daced157a9e3 introduced two versions of
the module one which handles function closure, other which handled message but
only in this commit messages are starting to be sent.

To avoid breakages in attachemnt uploads during the period of a rolling cluster
upgrade, make sure to upgrade to that commit first then upgrade to this change.

Issue #1394

5 weeks agoAdd missing default case clause, fixes #1450
Joan Touzet [Tue, 17 Jul 2018 18:00:21 +0000 (14:00 -0400)] 
Add missing default case clause, fixes #1450

5 weeks agore-raise max_http_request_size to 4GB
Jan Lehnardt [Mon, 16 Jul 2018 08:37:33 +0000 (10:37 +0200)] 
re-raise max_http_request_size to 4GB

5 weeks agoadd retry to test_request:request()
Jan Lehnardt [Mon, 16 Jul 2018 10:47:23 +0000 (12:47 +0200)] 
add retry to test_request:request()

5 weeks agoImprove detection of git tags/dirty status
Joan Touzet [Mon, 16 Jul 2018 20:56:52 +0000 (16:56 -0400)] 
Improve detection of git tags/dirty status

Unfortuantely, #1437 brought in a build bug that caused dist tarballs
to be created always using the last tag that could be found on the
tree. This lead to `master` building tarballs labelled `2.1.0`.

The new approach includes extensive comments to explain the approach,
fixes the bug, and for an encore adds -dirty if you're building a
CouchDB with local changes that aren't committed to git.

5 weeks agoReport git sha on welcome request
Jan Lehnardt [Fri, 13 Jul 2018 12:26:43 +0000 (14:26 +0200)] 
Report git sha on welcome request

Closes #1309

5 weeks agoMake MD5 hash implementation configurable (#1171)
rokek [Mon, 16 Jul 2018 19:38:07 +0000 (15:38 -0400)] 
Make MD5 hash implementation configurable (#1171)

5 weeks agoimprove ddoc test reliablilty by waiting for ddoc deletion
Jan Lehnardt [Mon, 16 Jul 2018 10:01:54 +0000 (12:01 +0200)] 
improve ddoc test reliablilty by waiting for ddoc deletion

5 weeks agoOptionally prevent non-admins from accessing /_all_dbs
Jan Lehnardt [Sat, 14 Jul 2018 10:01:58 +0000 (12:01 +0200)] 
Optionally prevent non-admins from accessing /_all_dbs

5 weeks agodemote notice to debug logs by @wyc
Jan Lehnardt [Sat, 14 Jul 2018 09:38:49 +0000 (11:38 +0200)] 
demote notice to debug logs by @wyc

5 weeks agovalidate bind address by @wyc
Jan Lehnardt [Sat, 14 Jul 2018 09:37:12 +0000 (11:37 +0200)] 
validate bind address by @wyc

5 weeks agoClarify bad index creation error messages
Brandon Gottlob [Wed, 2 May 2018 11:15:55 +0000 (07:15 -0400)] 
Clarify bad index creation error messages

Fixes #927

5 weeks agowait for db deletion before db re-creation. Closes #1197
Jan Lehnardt [Sat, 14 Jul 2018 10:51:02 +0000 (12:51 +0200)] 
wait for db deletion before db re-creation. Closes #1197

5 weeks agoraise timeouts in attachment tests
Jan Lehnardt [Sat, 14 Jul 2018 10:54:47 +0000 (12:54 +0200)] 
raise timeouts in attachment tests

5 weeks agobump deps
Jan Lehnardt [Fri, 13 Jul 2018 15:44:06 +0000 (17:44 +0200)] 
bump deps

5 weeks agostring:trim() compat for couch_util:trim()
Jan Lehnardt [Sun, 17 Jun 2018 15:17:37 +0000 (17:17 +0200)] 
string:trim() compat for couch_util:trim()

5 weeks agoUse couch_util:trim for greater erlang compatibility
Robert Newson [Tue, 8 May 2018 18:44:51 +0000 (19:44 +0100)] 
Use couch_util:trim for greater erlang compatibility

5 weeks agobuild: release candidate tarball should have -RCx
Dave Cottlehuber [Wed, 2 May 2018 09:39:25 +0000 (09:39 +0000)] 
build: release candidate tarball should have -RCx

the released tarball should have -RCx in the name, but not in the
extracted file, otherwise we can't simply rename the final artefact for
our public release.

5 weeks agobuild: ease pattern matching to be less pedantic about RC
Dave Cottlehuber [Wed, 2 May 2018 08:13:33 +0000 (08:13 +0000)] 
build: ease pattern matching to be less pedantic about RC

git-describe may have changed output formats. The previous regex doesn't
allow any trailing content, and git-describe always appends -g<SHA> unless
abbrev=0 is added.

This approach pulls out only the matching tag, leaving behind any
trailing garbage from git-describe, and as a bonus allows tagging a
commit from within a branch, and running a release directly from the
branch without needing to re-check out the tag itself.

5 weeks agoconfig: improve handling of admin-supplied changes
Dave Cottlehuber [Mon, 30 Apr 2018 10:10:26 +0000 (10:10 +0000)] 
config: improve handling of admin-supplied changes

- send a readable error response from failed config set
- trust but verify admin-supplied content in separate function
- return specific error conditions for logging

5 weeks agoDrop Erlang R16 support
Jan Lehnardt [Mon, 9 Jul 2018 14:11:32 +0000 (16:11 +0200)] 
Drop Erlang R16 support

5 weeks agoReplace R16B03 with 17.5.3 for PR#1427
Joan Touzet [Fri, 13 Jul 2018 15:34:39 +0000 (11:34 -0400)] 
Replace R16B03 with 17.5.3 for PR#1427

5 weeks agoViews now retain update_seq after compaction.
Jan Lehnardt [Fri, 13 Jul 2018 11:39:28 +0000 (13:39 +0200)] 
Views now retain update_seq after compaction.

Previously on view compaction, the new index state would not carry
over the associated database update and purge sequences. As a
result, views who were compacted at least once could no longer
use the ?update_seq query option, or purge properly.

Closes #984

5 weeks agohook up proxy auth handler to chttpd
Jan Lehnardt [Fri, 13 Jul 2018 15:17:59 +0000 (17:17 +0200)] 
hook up proxy auth handler to chttpd

5 weeks agoAdd tests for mango conflict finding
Jan Lehnardt [Fri, 13 Jul 2018 13:08:46 +0000 (15:08 +0200)] 
Add tests for mango conflict finding

I couldn’t find a cleaner way to add _bulk_docs with
`new_edits: false`.

5 weeks agoAdd `conflicts: true` option to mango selectors
Jan Lehnardt [Sun, 8 Jul 2018 13:48:39 +0000 (15:48 +0200)] 
Add `conflicts: true` option to mango selectors

This allows for using Mango queries for finding docs with conflicts.

Closes #1101

5 weeks agoAdjust deletion tests in different cluster quorum conditions
jjrodrig [Mon, 9 Jul 2018 21:10:42 +0000 (23:10 +0200)] 
Adjust deletion tests in different cluster quorum conditions

5 weeks agoDeletion responds 200 after a response from every node, and 202 in other case
jjrodrig [Thu, 29 Mar 2018 21:34:42 +0000 (23:34 +0200)] 
Deletion responds 200 after a response from every node, and 202 in other case

5 weeks agoFix for issue #1136 - Error 500 deleting DB without quorum
jjrodrig [Thu, 1 Feb 2018 18:31:29 +0000 (19:31 +0100)] 
Fix for issue #1136 - Error 500 deleting DB without quorum

Complete deletion tests with not found

5 weeks agoMake stem_interactive_updates option work again
Nick Vatamaniuc [Wed, 11 Jul 2018 17:35:25 +0000 (13:35 -0400)] 
Make stem_interactive_updates option work again

After the aebdbc452573f70f4e50d88af5814d0fbe936333 stemming is done separately
from merge so stem interactive option didn't take effect. That is mostly ok as
speed improvements should reduce the need for that option, but it still might
be nice to keep the option (just in case).

Also, a nice side effect is it removes an extra external function from
couch_key_tree module and simplifies the tests a bit.

Related PR: #958

6 weeks agoAdded tests for checking http status code depending on cluster quorum
jjrodrig [Fri, 9 Feb 2018 09:22:10 +0000 (10:22 +0100)] 
Added tests for checking http status code depending on cluster quorum

6 weeks agofeat: remove duplicate authentication check
Jan Lehnardt [Tue, 27 Mar 2018 14:29:32 +0000 (16:29 +0200)] 
feat: remove duplicate authentication check

6 weeks agofeat: avoid double builds on PRs
Jan Lehnardt [Sun, 8 Jul 2018 13:57:34 +0000 (15:57 +0200)] 
feat: avoid double builds on PRs

6 weeks agoallow socket server configuration for TLS httpd
Jan Lehnardt [Fri, 29 Jun 2018 17:31:44 +0000 (19:31 +0200)] 
allow socket server configuration for TLS httpd

6 weeks agoremove mention of os daemons and externals in ini files
Jan Lehnardt [Wed, 16 May 2018 16:44:16 +0000 (18:44 +0200)] 
remove mention of os daemons and externals in ini files

7 weeks agoDon't set context for compaction files during db delete (#1419)
Eric Avdey [Fri, 29 Jun 2018 21:31:55 +0000 (18:31 -0300)] 
Don't set context for compaction files during db delete (#1419)

When we delete files with context option set to `delete`,
`couch_file` respects configuration flag "enable_database_recovery"
and just renames the files in case it's set to true.

This change removes context for compaction files deleted
during database deletion to make sure we are actually
erasing them and not just renaming and leaving behind.