couchdb-fabric.git
6 years agoreturn packed update_seq (WIP) 2849-update_seq_view
Robert Newson [Wed, 30 Mar 2016 11:14:20 +0000 (12:14 +0100)] 
return packed update_seq (WIP)

6 years agoExtent fetch ddoc logic for view filtered changes feeds 40/head
Eric Avdey [Mon, 29 Feb 2016 20:07:09 +0000 (16:07 -0400)] 
Extent fetch ddoc logic for view filtered changes feeds

6 years agoAvoid spamming logs with write_quorum errors
Mike Wallace [Thu, 3 Mar 2016 17:13:48 +0000 (17:13 +0000)] 
Avoid spamming logs with write_quorum errors

The three metrics in fabric_doc_update:force_reply/3 were
introduced to replace the write_quorum log message however the
log line was not removed. This commit removes the log message.

Close COUCHDB-2958

This closes #41

6 years ago Merge remote branch 'github/pr/39'
ILYA Khlopotov [Wed, 2 Mar 2016 20:36:37 +0000 (12:36 -0800)] 
Merge remote branch 'github/pr/39'

    - https://github.com/apache/couchdb-fabric/pull/39

    This closes #39

Signed-off-by: ILYA Khlopotov <iilyak@ca.ibm.com>
6 years agoRemove configurable replicator db name 39/head
Nick Vatamaniuc [Fri, 26 Feb 2016 21:11:39 +0000 (16:11 -0500)] 
Remove configurable replicator db name

JIRA: COUCHDB-2954

6 years agoMerge branch '2938-use-ddoc-cache-in-filtered-changes'
Russell Branca [Wed, 10 Feb 2016 22:31:07 +0000 (22:31 +0000)] 
Merge branch '2938-use-ddoc-cache-in-filtered-changes'

6 years agoSupport fetch ddoc logic for filtered changes feeds 2938-use-ddoc-cache-in-filtered-changes 37/head
Russell Branca [Thu, 21 Jan 2016 22:55:13 +0000 (22:55 +0000)] 
Support fetch ddoc logic for filtered changes feeds

6 years agoReturn forbidden error if encountered on any shard
Mike Wallace [Tue, 9 Feb 2016 14:31:55 +0000 (14:31 +0000)] 
Return forbidden error if encountered on any shard

This commit fixes an issue which caused HTTP 500 errors to be
returned when an authorized user attempted to access a database
that they did not have permission to access, when cassim is
disabled.

Instead of returning an internal server error once all shards
have failed to open we add a receive clause so that we throw
a forbidden error if one is encountered on any shard. This is the
same approach we already take for unauthorized errors.

Closes COUCHDB-2948

This closes #38

6 years agoAdd owner of shard to last_seq of changes response
Benjamin Bastian [Wed, 20 Jan 2016 00:29:34 +0000 (16:29 -0800)] 
Add owner of shard to last_seq of changes response

Prior to this commit, there was a bug where the response of a changes
query would include epoch information for every change, but not for the
last sequence. Subsequent shard changes queries would rewind from zero
if the last_seq field was used in the clustered changes query and any
shard had previously been moved.

6 years agoFix incorrect size values for view index information 34/head
Alexander Shorin [Tue, 27 Oct 2015 13:55:42 +0000 (16:55 +0300)] 
Fix incorrect size values for view index information

We need to filter overlapping shards to prevent summarizing
all sizes from all the shards, since we actually need the
value from the one.

This bug was introduced in ba27f81.

6 years agoUpdate .travis.yml
Alexander Shorin [Wed, 7 Oct 2015 17:57:02 +0000 (20:57 +0300)] 
Update .travis.yml

6 years agoFix eunit tests. 32/head
Nick Vatamaniuc [Fri, 2 Oct 2015 18:47:10 +0000 (14:47 -0400)] 
Fix eunit tests.

 Mock couch_log alongside couch_stats.

6 years agofix assertions: first arg is 'expected'
Robert Newson [Tue, 22 Sep 2015 13:18:07 +0000 (14:18 +0100)] 
fix assertions: first arg is 'expected'

6 years agoDon't create dbs doc if shard files fail for enametoolong reason 31/head
Robert Newson [Tue, 22 Sep 2015 12:10:23 +0000 (13:10 +0100)] 
Don't create dbs doc if shard files fail for enametoolong reason

COUCHDB-2821

6 years agoMerge remote-tracking branch 'smithsz/2819-db-create-check-already-exists'
Robert Newson [Tue, 22 Sep 2015 12:56:19 +0000 (13:56 +0100)] 
Merge remote-tracking branch 'smithsz/2819-db-create-check-already-exists'

6 years agoCheck if a DB exists prior to creating 30/head
Sam Smith [Mon, 21 Sep 2015 15:05:16 +0000 (16:05 +0100)] 
Check if a DB exists prior to creating

COUCHDB-2819

6 years agoAdded counter metric for fabric-worker-timeout.
Bryan Green [Sat, 8 Aug 2015 00:06:14 +0000 (19:06 -0500)] 
Added counter metric for fabric-worker-timeout.

This is being done as a good indicator for when mspot is
having issues.  An alert threshold will be set in
centinela.

BugzID: 50369

6 years agoRevert "add clusterwide compaction for dbs"
Robert Newson [Sat, 12 Sep 2015 18:30:58 +0000 (19:30 +0100)] 
Revert "add clusterwide compaction for dbs"

This reverts commit 140bcfacbb220bc3b47697fdd7b8b306eb6d7330.

6 years agoadd clusterwide compaction for dbs
Robert Kowalski [Fri, 21 Aug 2015 12:16:34 +0000 (14:16 +0200)] 
add clusterwide compaction for dbs

PRs:
https://github.com/apache/couchdb-fabric/pull/27
https://github.com/apache/couchdb-chttpd/pull/62

COUCHDB-2492

PR: #27
PR-URL: https://github.com/apache/couchdb-fabric/pull/27
Reviewed-By: Robert Newson <rnewson@apache.org>
6 years agoUse BigCouch/Cloudant seq format
Robert Newson [Mon, 24 Aug 2015 11:11:58 +0000 (12:11 +0100)] 
Use BigCouch/Cloudant seq format

Let's ease 2.0 transition by not introducing a new sequence format,
it's not even necessary now.

6 years agoIntegrate with Travis CI 29/head
Alexander Shorin [Mon, 24 Aug 2015 11:30:01 +0000 (14:30 +0300)] 
Integrate with Travis CI

6 years agoFix type specs
Klaus Trainer [Wed, 19 Aug 2015 09:52:46 +0000 (11:52 +0200)] 
Fix type specs

6 years agofabric:cleanup_index_files should check mrview/ directories 25/head
Robert Newson [Thu, 13 Aug 2015 12:59:00 +0000 (13:59 +0100)] 
fabric:cleanup_index_files should check mrview/ directories

COUCHDB-2773

6 years agoUse `couch_db:validate_dbname/1` 24/head
ILYA Khlopotov [Fri, 31 Jul 2015 18:41:48 +0000 (11:41 -0700)] 
Use `couch_db:validate_dbname/1`

6 years agoMerge branch '2724-chunked-buffering'
Adam Kocoloski [Wed, 22 Jul 2015 16:31:12 +0000 (12:31 -0400)] 
Merge branch '2724-chunked-buffering'

6 years agoRestrict views in the authentication DB to admins
Mike Wallace [Wed, 15 Jul 2015 09:54:05 +0000 (10:54 +0100)] 
Restrict views in the authentication DB to admins

This commit teaches fabric to restrict access to views in the
authentication DB to administrators by calling through to
couch_db_users:after_doc_read/2 after the design doc is read.

COUCHDB-2738

This closes #23

6 years agoMove is_*_db funs and helpers to fabric_util
Mike Wallace [Wed, 15 Jul 2015 09:50:21 +0000 (10:50 +0100)] 
Move is_*_db funs and helpers to fabric_util

This commit moves the is_*_db/1 funs and their helpers from
fabric_doc_update to fabric_util and exports them so they
can be re-used elsewhere in fabric.

COUCHDB-2738

7 years agoExecute a callback at end of each traversal 2724-chunked-buffering 22/head 96/head
Adam Kocoloski [Wed, 24 Jun 2015 18:05:20 +0000 (14:05 -0400)] 
Execute a callback at end of each traversal

We add this notification before entering a (potentially long) wait for
new DB updates so that the client can choose to finalize any processing
it might have been doing on earlier updates. The specific use case in
chttpd is to flush the client's internal buffer in a timely manner.

COUCHDB-2724

7 years agoRemove meck twig:log/3
Alexander Shorin [Wed, 17 Jun 2015 09:24:36 +0000 (12:24 +0300)] 
Remove meck twig:log/3

7 years agoUse couch_db:normalize_dbname 21/head
ILYA Khlopotov [Mon, 15 Jun 2015 12:54:08 +0000 (05:54 -0700)] 
Use couch_db:normalize_dbname

COUCHDB-2715

7 years agoRevert "Module-wide s/proplists:get_value/couch_util:get_value/"
Alexander Shorin [Thu, 11 Jun 2015 01:18:51 +0000 (04:18 +0300)] 
Revert "Module-wide s/proplists:get_value/couch_util:get_value/"

Behaviour of `get_value/2,3` function of couch_util and proplists
modules is completely different when searched key exists and is an
atom, but there is no paired value for it: proplists function
implicitly assumes that a value is `true` while couch_util returns
`undefined` instead.

Such replacement breaks any logic that depends on `true` or
`undefined` result of the function call, so we need to be
careful next time.

This reverts commit 19c273ac54e8f1bec81be3b74c443ecafa01e599.

7 years agoModule-wide s/proplists:get_value/couch_util:get_value/ merge-diff-from-cloudant-fork 20/head
Mike Wallace [Fri, 5 Jun 2015 16:01:13 +0000 (17:01 +0100)] 
Module-wide s/proplists:get_value/couch_util:get_value/

7 years agoexport dbname/1
Robert Newson [Thu, 9 Apr 2015 22:07:19 +0000 (23:07 +0100)] 
export dbname/1

BugzID: 45853

This is a cherry-pick of:

https://github.com/cloudant/fabric/commit/818dd09e9ea645e866fc62fd1a4302ce73b2422a

7 years agoAdd ability to gracefully terminate existing changes feeds
Robert Newson [Sun, 29 Mar 2015 12:38:38 +0000 (13:38 +0100)] 
Add ability to gracefully terminate existing changes feeds

Calling fabric:end_changes() will cause all current changes
feeds to gracefully exit without preventing new changes requests from
starting (unlike maintenance_mode).

BugzID: 45762

This is a cherry-pick of:

https://github.com/cloudant/fabric/commit/1b45cede8f11e209f28e3d06b9fda4cbdcd719cc

Conflicts:
src/fabric.erl
src/fabric_view_changes.erl

7 years agoProvide an access to a document info
Eric Avdey [Tue, 11 Nov 2014 00:12:42 +0000 (20:12 -0400)] 
Provide an access to a document info

The functions `get_doc_info/3` and `get_full_doc_info/3` were
added to API to provide an access to the records `#doc_info{}`
and `#full_doc_info{}` accordingly.

The functions are re-using `fabric_open_doc` coordinator
and consequently are the subject of the same read quorum rules
as `open_doc/3` function. However the info functions do not trigger
read repair on a not fully complete quorum.

Function `get_full_doc_info/3` accepts an option `deleted` to
allow to provide an information for a deleted document,
similar to `open_doc/3`.

FogBugz: 12933

This is a cherry-pick of:

https://github.com/cloudant/fabric/commit/c85569287ad8f86122b47775adc2ab9218db0322

Conflicts:
src/fabric_rpc.erl

7 years agouse the right config section for clustered authentication db
Robert Newson [Tue, 19 May 2015 11:53:07 +0000 (12:53 +0100)] 
use the right config section for clustered authentication db

7 years agoMerge remote-tracking branch 'iilyak/2526-pending-updates-info'
Alexander Shorin [Tue, 12 May 2015 19:52:39 +0000 (22:52 +0300)] 
Merge remote-tracking branch 'iilyak/2526-pending-updates-info'

This closes #9

COUCHDB-2526

7 years agoFix typo
Russell Branca [Fri, 24 Apr 2015 00:21:02 +0000 (00:21 +0000)] 
Fix typo

7 years agoHandle case in fake_db where there's no user context
Robert Newson [Thu, 9 Apr 2015 16:50:54 +0000 (17:50 +0100)] 
Handle case in fake_db where there's no user context

7 years agoCall the correct before_doc_update arity 2643-fix-arity
Robert Newson [Tue, 31 Mar 2015 13:04:19 +0000 (14:04 +0100)] 
Call the correct before_doc_update arity

bug introduced in COUCHDB-510

fixes COUCHDB-2630 COUCHDB-2632 COUCHDB-2643

7 years agoRename variables for clarification of intend 9/head
ILYA Khlopotov [Tue, 23 Dec 2014 15:54:02 +0000 (07:54 -0800)] 
Rename variables for clarification of intend

COUCHDB-2526

7 years agoAdd updates_pending field to get_view_group_info
ILYA Khlopotov [Mon, 22 Dec 2014 21:57:38 +0000 (13:57 -0800)] 
Add updates_pending field to get_view_group_info

We also change the beheviour of this function to wait
response from all shards.

COUCHDB-2526

7 years agoadd license file
Robert Kowalski [Sat, 28 Feb 2015 22:16:59 +0000 (23:16 +0100)] 
add license file

7 years agoreadme: change to couchdb project
Robert Kowalski [Sat, 28 Feb 2015 22:18:46 +0000 (23:18 +0100)] 
readme: change to couchdb project

7 years agoremove compile warning 16/head
Robert Kowalski [Sat, 28 Feb 2015 22:07:20 +0000 (23:07 +0100)] 
remove compile warning

make `Options` optional :)

7 years agoUse list of system database names
Alexander Shorin [Wed, 25 Feb 2015 19:03:21 +0000 (22:03 +0300)] 
Use list of system database names

This closes #15

COUCHDB-2619
COUCHDB-2620

7 years agoRespect doc_options during _all_docs serialization
Will Holley [Fri, 6 Feb 2015 17:52:31 +0000 (17:52 +0000)] 
Respect doc_options during _all_docs serialization

When full documents are returned as part of an _all_docs response,
respect doc_options (e.g. attachments=true) during JSON serialization.

This closes #13

COUCHDB-2568

Signed-off-by: Alexander Shorin <kxepal@apache.org>
7 years agoFix waterfall of `rexi_server error:badarg` errors
Alexander Shorin [Thu, 5 Feb 2015 13:05:20 +0000 (16:05 +0300)] 
Fix waterfall of `rexi_server error:badarg` errors

7 years agoRespect doc_options during _changes serialization
Will Holley [Mon, 2 Feb 2015 21:02:27 +0000 (21:02 +0000)] 
Respect doc_options during _changes serialization

When including full documents in the changes feed response, respect
doc_options (e.g. attachments=true) during JSON serialization.

This closes #12

COUCHDB-2522

Signed-off-by: Alexander Shorin <kxepal@apache.org>
7 years agoRespect conflicts=true when collecting clustered changes
Will Holley [Sun, 1 Feb 2015 21:34:05 +0000 (21:34 +0000)] 
Respect conflicts=true when collecting clustered changes

This correctly reads the conflicts parameter from #changes_args.
Previously we were attempting to look for a conflicts property in
Options which was never set (so always evaluated to false).

Fixes COUCHDB-2518

This closes #11

Signed-off-by: Alexander Shorin <kxepal@apache.org>
7 years agoHandle empty list of id/revs in fabric_doc_missing_revs:go
Will Holley [Sat, 31 Jan 2015 20:20:02 +0000 (20:20 +0000)] 
Handle empty list of id/revs in fabric_doc_missing_revs:go

When AllIdsRevs is an empty list the result will always be an empty
response. Handle this explicitly given there is no work to do
(attempting to delegate to the shards previously resulted in a timeout).

This closes #10

COUCHDB-2531

Signed-off-by: Alexander Shorin <kxepal@apache.org>
7 years agoMerge remote-tracking branch 'iilyak/1145-_local_docs-handler'
Alexander Shorin [Fri, 30 Jan 2015 06:45:56 +0000 (09:45 +0300)] 
Merge remote-tracking branch 'iilyak/1145-_local_docs-handler'

This closes #8

COUCHDB-1145

7 years agoUse ADMIN_CTX macro from couch_db.hrl
Alexander Shorin [Mon, 26 Jan 2015 04:11:54 +0000 (07:11 +0300)] 
Use ADMIN_CTX macro from couch_db.hrl

7 years agoFix incomplete spec of fabric:get_view_group_info
ILYA Khlopotov [Mon, 22 Dec 2014 21:56:56 +0000 (13:56 -0800)] 
Fix incomplete spec of fabric:get_view_group_info

COUCHDB-2526

7 years agoDon't do manual filtering of _design docs 8/head
ILYA Khlopotov [Fri, 19 Dec 2014 16:36:05 +0000 (08:36 -0800)] 
Don't do manual filtering of _design docs

Since namespace support was added we can rely on it instead

COUCHDB-1145

7 years agoMerge branch '2517-open_revs=all-function_clause'
Paul J. Davis [Tue, 16 Dec 2014 20:58:43 +0000 (14:58 -0600)] 
Merge branch '2517-open_revs=all-function_clause'

Fixes #7

7 years agoExit early when there is a quorum but list of docs is empty
ILYA Khlopotov [Mon, 8 Dec 2014 15:52:12 +0000 (07:52 -0800)] 
Exit early when there is a quorum but list of docs is empty

There is no point to go through repair stage since this set of
conditions meens that the doc with given id doesn't exists.
Besides not going through repair process fixes the case when
we request open_revs=all for non existent document

COUCHDB-2517

7 years agoMerge branch '2510-add-system-db-handlers'
Paul J. Davis [Fri, 12 Dec 2014 20:35:08 +0000 (14:35 -0600)] 
Merge branch '2510-add-system-db-handlers'

Fixes #6

7 years agoAdd system db handlers to fabric_doc_update 2510-add-system-db-handlers 6/head 94/head
Paul J. Davis [Fri, 12 Dec 2014 02:13:57 +0000 (20:13 -0600)] 
Add system db handlers to fabric_doc_update

This hard codes the before_doc_write functions for replicator and user
databases like they are in couch_server. This is mostly motivated by the
fact that couch_users_db:before_doc_update/1 adds random salt values to
user documents when a password is changed. The replicator databases are
included at this level for consistency with couch_server.

COUCHDB-2510

7 years agoFix compilation warning
Alexander Shorin [Fri, 10 Oct 2014 15:56:56 +0000 (19:56 +0400)] 
Fix compilation warning

7 years agoDon't flatten iolist
Robert Newson [Mon, 22 Sep 2014 13:18:58 +0000 (14:18 +0100)] 
Don't flatten iolist

7 years agoReturn unauthorized error if encountered on any shard
Robert Newson [Tue, 30 Sep 2014 16:24:59 +0000 (17:24 +0100)] 
Return unauthorized error if encountered on any shard

We had a mismatch here that caused us to return an unpleasant 500
status code (after discarding lots of unauthorized events from each
shard).

The mismatch was introduced in "Teach fabric_util:get_db/2 about maintenance
mode".

I also simplified the open_shard function and tuple syntax for further
clarity in this critical section.

7 years agoUse couch_log instead of ?LOG_*
Alexander Shorin [Wed, 24 Sep 2014 20:26:32 +0000 (00:26 +0400)] 
Use couch_log instead of ?LOG_*

7 years agoRestore write_quorum_errors metric collection
Robert Newson [Mon, 15 Sep 2014 13:37:41 +0000 (14:37 +0100)] 
Restore write_quorum_errors metric collection

7 years agoDon't receive foreign rexi replies
Robert Newson [Fri, 12 Sep 2014 08:55:38 +0000 (09:55 +0100)] 
Don't receive foreign rexi replies

7 years agoTeach fabric_util:get_db/2 about maintenance mode 2325-teach-fabric-get_db-about-maintenance-mode
Mike Wallace [Wed, 10 Sep 2014 11:14:30 +0000 (12:14 +0100)] 
Teach fabric_util:get_db/2 about maintenance mode

If the node servicing a request does not have a shard for the db
involved then fabric_util:get_db/2 can return a shard from a node
which is in maintenance mode. If that node is a replacement node
that has not yet been brought into the cluster then the security
object will be empty.

Because fabric:get_security/2 calls fabric_util:get_db/2 and is in
the code path for authorizing requests at the HTTP layer, this can
result in live nodes returning 403s.

This commit replaces an rpc:call/4 with a rexi:cast/4 and adds
a new rpc endpoint in fabric_rpc for opening single shards. This
uses set_io_priority which will reply with a rexi_EXIT if
maintenance mode is set.

Closes COUCHDB-2325

7 years agoMove maintenance_mode to the couchdb section
Mike Wallace [Wed, 10 Sep 2014 10:02:21 +0000 (11:02 +0100)] 
Move maintenance_mode to the couchdb section

7 years agoAllow clients to pass a db record to query_view
Paul J. Davis [Tue, 9 Sep 2014 23:06:26 +0000 (18:06 -0500)] 
Allow clients to pass a db record to query_view

The recent updates to use couch_mrview prevented the use of a #db{}
record in concjunction with a binary GroupId because the ddoc_cache
doesn't extract the name from a db record and instead uses it as the key
directly.

7 years agoExpose new size values windsor-merge 102/head
Robert Newson [Fri, 22 Aug 2014 18:26:45 +0000 (19:26 +0100)] 
Expose new size values

7 years agoPass #changes_args.db_open_options to couch_db:open/2
Paul J. Davis [Fri, 22 Aug 2014 10:05:40 +0000 (05:05 -0500)] 
Pass #changes_args.db_open_options to couch_db:open/2

7 years agoUpdate fabric to use couch_stats
Paul J. Davis [Thu, 21 Aug 2014 06:22:16 +0000 (01:22 -0500)] 
Update fabric to use couch_stats

7 years agoAdd a no-options adapter to all_docs.
Paul J. Davis [Thu, 14 Aug 2014 18:56:04 +0000 (13:56 -0500)] 
Add a no-options adapter to all_docs.

7 years agoFix use of the rexi:stream2 API
Paul J. Davis [Thu, 14 Aug 2014 18:12:15 +0000 (13:12 -0500)] 
Fix use of the rexi:stream2 API

Bit of a mixup during the merge on this one as it included both the
switch to stream2 as well as the switch to using couch_mrview.

7 years agoFix merge of new changes behaviors
Paul J. Davis [Wed, 13 Aug 2014 21:25:40 +0000 (16:25 -0500)] 
Fix merge of new changes behaviors

7 years agosquashme! stream_last
Robert Newson [Fri, 8 Aug 2014 09:17:51 +0000 (10:17 +0100)] 
squashme! stream_last

7 years agosquashme! use stream2
Robert Newson [Thu, 7 Aug 2014 21:19:16 +0000 (22:19 +0100)] 
squashme! use stream2

7 years agoConsistently log fabric worker timeouts
Mike Wallace [Sat, 11 Jan 2014 22:11:20 +0000 (22:11 +0000)] 
Consistently log fabric worker timeouts

Write a log line for each worker that did not return a response
when a fabric request times out. The format of the log line is:

    fabric_worker_timeout ENDPOINT,NODE,SHARD_NAME

This is intented to be easily consumable by downstream tools
(e.g., Splunk).

BugzID: 26984

7 years agoSet non-interactive default for set_security
Benjamin Bastian [Thu, 13 Mar 2014 02:50:52 +0000 (22:50 -0400)] 
Set non-interactive default for set_security

Prior to this commit, fabric:set_security would default to interactive
IO priority. This would cause all nodes to set the security header
except shards on any nodes which were in MM. Subsequent calls to
fabric:get_security would cause inconsistent security properties to be
applied to the request. This patch sets a non-interactive default for
fabric:set_security, which will cause security updates to be applied to
all nodes regardless of MM by default.

BugzID: 28847

7 years agoClear worker references for get_shard_replacements
Paul J. Davis [Wed, 12 Mar 2014 06:25:35 +0000 (02:25 -0400)] 
Clear worker references for get_shard_replacements

This function relies on record equality. If a request passes workers
that have references defined it would return all copies of the shard
range incorrectly.

BugzId: 28992

7 years agoFix fabric_db_update_listener code upgrades
Paul J. Davis [Fri, 31 Jan 2014 00:40:04 +0000 (18:40 -0600)] 
Fix fabric_db_update_listener code upgrades

Changes procesess waiting idle for db update messages will never upgrade
their code. This just adds a timeout before recursing through the module
exports table to load new code.

BugzId: 27660

7 years agoInclude epoch node in seqs and replacement logic
Adam Kocoloski [Fri, 17 Jan 2014 01:32:48 +0000 (17:32 -0800)] 
Include epoch node in seqs and replacement logic

The {Seq, UUID, Node} triplet uniquely identifies an update in a
cluster.  Including the node that originally accepted the update in
sequences allows us to defend against byzantine cases of shards moving
back and forth between the nodes in a cluster.

This patch updates the sequence generation to include the full triplet
in each element of the _changes sequence.  It also uses the epoch node
instead of the node currently hosting the shard when determining the
safe replacement sequence.

BugzID: 27193

7 years agoMove attachment code into couch_att
Brian Mitchell [Thu, 12 Dec 2013 04:11:48 +0000 (23:11 -0500)] 
Move attachment code into couch_att

This is an attempt to isolate the attachment record and
some related code. This will allow seamless upgrades
over time.

7 years agoAvoid badarith error during cluster upgrade
Adam Kocoloski [Mon, 6 Jan 2014 17:02:18 +0000 (12:02 -0500)] 
Avoid badarith error during cluster upgrade

BugzID: 26631

7 years agoMake _changes?descending=true aware of maint. mode
Adam Kocoloski [Thu, 2 Jan 2014 14:18:37 +0000 (09:18 -0500)] 
Make _changes?descending=true aware of maint. mode

The _changes coordinator was using a one-off thing to get the start
sequence for a database which did not provide any error handling at all.
This patch replaces that one-off with a call to get_db_info/1, which
does include all the normal error handling.

A future enhancement could teach the coordinator to understand a
"sequence" like eof that would cause all the workers to automatically
seek to the end, but I wanted to stay conservative here in order to land
this for the next release.

BugzID: 25944

7 years agoFix logging of errors in _all_docs open_doc calls
Paul J. Davis [Fri, 13 Dec 2013 19:56:44 +0000 (13:56 -0600)] 
Fix logging of errors in _all_docs open_doc calls

Our exception handling accidentally captured the successful return
value. Using try/of/catch we can exit from the of clause without
catching.

BugzId: 26162

7 years agoDon't attempt to replace replacement shards
Paul J. Davis [Thu, 12 Dec 2013 18:58:35 +0000 (12:58 -0600)] 
Don't attempt to replace replacement shards

The old test was looking for {_, _} tuple but that doesn't account for
old since sequences. This just switches to using a negative selection.
The only bad behavior was that we were making lots of log messages about
rewinding from 0 in known conditions.

BugzId: 26125

7 years agoOnly attempt replacement when we have replacements
Paul J. Davis [Thu, 12 Dec 2013 18:51:46 +0000 (12:51 -0600)] 
Only attempt replacement when we have replacements

We weren't checking if we were even provided replacements before using
them. This just adds a guard so we don't end up with a badmatch error on
the `lists:keytake/3`.

BugzId: 26124

7 years agoThrow errors when starting changes streams
Paul J. Davis [Thu, 12 Dec 2013 18:45:10 +0000 (12:45 -0600)] 
Throw errors when starting changes streams

Previously we were calling the user supplied callback on these errors
which caused a badmatch in keep_sending_changes. This just throws the
error and lets chttpd handle formatting it for the user.

BugzId: 26122

7 years agoLog errors when doc reads fail for _all_docs
Paul J. Davis [Thu, 12 Dec 2013 15:25:39 +0000 (09:25 -0600)] 
Log errors when doc reads fail for _all_docs

This makes sure that we log any errors while opening a doc during an
_all_docs request. It also passes the failed DbName and DocId to the
user supplied callback function.

BugzId: 24580

7 years agoBe defensive when calcualting pending_count
Paul J. Davis [Tue, 10 Dec 2013 17:35:22 +0000 (11:35 -0600)] 
Be defensive when calcualting pending_count

Its theoretically possible that during the upgrade we end up with a
mixed set of RPC workers. This makes sure that we don't crash the
coordinator with a badarith error.

BugzId: 24236

7 years agoCollect pending counts and report sum at end
Adam Kocoloski [Thu, 17 Oct 2013 20:33:14 +0000 (16:33 -0400)] 
Collect pending counts and report sum at end

BugzID: 24236

7 years agoReport number of pending changes in shard
Adam Kocoloski [Thu, 17 Oct 2013 18:17:12 +0000 (14:17 -0400)] 
Report number of pending changes in shard

Also refactor the _changes accumulator into a record.

BugzID: 24236

7 years agoRefactor payload to simplify coordinator
Adam Kocoloski [Thu, 17 Oct 2013 18:03:47 +0000 (14:03 -0400)] 
Refactor payload to simplify coordinator

Two things we're doing here:

1) We're unwrapping {doc, {error, Reason}} on the RPC side.
2) We're already suppressing the doc field on the RPC side so we don't
have to duplicate the work on the coordinator.

7 years ago[2/2] Use proplist instead of #change record
Adam Kocoloski [Thu, 17 Oct 2013 17:27:58 +0000 (13:27 -0400)] 
[2/2] Use proplist instead of #change record

7 years agoAdjust replacement rewind exclusion
Paul J. Davis [Fri, 6 Dec 2013 20:44:44 +0000 (14:44 -0600)] 
Adjust replacement rewind exclusion

We're wanting to avoid attempting to use existing replacement
information for down shards. The is_integer guard didn't work because
the sequences are {UUID, Seq} tuples so we pattern match on that tuple
shape instead.

BugzId: 22698

7 years ago[squash] add deprecation info
Adam Kocoloski [Mon, 25 Nov 2013 20:31:47 +0000 (15:31 -0500)] 
[squash] add deprecation info

7 years agoUse smarter replacements for stream_start failures
Adam Kocoloski [Sat, 23 Nov 2013 03:03:22 +0000 (22:03 -0500)] 
Use smarter replacements for stream_start failures

This patch enhances the StartFun to search the original list of workers
for the sequence from which the worker being replaced had started.

BugzID: 22698

7 years agoUse smarter seqs when original shard is down/gone
Adam Kocoloski [Sun, 17 Nov 2013 23:16:07 +0000 (18:16 -0500)] 
Use smarter seqs when original shard is down/gone

If a shard that contributes to a sequence a) is on a down node or b) is
no longer part of the shard set we can provide enough detail to the
replacement workers to allow them to choose a smarter replacement
sequence.

This patch does not cover the case where the shard that originally
contributed to the sequence is alive but crashes before initializing the
stream to the coordinator.  In that case we're still firing up
replacement workers and asking them to stream from zero.  The
stream_start semantics make reworking that bit a tad trickier.

BugzID: 22698

7 years agoCalculate safe worker seq on shard replacement
Adam Kocoloski [Sun, 17 Nov 2013 17:01:05 +0000 (12:01 -0500)] 
Calculate safe worker seq on shard replacement

When a shard is being replaced in the changes feed the RPC workers for
the other shard copies can leverage epoch and checkpoint information to
start from a sequence greater than zero.  This commit (and an associated
one in mem3) make that possible, though we still need to teach the
coordinator to share the fact that the worker is being summoned to
replace another copy that previously contributed to the feed.

BugzID: 22698

7 years agoReport errors opening documents during _all_docs
Adam Kocoloski [Sat, 26 Oct 2013 02:59:23 +0000 (22:59 -0400)] 
Report errors opening documents during _all_docs

All errors are currently ignored in the receive statements and
eventually we timeout.  This patch causes fabric to report the error and
terminate quickly instead of waiting.

BugzID: 24580