couchdb-couch-index.git
2 years agoConfigurable timeout for view group get_info/1 master 20/head
Nick Vatamaniuc [Thu, 24 Nov 2016 04:05:00 +0000 (23:05 -0500)] 
Configurable timeout for view group get_info/1

A busy view group server might take longer than 5 seconds to respond, so make
the timeout configurable. "infinity" is also a valid configuration value.

Configuration section, key and default value (in milliseconds):

   `query_server_config.group_info_timeout = 5000`

COUCHDB-3242

2 years agoMerge remote branch 'cloudant:3135-remove_config_subscribe_from_couch_index'
ILYA Khlopotov [Tue, 6 Sep 2016 20:30:29 +0000 (13:30 -0700)] 
Merge remote branch 'cloudant:3135-remove_config_subscribe_from_couch_index'

This closes #19

Signed-off-by: ILYA Khlopotov <iilyak@ca.ibm.com>
2 years agoDo not use config subscription in couch_index 19/head
ILYA Khlopotov [Tue, 6 Sep 2016 20:07:28 +0000 (13:07 -0700)] 
Do not use config subscription in couch_index

COUCHDB-3135

2 years agoMerge remote branch 'cloudant:3102-fix-config_subscription'
ILYA Khlopotov [Tue, 23 Aug 2016 21:59:39 +0000 (14:59 -0700)] 
Merge remote branch 'cloudant:3102-fix-config_subscription'

This closes #18

Signed-off-by: ILYA Khlopotov <iilyak@ca.ibm.com>
2 years agoUse `define` to avoid repeating yourself 18/head
ILYA Khlopotov [Tue, 23 Aug 2016 20:17:31 +0000 (13:17 -0700)] 
Use `define` to avoid repeating yourself

COUCHDB-3102

2 years agoUpdate handle_config_terminate API
ILYA Khlopotov [Wed, 17 Aug 2016 18:50:54 +0000 (11:50 -0700)] 
Update handle_config_terminate API

COUCHDB-3102

2 years agoUse new reason formatting
Paul J. Davis [Mon, 25 Jul 2016 19:30:20 +0000 (14:30 -0500)] 
Use new reason formatting

2 years agoMerge remote branch 'cloudant:66522-hold-db-open-for-compact-recompact'
ILYA Khlopotov [Thu, 2 Jun 2016 22:02:53 +0000 (15:02 -0700)] 
Merge remote branch 'cloudant:66522-hold-db-open-for-compact-recompact'

This closes #17

Signed-off-by: ILYA Khlopotov <iilyak@ca.ibm.com>
2 years agoHold db opened for compact/recompact switch 17/head
ILYA Khlopotov [Mon, 30 May 2016 22:16:02 +0000 (15:16 -0700)] 
Hold db opened for compact/recompact switch

couch_index_compactor should hold the underlying DB open during
the switch from the compaction phase to the recompaction phase.
Otherwise db could be LRU-closed while in bettwen the two.
Which would cause the compaction job to fail possibly wasting
hours of work.

2 years agoMerge remote branch 'cloudant:65501-disable-recompaction'
ILYA Khlopotov [Tue, 24 May 2016 18:30:13 +0000 (11:30 -0700)] 
Merge remote branch 'cloudant:65501-disable-recompaction'

This closes #16

Signed-off-by: ILYA Khlopotov <iilyak@ca.ibm.com>
2 years agoAdd test suite for is_recompaction_enabled/2 16/head
ILYA Khlopotov [Tue, 17 May 2016 18:46:33 +0000 (11:46 -0700)] 
Add test suite for is_recompaction_enabled/2

2 years agoImplement a way to disable recompaction
ILYA Khlopotov [Mon, 16 May 2016 19:54:45 +0000 (12:54 -0700)] 
Implement a way to disable recompaction

In some cases it is a beneficial to be able to temporary disable
recompaction. This commit introduces a way to do so globally or
individually per database or specific index.

The configuration of the feature is as follows:

[view_compaction.recompaction]
enable = true / false %% to set globaly
<db_name> = enable / disable %% to set per db
<db_name>:<idx_name> = enable / disable %% to set per index

2 years agoExtract commit_compacted function
ILYA Khlopotov [Fri, 13 May 2016 15:22:55 +0000 (08:22 -0700)] 
Extract commit_compacted function

2 years agoAdd travis config 12/head
Eric Avdey [Mon, 11 Jan 2016 20:27:37 +0000 (16:27 -0400)] 
Add travis config

2 years agoAdding couch_index_plugin:before_open/2 EPI hook
ILYA Khlopotov [Thu, 17 Dec 2015 17:56:24 +0000 (09:56 -0800)] 
Adding couch_index_plugin:before_open/2 EPI hook

This would allow to extend existent indexers with additional
functionality such as for example:

- index file validation
- index file moves
- injection of options into #mrview.options
- any other tasks which could be done on `open`

3 years agoAdd .gitignore file 13/head
Eric Avdey [Mon, 11 Jan 2016 20:39:13 +0000 (16:39 -0400)] 
Add .gitignore file

3 years agoPlug couch_epi using supervisor 10/head
ILYA Khlopotov [Tue, 29 Sep 2015 20:18:21 +0000 (13:18 -0700)] 
Plug couch_epi using supervisor

3 years agoUpdate to new couch_epi API
ILYA Khlopotov [Mon, 28 Sep 2015 16:51:35 +0000 (09:51 -0700)] 
Update to new couch_epi API

3 years agoThrow validation error in all cases 9/head
Robert Newson [Mon, 7 Sep 2015 12:07:57 +0000 (13:07 +0100)] 
Throw validation error in all cases

This clarifies the validation contract; we always throw if there's a
problem with the design document.

COUCHDB-2772

3 years agoRegister service 8/head
ILYA Khlopotov [Wed, 2 Sep 2015 16:51:43 +0000 (09:51 -0700)] 
Register service

COUCHDB-2796

3 years agoRemove `ignore_providers` option
ILYA Khlopotov [Tue, 1 Sep 2015 13:29:52 +0000 (06:29 -0700)] 
Remove `ignore_providers` option

COUCHDB-2796

3 years agoAdd LICENSE file
Alexander Shorin [Sun, 23 Aug 2015 10:24:37 +0000 (13:24 +0300)] 
Add LICENSE file

3 years agoSupport transformer functions as args 7/head
ILYA Khlopotov [Wed, 5 Aug 2015 21:56:42 +0000 (14:56 -0700)] 
Support transformer functions as args

3 years agoAdd couch_index_plugin:index_update/4
ILYA Khlopotov [Fri, 19 Jun 2015 19:01:23 +0000 (12:01 -0700)] 
Add couch_index_plugin:index_update/4

3 years agoAllow couch_index to open restricted design docs
Mike Wallace [Tue, 30 Jun 2015 20:52:58 +0000 (21:52 +0100)] 
Allow couch_index to open restricted design docs

Design docs in the authentication DB cannot be opened without an
admin context so this commit teaches couch_index to always use
?ADMIN_CTX when opening documents.

COUCHDB-2738

This closes #6

3 years agoAllow get_index to receive DDoc as a doc record 5/head
Eric Avdey [Thu, 25 Jun 2015 21:42:14 +0000 (18:42 -0300)] 
Allow get_index to receive DDoc as a doc record

The function `couch_mrview:compact/3` is passing to `get_index`
a prepared #doc record, when `get_index` is expecting DDoc
to be a ddoc's id. As a result view's compaction is crashing
after `fabric:open_doc` call.

This patch allows get_index to receive #doc record directly.

This closes COUCH-2729

3 years agoMerge remote-tracking branch 'iilyak/2526-pending-updates-info'
Alexander Shorin [Tue, 12 May 2015 19:55:28 +0000 (22:55 +0300)] 
Merge remote-tracking branch 'iilyak/2526-pending-updates-info'

This closes #3

COUCHDB-2526

3 years agoMerge remote-tracking branch 'iilyak/2561-make-config-API-consistent'
Alexander Shorin [Wed, 4 Feb 2015 15:45:05 +0000 (18:45 +0300)] 
Merge remote-tracking branch 'iilyak/2561-make-config-API-consistent'

This closes #4

COUCHDB-2561

3 years agoAdd vsn to modules implementing config_listener 4/head
ILYA Khlopotov [Fri, 30 Jan 2015 20:04:27 +0000 (12:04 -0800)] 
Add vsn to modules implementing config_listener

COUCHDB-2561

3 years agoDon't restart event handler on termination
ILYA Khlopotov [Fri, 30 Jan 2015 19:17:15 +0000 (11:17 -0800)] 
Don't restart event handler on termination

COUCHDB-2561

3 years agoUpdate config_listener behaviuor
ILYA Khlopotov [Thu, 29 Jan 2015 21:40:27 +0000 (13:40 -0800)] 
Update config_listener behaviuor

COUCHDB-2561

4 years agoReturn pending_updates field from couch_index:get_info/1 3/head
ILYA Khlopotov [Mon, 22 Dec 2014 22:04:44 +0000 (14:04 -0800)] 
Return pending_updates field from couch_index:get_info/1

COUCHDB-2526

4 years agoUse fabric for shard #db{} records
Robert Newson [Sat, 22 Nov 2014 22:06:07 +0000 (22:06 +0000)] 
Use fabric for shard #db{} records

Fixes _info calls on clustered views.

closes COUCHDB-2461

4 years agoUse couch_log instead of ?LOG_*
Alexander Shorin [Wed, 24 Sep 2014 11:42:32 +0000 (15:42 +0400)] 
Use couch_log instead of ?LOG_*

4 years agoUse ejson_body when opening design documents windsor-merge
Paul J. Davis [Fri, 22 Aug 2014 10:04:49 +0000 (05:04 -0500)] 
Use ejson_body when opening design documents

4 years agoPeriodically check if an index should close
Paul J. Davis [Sun, 17 Aug 2014 20:12:02 +0000 (15:12 -0500)] 
Periodically check if an index should close

We need to periodically check if our index file still exists on disk
because index cleanups don't notify the couch_index process when a file
has been deleted. If we don't check for this condition then the index
can remain open indefinitely wasting disk space.

We make sure that we're idle before closing by looking to see if we
have any clients waiting for an update.

4 years agoAdd an index committed event
Paul J. Davis [Sun, 17 Aug 2014 19:39:37 +0000 (14:39 -0500)] 
Add an index committed event

4 years ago[WIP] Update couch_index_server to use couch_event
Paul J. Davis [Thu, 7 Aug 2014 18:30:41 +0000 (13:30 -0500)] 
[WIP] Update couch_index_server to use couch_event

4 years agoUpdate couch_view to use new couch_event app
Robert Newson [Mon, 4 Aug 2014 14:05:17 +0000 (15:05 +0100)] 
Update couch_view to use new couch_event app

4 years agoMerge remote-tracking branch 'origin/import-master'
Robert Newson [Wed, 14 May 2014 15:06:19 +0000 (16:06 +0100)] 
Merge remote-tracking branch 'origin/import-master'

Conflicts:
Makefile.am
src/couch_index.erl
src/couch_index_server.erl
src/couch_index_util.erl

4 years agoChange API to function per level 1843-feature-bigcouch
Robert Newson [Wed, 12 Feb 2014 23:24:08 +0000 (23:24 +0000)] 
Change API to function per level

4 years agoSwitch to couch_log
Robert Newson [Wed, 12 Feb 2014 20:12:19 +0000 (20:12 +0000)] 
Switch to couch_log

4 years agoRemove old notes module
Paul J. Davis [Tue, 4 Feb 2014 23:31:52 +0000 (17:31 -0600)] 
Remove old notes module

This module was never intended to be compiled. Including it just
confuses rebar so we'll remove it.

5 years agoBuild with rebar
Robert Newson [Wed, 18 Dec 2013 14:04:59 +0000 (14:04 +0000)] 
Build with rebar

5 years agoExpose get_compactor_pid/1 import-master
Robert Newson [Tue, 20 Aug 2013 11:32:29 +0000 (12:32 +0100)] 
Expose get_compactor_pid/1

5 years agoReject design docs with compilation errors
Paul J. Davis [Sun, 10 Mar 2013 21:03:00 +0000 (16:03 -0500)] 
Reject design docs with compilation errors

If we know the language for a given design doc we'll attempt to compile
the map/reduce functions and reject updates that introduce code that
doesn't compile. We don't yet do the other types of functions because
those errors are more user visible.

5 years agoMajor change to use the new config app
Robert Newson [Sun, 10 Mar 2013 00:22:19 +0000 (18:22 -0600)] 
Major change to use the new config app

This replaces couch_config with the config app. The config app is mostly
a direct port of couch_config to a new namespace with the addition of a
rewritten config change notification mechanism. The new change mechanism
removes the ability to register an anonymous function that breaks code
upgrades and generally cleans up the various listening patterns using a
new behavior definition.

5 years agoFix views for node-local databases
Paul J. Davis [Sat, 1 Sep 2012 01:48:16 +0000 (20:48 -0500)] 
Fix views for node-local databases

Simple fix matches on `<<"shards/", _/binary>>` to detect if it should
use a Fabric call or open a local database.

BugzId: 14565

5 years agoIgnore unexpected messages to couch_index_server
Paul J. Davis [Sat, 9 Mar 2013 01:54:27 +0000 (19:54 -0600)] 
Ignore unexpected messages to couch_index_server

5 years agoGC liberally throughout view indexing code path
Adam Kocoloski [Fri, 28 Oct 2011 19:53:07 +0000 (15:53 -0400)] 
GC liberally throughout view indexing code path

5 years agoMake couch_view_updater amenable to hot upgrades
Adam Kocoloski [Mon, 9 Aug 2010 13:53:16 +0000 (09:53 -0400)] 
Make couch_view_updater amenable to hot upgrades

5 years agoAPI for triggering view updates
Paul J. Davis [Fri, 8 Mar 2013 22:24:04 +0000 (16:24 -0600)] 
API for triggering view updates

5 years agoSquash to #full_doc_info{} in by_seq tree
Paul J. Davis [Mon, 11 Mar 2013 22:50:13 +0000 (17:50 -0500)] 
Squash to #full_doc_info{} in by_seq tree

5 years agoStore #full_doc_info{} records in the seq tree
Paul J. Davis [Thu, 7 Mar 2013 22:03:31 +0000 (16:03 -0600)] 
Store #full_doc_info{} records in the seq tree

This improves the performance of compaction and replication by not
requiring a btree lookup for every row in the by_seq tree at the cost of
storing extra bytes on disk.

5 years agoRefactor couch source tree
Paul J. Davis [Tue, 5 Mar 2013 19:51:39 +0000 (13:51 -0600)] 
Refactor couch source tree

5 years agofixed build
Noah Slater [Thu, 21 Feb 2013 20:59:46 +0000 (20:59 +0000)] 
fixed build

5 years agoCOUCHDB-1667: Revert index_dir to view_index_dir config option.
Jan Lehnardt [Wed, 6 Feb 2013 20:45:48 +0000 (21:45 +0100)] 
COUCHDB-1667: Revert index_dir to view_index_dir config option.

Patch by Alexander Shorin.

6 years agoClose view files cleanly on DB create/delete
Adam Kocoloski [Sun, 2 Dec 2012 14:04:48 +0000 (09:04 -0500)] 
Close view files cleanly on DB create/delete

The shutdown_sync was actually interfering with a clean shutdown
initiated by the index processes themselves when they received a 'DOWN'
message from their monitoring of the DB process.  The explicit 'delete'
cast message in this patch is very probably redundant, but it makes the
intent clear.  The important bit is to let the index process terminate
*and* get confirmation of the termination before attempting to nuke the
directory.

COUCHDB-1364

6 years agoCOUCHDB-1444 Fix missing_named_view
Paul Joseph Davis [Wed, 15 Aug 2012 19:50:18 +0000 (14:50 -0500)] 
COUCHDB-1444 Fix missing_named_view

Instead of replacing 'EXIT' signals we can just stuff the process that
exited into the exit reason and match on that.

6 years agoRevert "COUCHDB-1444 Fix missing_named_view errors"
Paul Joseph Davis [Wed, 15 Aug 2012 19:53:50 +0000 (14:53 -0500)] 
Revert "COUCHDB-1444 Fix missing_named_view errors"

This reverts commit f5341a7dc65e96b806a958a27c9ccee5ec431162.

6 years agoCOUCHDB-1444 Fix missing_named_view errors
Paul Joseph Davis [Fri, 10 Aug 2012 15:45:03 +0000 (10:45 -0500)] 
COUCHDB-1444 Fix missing_named_view errors

Relying on exit signals we were accidentally replacing view state
incorrectly when an exit signal accidentally spidered between various
processes related to couch_os_processes. This changes the state updates
to be a '$gen_cast' instead.

6 years agoAssert that index sig never changes in the lifetime of a couch index process
Robert Newson [Thu, 9 Aug 2012 15:06:57 +0000 (16:06 +0100)] 
Assert that index sig never changes in the lifetime of a couch index process

COUCHDB-1444 demonstrates that #st{} state get out of sync
somehow, leading to unexpected 404's (missing_named_view) and
200's. This patch ensures that index sig never changes in the hope
that it will lead to the cause of 1444 and prevent other occurrences
of the same class of bug.

6 years agoCOUCHDB-1491 - cleanup view tables
Robert Newson [Mon, 30 Jul 2012 15:54:16 +0000 (16:54 +0100)] 
COUCHDB-1491 - cleanup view tables

A bad pattern in couch_index_server for the EXIT case prevented the
cleanup of the BY_SIG table, causing subsequent view requests to
receive a dead pid.

6 years agoUse the correct utility function to get the index directory path
Jason Smith (air) [Mon, 23 Jan 2012 01:31:11 +0000 (01:31 +0000)] 
Use the correct utility function to get the index directory path

7 years agoFix ets lookup case_clause error after ddoc is updated
Filipe David Borba Manana [Tue, 17 Jan 2012 16:39:31 +0000 (16:39 +0000)] 
Fix ets lookup case_clause error after ddoc is updated

This happens if the ddoc_updated event is received after
a client opens the new view group or if a design document
is updated several times in a row and there are still
clients streaming views from 2 or more view groups that
match old versions of the design document.

This relates to COUCHDB-1309

7 years agoShutdown view group on ddoc update
Filipe David Borba Manana [Sun, 13 Nov 2011 18:22:42 +0000 (18:22 +0000)] 
Shutdown view group on ddoc update

If a design document is updated (or deleted), its associated
view groups that are open will be shutdown as soon as they
finish serving all their requests. Not doing this prevented
a proper cleanup of outdated view files (file descriptor leaks)
and unncessary processes in the system (old view groups and index
file ref counters, for e.g.).

COUCHDB-1309

7 years agoreflect api changes in couch_api_index. While I'm here, even if the
Benoit Chesneau [Tue, 27 Sep 2011 08:25:08 +0000 (08:25 +0000)] 
reflect api changes in couch_api_index. While I'm here, even if the
module isn't used give it a proper module declaration.

git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1176269 13f79535-47bb-0310-9956-ffa450edef68

7 years agofix typos
Benoit Chesneau [Tue, 27 Sep 2011 08:18:22 +0000 (08:18 +0000)] 
fix typos

git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1176267 13f79535-47bb-0310-9956-ffa450edef68

7 years agoCommit the compacted index before returning it.
Paul Joseph Davis [Fri, 23 Sep 2011 19:56:14 +0000 (19:56 +0000)] 
Commit the compacted index before returning it.

Failing to compact here could cause errors trying to resume a compaction
due to timeouts while searching for a header.

See COUCHDB-994

git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1174984 13f79535-47bb-0310-9956-ffa450edef68

7 years agoRemember to provide a 2-arity wrapper to udpate.
Paul Joseph Davis [Wed, 21 Sep 2011 00:33:36 +0000 (00:33 +0000)] 
Remember to provide a 2-arity wrapper to udpate.

Expose the proper API to allow the compactor to use the updater to
top off a compaction when the view was udpated during compaction.

git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1173439 13f79535-47bb-0310-9956-ffa450edef68

7 years agoKeep the databae open for index compaction.
Paul Joseph Davis [Wed, 21 Sep 2011 00:14:25 +0000 (00:14 +0000)] 
Keep the databae open for index compaction.

Long running index compactions could fail to complete if something was
iterating db's due to the underlying db being closed. Seeing as index
compaction is non-resumable this prevented index compaction from
completing.

Fixes COUCHDB-1283

git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1173435 13f79535-47bb-0310-9956-ffa450edef68

7 years agoAvoid a race condition for monitored compaction.
Paul Joseph Davis [Sun, 18 Sep 2011 21:52:09 +0000 (21:52 +0000)] 
Avoid a race condition for monitored compaction.

It was technically possible for compaction to finish before a monitor
was requested. This fixes that by giving couch_index:compact a third
options parameter that accepts the atom 'monitor' which will then return
{ok, MonRef} instead of a bare ok.

git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1172378 13f79535-47bb-0310-9956-ffa450edef68

7 years agoOnly restart the updater if it was running.
Paul Joseph Davis [Sun, 18 Sep 2011 21:51:35 +0000 (21:51 +0000)] 
Only restart the updater if it was running.

When compaction finished it was forcibly starting a new indexer process
even if one wasn't already running. This reverts to the old behavior.

git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1172377 13f79535-47bb-0310-9956-ffa450edef68

7 years agoBetter lifetime error messages for couch_index.
Paul Joseph Davis [Sun, 18 Sep 2011 21:50:50 +0000 (21:50 +0000)] 
Better lifetime error messages for couch_index.

Add a log message when the indexer exits as well as format the sig in
log messages has hex.

git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1172375 13f79535-47bb-0310-9956-ffa450edef68

7 years agoUse #doc_info{} optimization for _all_docs.
Paul Joseph Davis [Sun, 18 Sep 2011 17:57:16 +0000 (17:57 +0000)] 
Use #doc_info{} optimization for _all_docs.

During the refactor I accidentally undid the optimization from
COUCHDB-1061 which uses a #doc_info{} to get a #doc{} from disk
which avoids a lookup in the id btree.

git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1172309 13f79535-47bb-0310-9956-ffa450edef68

7 years agoImproved _active_tasks API
Filipe David Borba Manana [Sat, 17 Sep 2011 03:31:12 +0000 (03:31 +0000)] 
Improved _active_tasks API

Tasks are now free to set any properties they wish (as an
Erlang proplist). Different tasks can have different properties
and the status string doesn't exist anymore - instead client
applications can build it using more granular properties from
_active_tasks. Some of these properties are:

1) "progress" (an integer percentage, for all tasks)
2) "database" (for compactions and indexer tasks)
3) "design_document" (for indexer and view compaction tasks)
4) "source" and "target" (for replications)
5) "docs_read", "docs_written", "doc_write_failures",
   "missing_revs_found", "missing_revs_checked", "source_seq",
   "checkpointed_source_seq" and "continuous" for replications

git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1171888 13f79535-47bb-0310-9956-ffa450edef68

7 years agoAdd logging for indexer events.
Paul Joseph Davis [Fri, 16 Sep 2011 17:09:34 +0000 (17:09 +0000)] 
Add logging for indexer events.

This includes info messages for update start/finish, compaction
start/finish and index close on db exit. There are also debug
messages that display the update_seq during updates.

git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1171675 13f79535-47bb-0310-9956-ffa450edef68

7 years agoPaul is an idiot commit 1 of 3.
Paul Joseph Davis [Thu, 15 Sep 2011 17:22:30 +0000 (17:22 +0000)] 
Paul is an idiot commit 1 of 3.

Forgot I was working with a retarded VCS. These are the file changes
for the previous three commits.

git-svn-id: https://svn.apache.org/repos/asf/couchdb/trunk@1171180 13f79535-47bb-0310-9956-ffa450edef68