couchdb.git
14 hours agoOptimize couch_util:to_hex/1 main
Nick Vatamaniuc [Wed, 6 Jul 2022 06:49:15 +0000 (02:49 -0400)] 
Optimize couch_util:to_hex/1

When profiling [1] a cluster acting as a replication source, noticed a lot of
time spent in `couch_util:to_hex/1`. That function is used to emit revision ids
amongst other things. When processing a million documents with a 1000 revisions
each, it ends up in the hotpath, so to speak.

Remembering that in Erlang/OTP 24 there is a new `binary:encode_hex/1` [2]
function, decided to benchmark ours vs the OTP implementation. It turns ours is
slower [3] so let's try to use the OTP one.

One difference from the OTP's version is ours emits lower case hex letters,
while the OTP one emits upper case ones. That's why the lookup table had all
the "A"s replaced with "a"s, "B"s with "b"s, etc.

As a bonus, replaced a few calls to `couch_util:to_hex/1` wrapped in `?l2b/1`
or `list_to_binary/1` with just a single call to `couch_util:to_hex_bin/1`.

Existing `couch_util:to_hex/1` version, returning a list ,was left as is and
just calls `to_hex_bin/1` internally and converts the result to a list.

This altered the peak memory usage for the key tree stemming test so had to
alter the magic constants there a bit to avoid flakiness.

[1]
```
> ... eprof:analyze(total,[{sort, time}, {filter, [{time, 1000}, {calls, 100}]}]).

FUNCTION                                                            CALLS        %      TIME  [uS / CALLS]
--------                                                            -----  -------      ----  [----------]
...
couch_doc:revid_to_str/1                                          1165641     1.71    402860  [      0.35]
couch_key_tree:get_key_leafs_simple/4                             1304102     1.85    435700  [      0.33]
erlang:list_to_integer/2                                           873172     2.00    471140  [      0.54]
gen_server:loop/7                                                   62209     2.11    496932  [      7.99]
couch_key_tree:map_simple/3                                       1829235     2.36    554429  [      0.30]
couch_util:nibble_to_hex/1                                       37334650    16.67   3917127  [      0.10]
couch_util:to_hex/1                                              19834192    34.37   8077050  [      0.41]
---------------------------------------------------------------  --------  -------  --------  [----------]
Total:                                                           91072005  100.00%  23503072  [      0.26]
```

[2] https://www.erlang.org/doc/man/binary.html#encode_hex-1

[3]

```
% ~/src/erlperf/erlperf 'hex:to_hex1(<<210,90,95,232,68,185,66,248,160,33,184,103,181,221,158,96>>).' 'hex:to_hex3(<<210,90,95,232,68,185,66,248,160,33,184,103,181,221,158,96>>).'
Code                                                                                ||        QPS       Time     Rel
hex:to_hex3(<<210,90,95,232,68,185,66,248,160,33,184,103,181,221,158,96>>).          1    2746 Ki     364 ns    100%
hex:to_hex1(<<210,90,95,232,68,185,66,248,160,33,184,103,181,221,158,96>>).          1    1593 Ki     627 ns     58%
```

(`to_hex1/1` is the existing version and `to_hex3/1` is the OTP version).

36 hours agoAdd an option to let custodian always use [cluster] n value
Nick Vatamaniuc [Fri, 1 Jul 2022 21:00:05 +0000 (17:00 -0400)] 
Add an option to let custodian always use [cluster] n value

In cases when the application prevents creating dbs with non-default n values
(n=1, n=2) setting `[custodian] use_cluster_n_as_expected_n = true` can help
detect cases when the shard map itself was manpulated and by mistake ended up
with less than `[cluster] n` copies for some ranges.

By default the behavior is unchanged and n value will be deduced from the
number of copies indicated in the shard map documents.

2 days agoAdd Erlang 25 to PR CI pipeline and Ubuntu Jammy to full CI
Nick Vatamaniuc [Sat, 2 Jul 2022 22:25:45 +0000 (18:25 -0400)] 
Add Erlang 25 to PR CI pipeline and Ubuntu Jammy to full CI

5 days agoBring back POWER full builds
Nick Vatamaniuc [Mon, 27 Jun 2022 17:07:41 +0000 (13:07 -0400)] 
Bring back POWER full builds

5 days agoUpdate vm.args for Erlang 23+
Nick Vatamaniuc [Thu, 30 Jun 2022 19:41:33 +0000 (15:41 -0400)] 
Update vm.args for Erlang 23+

 * file IO is using dirty IO schedulers and not the async IO thread pool

 * `-smp enable` does nothing, it's always enalbed

 * kernel polling is always enabled

```
% erl +SDio 16
Erlang/OTP 23 [erts-11.2.2.13] [source] [64-bit] [smp:12:12] [ds:12:12:16] [async-threads:1]

> erlang:system_info(kernel_poll).
true

> erlang:system_info(dirty_io_schedulers).
16
``

6 days agoFix Elixir 13 compatibility
Nick Vatamaniuc [Mon, 27 Jun 2022 21:25:46 +0000 (17:25 -0400)] 
Fix Elixir 13 compatibility

Elixir 13 is the only one which seems to work with Erlang 25.

But we can't use Elixir 13 because of a few compatibility issues so have to fix
this. That involves upgrading credo. However, as soon as credo was upgraded it
noticed a bunch of issues which also had to be fixed. Some of those were:

 * Enum.map + join should be Enum.map_join
 * Invalid module name in teardown `%___MODULE__` -> `%__MODULE__`
 * `Mix.Config` is deprecated, use `Config`
 * `db_test.ex` looks like a test module to credo, so renamed it, otherwise it has to have an .exs extension
 * ibrowse had to be compiled so it could be properly analysed

Tested on macos (intel) with asdf installed elixir and erlang:
```
  25.0.1    / 1.13.4-otp-25
  24.3.4.1  / 1.13.4-otp-24
  23.3.4.14 / 1.10.4-otp-23
```

12 days agoMerge pull request #4080 from apache/change-irc-url
Ronny [Fri, 24 Jun 2022 12:49:58 +0000 (14:49 +0200)] 
Merge pull request #4080 from apache/change-irc-url

Small url fixes

12 days agoSmall url fixes change-irc-url 4080/head
Ronny Berndt [Fri, 24 Jun 2022 11:04:08 +0000 (13:04 +0200)] 
Small url fixes

Fix IRC url
Renaming master -> main for contributing template

13 days agoMerge pull request #4079 from apache/backport-from-fdbmain
Ronny [Thu, 23 Jun 2022 14:20:05 +0000 (16:20 +0200)] 
Merge pull request #4079 from apache/backport-from-fdbmain

Backport commits from fdbmain into main (old 3.x)

13 days agoBackport commits from fdbmain into main (old 3.x) 4079/head
Ronny Berndt [Thu, 23 Jun 2022 12:16:54 +0000 (14:16 +0200)] 
Backport commits from fdbmain into main (old 3.x)

Cherry-picked commits from 0156a55012b76adb652c11032596d9801c71665e
Thx @kianmeng

13 days agoUse https as default protocol for links
Ronny [Wed, 9 Feb 2022 18:59:32 +0000 (19:59 +0100)] 
Use https as default protocol for links

13 days agoAdding commit message conventions and update links
Ronny [Wed, 9 Feb 2022 12:25:03 +0000 (13:25 +0100)] 
Adding commit message conventions and update links

Added the commit message conventions from the proposal of discussion #3918 and updated all links to use https and moved all external links to the end of the file

13 days ago- rename master to main
Ronny [Mon, 31 Jan 2022 08:39:16 +0000 (09:39 +0100)] 
- rename master to main
- fixing links

2 weeks agoPrevent error:function_clause in check_security/3 if roles claim is malformed (#4070)
Ronny [Wed, 22 Jun 2022 13:32:55 +0000 (15:32 +0200)] 
Prevent error:function_clause in check_security/3 if roles claim is malformed (#4070)

If the given UserRoles/Roles/Names aren't lists, ordersets:from_list/1
or lists:member will fail with an error. Prevent this with Erlang
Pattern Matching and the Robot Butt Rule [1].
Thanks @nickva

[1] https://medium.com/erlang-battleground/ode-to-the-robot-butt-bbd69e69beb2

2 weeks agoFix couch_debug:opened_files* functions
Nick Vatamaniuc [Mon, 20 Jun 2022 21:45:51 +0000 (17:45 -0400)] 
Fix couch_debug:opened_files* functions

They relied on scanning async IO thread pool ports (`efile`) before . That only
worked in Erlang versions < 21. Since then file IO was moved to dirty
schedulers so listing efile ports doesn't yield any useful results.

The partial fix is iterate through `processes()` and pick out the ones for
which couch_file has put anything into the process dictionary. This will skip
non-couch-file files but it's probably still better than nothing.

2 weeks agoBypass macos builder for now while we sort the rebar issue
Nick Vatamaniuc [Mon, 20 Jun 2022 14:32:23 +0000 (10:32 -0400)] 
Bypass macos builder for now while we sort the rebar issue

2 weeks agoFix full CI build
Nick Vatamaniuc [Sun, 19 Jun 2022 03:04:17 +0000 (23:04 -0400)] 
Fix full CI build

We had to set spidermonkey to 78 in the PR CI builds so do the same
for full builds.

2 weeks agoFix "variable bound multiple times" warning
Jay Doane [Sat, 18 Jun 2022 15:22:28 +0000 (08:22 -0700)] 
Fix "variable bound multiple times" warning

2 weeks agoReplace include with include_lib
Jay Doane [Sat, 18 Jun 2022 15:19:36 +0000 (08:19 -0700)] 
Replace include with include_lib

2 weeks agoDelete unused include_lib
Jay Doane [Sat, 18 Jun 2022 15:18:38 +0000 (08:18 -0700)] 
Delete unused include_lib

2 weeks agoFix deprecated function warning
Jay Doane [Sat, 18 Jun 2022 15:17:45 +0000 (08:17 -0700)] 
Fix deprecated function warning

Function `httpd_util:hexlist_to_integer/1` will be removed in OTP 26.

2 weeks agoErlang 25 compatibility - http_uri:parse -> uri_string:parse
Nick Vatamaniuc [Fri, 17 Jun 2022 20:47:12 +0000 (16:47 -0400)] 
Erlang 25 compatibility - http_uri:parse -> uri_string:parse

This fixes the replicator eunit failure:
https://github.com/apache/couchdb/pull/4060#issuecomment-1154391783

2 weeks agoLowest CI image version Erlang 23 doesn't have spidermonkey 1.8.5
Nick Vatamaniuc [Fri, 17 Jun 2022 22:13:54 +0000 (18:13 -0400)] 
Lowest CI image version Erlang 23 doesn't have spidermonkey 1.8.5

2 weeks agoUpdate erlfmt formatting for smoosh_server
Nick Vatamaniuc [Fri, 17 Jun 2022 22:13:24 +0000 (18:13 -0400)] 
Update erlfmt formatting for smoosh_server

Since the ?STACKTRACE macro is gone now we have actual Erlang syntax in there
and erlfmt knows how to interpret it.

2 weeks agoRemove Erlang < 23 ifdefs and other macros
Nick Vatamaniuc [Fri, 17 Jun 2022 21:42:27 +0000 (17:42 -0400)] 
Remove Erlang < 23 ifdefs and other macros

2 weeks agoSet low erlang version in Jenkins.pr to 23 instead of 20
Nick Vatamaniuc [Fri, 17 Jun 2022 21:41:12 +0000 (17:41 -0400)] 
Set low erlang version in Jenkins.pr to 23 instead of 20

2 weeks agoSkip Erlang version != 20 in configure
Nick Vatamaniuc [Fri, 17 Jun 2022 21:40:12 +0000 (17:40 -0400)] 
Skip Erlang version != 20 in configure

They are all != 20 now

2 weeks agoUpdate rebar config to remove Erlang < 23 support
Nick Vatamaniuc [Fri, 17 Jun 2022 21:23:35 +0000 (17:23 -0400)] 
Update rebar config to remove Erlang < 23 support

2 weeks agoUpdate CI to remove Erlang < 23
Nick Vatamaniuc [Fri, 17 Jun 2022 21:20:50 +0000 (17:20 -0400)] 
Update CI to remove Erlang < 23

2 weeks agoIgnore repeated interactive purges
Nick Vatamaniuc [Tue, 14 Jun 2022 19:47:44 +0000 (15:47 -0400)] 
Ignore repeated interactive purges

After enabling default purge request replication, it was possible to get a
`badreq` error because the internal replicator would race with the interactive
worker updates. Since we're dealing with cluster-generated UUIDs if we see the
same UUID, DocId, and Revs again, assume it was applied already and ignore it.

For belt-and-suspenders protection ensure that revs lists don't have duplicates
and are sorted. Do it both in fabric and couch_db in case an operator wants to
do local purges. For extra safety also keep sorting the revs in the duplicate
check for the rare case that the existing purge btree already contain duplicate
revs. Eventually that check could be removed as all the new purge revs lists
would be de-duplicated and sorted.

As a bonus, if it turns out all the purges were applied simply return and avoid
calling the updater process which can be an expensive bottleneck.

2 weeks agoEnable replicating purge requests between nodes
Nick Vatamaniuc [Mon, 13 Jun 2022 21:21:27 +0000 (17:21 -0400)] 
Enable replicating purge requests between nodes

It seems we had forgotten to enable it and in the case a node if offline when
a clustered purge request is issued, and then re-join they won't know to
process any purge requests issued to other nodes.

3 weeks agoMerge pull request #4060 from apache/big-r81-add-erlang-25-support
Ronny [Mon, 13 Jun 2022 05:25:45 +0000 (07:25 +0200)] 
Merge pull request #4060 from apache/big-r81-add-erlang-25-support

Add Erlang 25 support

3 weeks agoAdd Erlang 25 support 4060/head
Ronny Berndt [Sat, 11 Jun 2022 12:18:54 +0000 (14:18 +0200)] 
Add Erlang 25 support

Before merging this PR, PR #16 in couchdb-snappy needs to get merged and tagged with CouchDB-1.0.8

3 weeks agoMerge pull request #4063 from apache/big-r81-patch-1
Ronny [Sun, 12 Jun 2022 16:01:53 +0000 (18:01 +0200)] 
Merge pull request #4063 from apache/big-r81-patch-1

Display main build status in README

3 weeks agoDisplay main build status in README 4063/head
Ronny [Sun, 12 Jun 2022 12:39:34 +0000 (14:39 +0200)] 
Display main build status in README

After changing branch 3.x -> main, show status in README

3 weeks agoReduce complexity of "possible_ancestors" from quadratic to linear.
Nick Vatamaniuc [Fri, 10 Jun 2022 20:33:04 +0000 (16:33 -0400)] 
Reduce complexity of "possible_ancestors" from quadratic to linear.

"Possible ancestors" are all the leaf revisions with a position less than any
of the missing revisions. Previously, for every leaf revision we potentially
checked every missing revision. And with missing revisions list being sorted
from smallest to largest, that meant often traversing most of the missing
revisions list. In general, it meant performing O(R * M) operations, where R is
the number of leaf revision, and M is the number of missing revisions.

The optimization is instead of comparing against every single missing revision
we can compare only against the highest one. If a leaf revision is less than
highest missing revison, then it is already a possible ancestor, so we can stop
checking. Thus, we can go from a quadratic O(R * M) to an O(R) + O(M)
complexity, which, when merging large conflicting documents could give us a
nice performance boost.

3 weeks agoSkip PPC64 CI builds
Nick Vatamaniuc [Sat, 11 Jun 2022 14:58:21 +0000 (10:58 -0400)] 
Skip PPC64 CI builds

Owner indicated the instance has gone away

3 weeks agoAdd two larger couch_key_tree:find_missing/2 tests
Nick Vatamaniuc [Thu, 9 Jun 2022 22:52:56 +0000 (18:52 -0400)] 
Add two larger couch_key_tree:find_missing/2 tests

These were used to generate the random rev trees used for bechmarking. But they
also assert useful properties for the function so let's add them to test_util.

Since we added a shuffle/1 function to test_util, DRY some extra shuffle/1
functions as we seem to have a bunch of them sprinkled here and there.

Also, since the rev tree generation code was changed, need to adjust the
`excessive_memory_when_stemming` test as well. To do it, reverted the commit
indicated in the test and then adjusted parameters until the test failed. Then
reset the revert and it failed.

3 weeks agoOptimize couch_key_tree:find_missing/2
Nick Vatamaniuc [Wed, 8 Jun 2022 16:25:24 +0000 (12:25 -0400)] 
Optimize couch_key_tree:find_missing/2

Previously we always fully traversed the search keys list, when we partitioned
it into possible and impossible sublists. The optimization is we can stop
traversing early if the search keys are sorted as soon as we hit a key that is
>= than the current position in the tree. To keep things simple we can use the
lists:splitwith/2 function which seems to do what we want [1].

[1] https://github.com/erlang/otp/blob/40922798411c2d23ee8a99456f96d6637c62b762/lib/stdlib/src/lists.erl#L1426-L1435

4 weeks agoAvoid using length/1 guard for >0 or ==0 tests in couch_key_tree
Nick Vatamaniuc [Tue, 7 Jun 2022 21:44:26 +0000 (17:44 -0400)] 
Avoid using length/1 guard for >0 or ==0 tests in couch_key_tree

Issue originally discovered by @iilyak using https://comby.dev/

4 weeks agoRevert protection for main branch after 3.x -> main move
Nick Vatamaniuc [Tue, 7 Jun 2022 20:22:57 +0000 (16:22 -0400)] 
Revert protection for main branch after 3.x -> main move

See dev list discussion:
  https://lists.apache.org/thread/x4lc6vhthj1vkt2xpd0ox5osh959qsc4

Previous PR to disable protection on main so we can replace it:
  https://github.com/apache/couchdb/pull/4053

4 weeks agoOptimize couch_util:reorder_results/2,3
Nick Vatamaniuc [Tue, 7 Jun 2022 14:30:03 +0000 (10:30 -0400)] 
Optimize couch_util:reorder_results/2,3

This function is used in the hot path of _revs_diff and _bulk_docs API calls.
Those could always use a bit more optimization:

  * In `_revs_diff` it's used when fetching all the FDIs to see which docs are
    missing in `couch_btree:lookup/2`.

  * In `_bulk_docs` it's used in the `fabric_doc_update` when finalizing the
    response.

Using erlperf in #4051 noticed an at most 5x speedup from using a map instead
of a dict. Since a map already falls back to a proplist for small sizes, skip
the length guard.

Some erlperf examples from #4051:

500 Keys
```
> f(Keys), f(Res), {Keys, Res} = Gen(500), ok.

> erlperf:run(#{runner => {couch_util, reorder_results2, [Keys, Res, 100, dict]}}).
2407
> erlperf:run(#{runner => {couch_util, reorder_results2, [Keys, Res, 100, map]}}).
11639
```

Using a map without the guard, which is the change in this this PR:

```
> f(Keys), f(Res), {Keys, Res} = Gen(500), ok.
ok

> erlperf:run(#{runner => {couch_util, reorder_results, [Keys, Res]}}).
12395
> erlperf:run(#{runner => {couch_util, reorder_results, [Keys, Res]}}).
12508
```

As a bonus this also cleans up the code a bit, too.

4 weeks agoRemove duplicate _revs_diff implementation
Nick Vatamaniuc [Sun, 5 Jun 2022 04:04:47 +0000 (00:04 -0400)] 
Remove duplicate _revs_diff implementation

The implementation was duplicated because fabric needed to return a response
([]) for existing revs. It doesn't seem worth it to keep a whole separate copy
just to handle the [] case. Instead, make the primary couchdb implementation
always return [] and just filter it out for the local httpd endpoint handlers.

It seems we didn't have any _revs_diff tests, so add a few basic tests along
the way.

Also, it turns out we still support the _missing_revs API. That's an endpoint
that's not used by the replicator since before 1.2 days. At some point it might
be worth removing it, but for now ensure we also test it alongside _revs_diff.

4 weeks agoHandle all_dbs_active in fabric_doc_update
Nick Vatamaniuc [Mon, 6 Jun 2022 19:22:14 +0000 (15:22 -0400)] 
Handle all_dbs_active in fabric_doc_update

Previously fabric_doc_update message handler crashed with a function_clause
error the first time it encountered an all_dbs_active error.

Opt to handle it as a `rexi_EXIT` with a hope at that some workers would still
return a valid result.

5 weeks agoAdd smoosh state diagram to directory and reference it
ncshaw [Wed, 1 Jun 2022 15:26:46 +0000 (10:26 -0500)] 
Add smoosh state diagram to directory and reference it

5 weeks agoFreeBSD workers are offline
Nick Vatamaniuc [Fri, 27 May 2022 22:20:24 +0000 (18:20 -0400)] 
FreeBSD workers are offline

Comment them out for the time being

6 weeks agoHandle exit(shutdown) error in chttpd
Nick Vatamaniuc [Thu, 26 May 2022 03:23:49 +0000 (23:23 -0400)] 
Handle exit(shutdown) error in chttpd

We handle mochiweb's `{shutdown, Error}` exit. However, chttpd itself exits
with a plain `exit(shutdown)`, and in that case, our nested catch statements
[1] will emit an "unknown_error" log line. So, make sure to handle that as well
in order to keep the log noise down.

[1] `handle_req_after_auth/2` is nested inside `process_request/1`
https://github.com/apache/couchdb/blob/3.x/src/chttpd/src/chttpd.erl#L386 and
both call `catch_error/4`

6 weeks agoMerge pull request #4037 from apache/jwtf-es256-fail
Robert Newson [Wed, 25 May 2022 12:24:46 +0000 (13:24 +0100)] 
Merge pull request #4037 from apache/jwtf-es256-fail

6 weeks agoFix ES{256,384,512} support 4037/head
Robert Newson [Tue, 24 May 2022 15:16:57 +0000 (16:16 +0100)] 
Fix ES{256,384,512} support

We forgot to transform the JOSE signature format to DER when verifying
ES signatures (and the reverse when signing).

Co-authored-by: Nick Vatamaniuc <vatamane@apache.org>
Co-authored-by: Jay Doane <jaydoane@apache.org>
6 weeks agoRemove all of the experimental "schemas" functionality from smoosh
Nick Vatamaniuc [Mon, 23 May 2022 18:55:44 +0000 (14:55 -0400)] 
Remove all of the experimental "schemas" functionality from smoosh

6 weeks agoRemove unused include in fabric.erl
Nick Vatamaniuc [Mon, 23 May 2022 18:53:36 +0000 (14:53 -0400)] 
Remove unused include in fabric.erl

As reported by erlang LSP server

6 weeks agoCleanup couch_file exports and unused code
Nick Vatamaniuc [Thu, 19 May 2022 22:08:03 +0000 (18:08 -0400)] 
Cleanup couch_file exports and unused code

Removed:
 * append_binary_md5/2
 * assemble_file_chunk/1
 * append_term_md5/2
 * append_term_md5/3
 * pread_binaries/2
 * pread_iolists/2
 * append_binaries/2

6 weeks agoRemove dead code from smoosh
Nick Vatamaniuc [Thu, 19 May 2022 22:02:53 +0000 (18:02 -0400)] 
Remove dead code from smoosh

6 weeks agoRemoved unused include files
Nick Vatamaniuc [Thu, 19 May 2022 21:53:11 +0000 (17:53 -0400)] 
Removed unused include files

Emacs + erlang_ls noticed those. There may be more, but didn't know how
systematically check all of the files.

6 weeks agoMerge pull request #3985 from noahshaw11/implement-resource_hoggers
iilyak [Thu, 19 May 2022 21:21:20 +0000 (14:21 -0700)] 
Merge pull request #3985 from noahshaw11/implement-resource_hoggers

Implement resource_hoggers

6 weeks agoImplement resource hoggers 3985/head
ncshaw [Mon, 4 Apr 2022 20:58:11 +0000 (15:58 -0500)] 
Implement resource hoggers

6 weeks agoDefault rexi use_kill_all to true
Nick Vatamaniuc [Thu, 19 May 2022 16:26:39 +0000 (12:26 -0400)] 
Default rexi use_kill_all to true

It should save some dist bandwidth when workers are canceled at the end of
fabric requests. The feature has been available since 3.0.x (3 years ago) so
opt to enable it by default.

Users who do a rolling upgrade from 2.x could still set it to `false` and then,
after the upgrade completes, delete it to return it to default (true).

6 weeks agoMerge pull request #4027 from apache/big-r81-patch-1
Ronny [Thu, 19 May 2022 14:42:24 +0000 (16:42 +0200)] 
Merge pull request #4027 from apache/big-r81-patch-1

Fix typo

6 weeks agoMerge branch '3.x' into big-r81-patch-1 4027/head
Ronny [Thu, 19 May 2022 14:00:28 +0000 (16:00 +0200)] 
Merge branch '3.x' into big-r81-patch-1

7 weeks agoMake sure to wait for the attachments to uploaded before responding to the user
Nick Vatamaniuc [Thu, 12 May 2022 18:22:29 +0000 (14:22 -0400)] 
Make sure to wait for the attachments to uploaded before responding to the user

If workers drop after when new_edits=false and revisions are already present,
attachment parser will go into `mp_abort_parse_atts` state but still keep
consuming the request data, draining the uploaded attachment data, which helps
ensure the connection process will be in a consistent state and ready to
process the next HTTP request.

Add a test to PUT the same attachment multiple times with new_edits=false and
check that those requests all succeed.

7 weeks agoMinor DRY-ing up of logic to handle hello_from_writer MP parser messages
Nick Vatamaniuc [Fri, 29 Apr 2022 21:41:31 +0000 (17:41 -0400)] 
Minor DRY-ing up of logic to handle hello_from_writer MP parser messages

7 weeks agoMake sure to send hello_from_writer only once for each worker
Nick Vatamaniuc [Fri, 29 Apr 2022 21:35:19 +0000 (17:35 -0400)] 
Make sure to send hello_from_writer only once for each worker

Assuming a document may have multiple attachments, or even the possibility of
having multiple documents each with a different parsers, send
`hello_from_writer` once per worker process as soon as possible in fabric_rpc.

For additional safety (belt and suspenders) assert that the closure is called
from the same process which sent hello and started monitoring the parser, and
if mp_parser_died is caught, and the data function is called again, re-throw
the same mp_parser_died error.

7 weeks agoPreserve support for single-node couch_httpd path
Adam Kocoloski [Thu, 24 Feb 2022 15:36:38 +0000 (10:36 -0500)] 
Preserve support for single-node couch_httpd path

The `hello_from_writer` message is only used in fabric, while we still
have unit tests that go through the couch_httpd side of things. We can
afford to be defensive here.

7 weeks agoEnsure the parser always monitors the worker
Adam Kocoloski [Wed, 23 Feb 2022 20:45:57 +0000 (15:45 -0500)] 
Ensure the parser always monitors the worker

This adds an extra `hello_from_writer` message into the handshake
between the process that reads the multipart attachment data from the
socket and the writer processes (potentially on remote nodes) that
persist the data into each shard file. This ensures that even in the
case where a writer does not end up asking for the data (e.g. because
the revision already exists in the tree), the parser will monitor the
writer and therefore know when the writer has exited.

The patch makes some assumptions that the attachment flush function is
executed in the same process as the initial one that is spawned to
handle the fabric_rpc work request. That's true today, but if it
changed in the future it would be a non-obvious breakage to debug.

7 weeks agoAdd an integration test
Adam Kocoloski [Wed, 23 Feb 2022 18:27:14 +0000 (13:27 -0500)] 
Add an integration test

This doesn't actually work as you'd expect. The response for the
new_edits=false request does return quickly, but the request body
is never consumed and so the _next_ request on the wire will hang.

7 weeks agoFix typo
Ronny [Wed, 18 May 2022 11:11:36 +0000 (13:11 +0200)] 
Fix typo

7 weeks agoMerge pull request #3987 from noahshaw11/fix-busy-functions-to-work-with-pids
iilyak [Mon, 16 May 2022 18:50:54 +0000 (11:50 -0700)] 
Merge pull request #3987 from noahshaw11/fix-busy-functions-to-work-with-pids

Fix busy functions to work with pids

7 weeks agoFix busy functions to work with pids 3987/head
ncshaw [Wed, 6 Apr 2022 20:28:14 +0000 (15:28 -0500)] 
Fix busy functions to work with pids

7 weeks agoMerge pull request #3984 from noahshaw11/implement-memory_info-functions
Robert Newson [Fri, 13 May 2022 07:39:00 +0000 (08:39 +0100)] 
Merge pull request #3984 from noahshaw11/implement-memory_info-functions

Implement memory_info functions

7 weeks agoImplement memory_info functions 3984/head
ncshaw [Mon, 4 Apr 2022 20:26:29 +0000 (15:26 -0500)] 
Implement memory_info functions

8 weeks agoReduce smoosh compaction log level to debug
Nick Vatamaniuc [Wed, 11 May 2022 21:07:21 +0000 (17:07 -0400)] 
Reduce smoosh compaction log level to debug

With a few channels around these add up and make local developement not as
ergonomic. Let's turn them to `debug` and users can always toggle them to info
or error if needed.

8 weeks agoFix maybe_handle_error clauses
Nick Vatamaniuc [Wed, 11 May 2022 15:28:49 +0000 (11:28 -0400)] 
Fix maybe_handle_error clauses

`{shutdown, Err}` should come before `{Err Reason}`, otherwise {Err, Reason} will always match.

Also, plugin `handle_error/1` response was forced to match initial `Error`,
which doesn't have to always be the case based on:

https://github.com/apache/couchdb/blob/42f2c1c534ed5c210b45ffcd9a621a31b781b5ae/src/chttpd/src/chttpd_plugin.erl#L39-L41

8 weeks agoBump jiffy to 1.1.1 and b64url to 1.0.3
Nick Vatamaniuc [Tue, 10 May 2022 16:40:44 +0000 (12:40 -0400)] 
Bump jiffy to 1.1.1 and b64url to 1.0.3

jiffy changelog: https://github.com/davisp/jiffy/compare/1.0.9...1.1.1
b64url changelog: https://github.com/apache/couchdb-b64url/compare/1.0.2...1.0.3

8 weeks agoBump mochiweb to 3.0
Nick Vatamaniuc [Tue, 10 May 2022 15:26:46 +0000 (11:26 -0400)] 
Bump mochiweb to 3.0

8 weeks agoMerge pull request #3983 from noahshaw11/implement-print_table/2
iilyak [Mon, 9 May 2022 15:37:58 +0000 (08:37 -0700)] 
Merge pull request #3983 from noahshaw11/implement-print_table/2

Implement print_table/2

8 weeks agoImplement print_table/2 3983/head
ncshaw [Mon, 4 Apr 2022 20:01:41 +0000 (15:01 -0500)] 
Implement print_table/2

2 months agoFix function_clause error for replicated changes with a target VDU
Nick Vatamaniuc [Fri, 6 May 2022 19:59:16 +0000 (15:59 -0400)] 
Fix function_clause error for replicated changes with a target VDU

Previously when handling mismatched responses for replicated changes, where
some response were `noreply` and some were `forbidden`, the `{DocID, Rev}`
tuple was dropped from the response, which resulted in `update_doc_result_to_json`
failing as it expects an `{{DocId, Rev}, Error}` argument instead of just `Error`.

Make sure to keep and return the reply as is in`check_forbidden_msg/1`.

2 months agoBump mochiweb to include the {shutdown, Error} in acceptor recycler
Nick Vatamaniuc [Fri, 6 May 2022 16:41:00 +0000 (12:41 -0400)] 
Bump mochiweb to include the {shutdown, Error} in acceptor recycler

https://github.com/apache/couchdb-mochiweb/commit/077b4f801ba8f853a9649a9c0f055b8e4f33dcca

2 months agoRevert mochiweb update until the next release
Nick Vatamaniuc [Fri, 6 May 2022 15:08:38 +0000 (11:08 -0400)] 
Revert mochiweb update until the next release

Waiting on https://github.com/mochi/mochiweb/pull/242

2 months agoHandle mochiweb client processes crashing with `{shutdown, Error}`
Nick Vatamaniuc [Mon, 2 May 2022 22:29:59 +0000 (18:29 -0400)] 
Handle mochiweb client processes crashing with `{shutdown, Error}`

Previously on error they exited `normal`, and now exit with the `{shutdown, Error}`
reason.

See: https://github.com/mochi/mochiweb/commit/e56a4dce6b360c5c5d037e8de33dd267790092e4

2 months agouse undefined for timed out responses (#3979)
Tony Sun [Tue, 3 May 2022 05:04:24 +0000 (22:04 -0700)] 
use undefined for timed out responses (#3979)

* fix badargs for timed out responses

Under heavy load, fabric_update_doc workers will return timeout via
rexi. Therefore no reponses will be populate the response dictionary.
This leads to badargs when we do dict:fetch for keys that do not exist.
This fix corrects this behavior by ensuring that each document update
must receive a response. If one document does not have a response,
then the entire update returns a 500.

2 months agoImprove index building during shard splitting
Nick Vatamaniuc [Fri, 22 Apr 2022 23:53:13 +0000 (19:53 -0400)] 
Improve index building during shard splitting

Previously we didn't check responses from get_state/2 or await/2 functions when
building indices. If an index updater crashed, and the index never finished
building, the get_state/2 call would simply return an error and the process
would exit normally. Then, the shard splitting job would count that as a
success and continue to make progress.

To fix that, make sure to check the response to all the supported indexing
types and wait until they return an `ok` result.

Additionally, increase the index building resilience to allow for more retries
on failure, and for configurable retries for individual index builders.

2 months agoBump version to 3.2.2
Nick Vatamaniuc [Sat, 2 Apr 2022 05:21:38 +0000 (01:21 -0400)] 
Bump version to 3.2.2

2 months agoFix undefined cookie in weatherreport_node
Nick Vatamaniuc [Wed, 6 Apr 2022 03:29:47 +0000 (23:29 -0400)] 
Fix undefined cookie in weatherreport_node

If the cookie is undefined then we should not set it so it can pick up
the ~/.erlang.cookie if it is there.

2 months agoBind locally by default
Robert Newson [Wed, 6 Apr 2022 03:23:00 +0000 (23:23 -0400)] 
Bind locally by default

2 months agoRemove the default "monster" cookie
Robert Newson [Sat, 2 Apr 2022 05:12:10 +0000 (01:12 -0400)] 
Remove the default "monster" cookie

2 months agoFix index creation with empty ddoc should return 400 (#3990)
Jiahui Li [Tue, 26 Apr 2022 21:10:55 +0000 (16:10 -0500)] 
Fix index creation with empty ddoc should return 400 (#3990)

Creating an index with "ddoc":"" or "name":"" should return a 400 Bad Request.

This fixes: https://github.com/apache/couchdb/issues/1472

2 months agoMerge pull request #4006 from lostnet/nose2
Will Young [Mon, 25 Apr 2022 20:01:11 +0000 (20:01 +0000)] 
Merge pull request #4006 from lostnet/nose2

Remove CI support for Debian 9 (stretch)

2 months agoRemove CI support for Debian 9 (stretch) 4006/head
Will Young [Mon, 25 Apr 2022 15:50:30 +0000 (17:50 +0200)] 
Remove CI support for Debian 9 (stretch)

2 months agoMerge pull request #4004 from lostnet/nose2
Will Young [Sat, 23 Apr 2022 11:13:56 +0000 (11:13 +0000)] 
Merge pull request #4004 from lostnet/nose2

mango_tests: revert hypothesis back for python3.6 compat

2 months agomango_tests: revert hypothesis back to python3.6 compat 4004/head
Will Young [Sat, 23 Apr 2022 08:38:36 +0000 (10:38 +0200)] 
mango_tests: revert hypothesis back to python3.6 compat

2 months agoMerge pull request #3980 from lostnet/nose2
Will Young [Fri, 22 Apr 2022 10:14:37 +0000 (10:14 +0000)] 
Merge pull request #3980 from lostnet/nose2

Window test suite difficulties (nose2, timestamps)

2 months agoMerge branch '3.x' into nose2 3980/head
Will Young [Fri, 22 Apr 2022 07:57:10 +0000 (07:57 +0000)] 
Merge branch '3.x' into nose2

2 months agoMerge pull request #4000 from apache/dreyfus-to-source
Robert Newson [Wed, 20 Apr 2022 16:45:57 +0000 (17:45 +0100)] 
Merge pull request #4000 from apache/dreyfus-to-source

Ensure Object.prototype.toSource is always available

2 months agoEnsure Object.prototype.toSource() is always available 4000/head
Robert Newson [Wed, 20 Apr 2022 12:35:38 +0000 (13:35 +0100)] 
Ensure Object.prototype.toSource() is always available

2 months agoMerge pull request #3994 from apache/fix-is-compacting
Robert Newson [Fri, 15 Apr 2022 07:35:37 +0000 (08:35 +0100)] 
Merge pull request #3994 from apache/fix-is-compacting

Convert DbName to list before cons

2 months agoConvert DbName to list before cons 3994/head
Robert Newson [Fri, 15 Apr 2022 06:50:06 +0000 (07:50 +0100)] 
Convert DbName to list before cons

2 months agoMerge pull request #3993 from apache/jwtf-error-handling
Robert Newson [Thu, 14 Apr 2022 10:50:10 +0000 (11:50 +0100)] 
Merge pull request #3993 from apache/jwtf-error-handling

Cause a 400 Bad Request if decoding JWT token fails

2 months agoCause a 400 Bad Request if decoding JWT token fails 3993/head
Robert Newson [Thu, 14 Apr 2022 09:32:01 +0000 (10:32 +0100)] 
Cause a 400 Bad Request if decoding JWT token fails

2 months agoMerge pull request #3992 from apache/improve_search_detection
Robert Newson [Wed, 13 Apr 2022 18:05:31 +0000 (19:05 +0100)] 
Merge pull request #3992 from apache/improve_search_detection

Search is available if it was ever available since start