2 min agoUpdate couch_replicator_use_checkpoints_tests main
Nick Vatamaniuc [Fri, 19 Aug 2022 01:45:29 +0000 (21:45 -0400)] 
Update couch_replicator_use_checkpoints_tests

Use the clustered version of source and target endpoints.

Also, use common setup and teardown functions. The test was previously quite
clever in trying to save an extra few lines by parameterizing the "use
checkpoing" vs "don't use checkpoints" scenarios with a foreachx and a custom
notifier function. Instead, opt for more clarity and use the usual TDEF_FE
macro and just setup two separate test one which uses checkpoints and one which

Another major win is is using the utility db comparison function instead of a
duplicate copy of it.

2 min agoUpdate couch_replicator_small_max_request_size_target
Nick Vatamaniuc [Fri, 19 Aug 2022 01:41:14 +0000 (21:41 -0400)] 
Update couch_replicator_small_max_request_size_target

Use the clustered version of the source and target endoints and switch to using
common test setup and teardown function functions. Overall it added to quite a
few number of lines saved.

2 min agoUpdate couch_replicator_selector_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:39:17 +0000 (21:39 -0400)] 
Update couch_replicator_selector_tests

Use the clustered versions of endpoints for the test.

Also, use the common setup and teardown helpers and remove the foreachx

2 min agoUpdate couch_replicator_retain_stats_between_job_runs
Nick Vatamaniuc [Fri, 19 Aug 2022 01:37:32 +0000 (21:37 -0400)] 
Update couch_replicator_retain_stats_between_job_runs

Switch the test to use the clustered endpoints.

Use the common test setup and teardown functions as well as the TDEF_FE macros.

2 min agoUpdate couch_replicator_rate_limiter_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:35:50 +0000 (21:35 -0400)] 
Update couch_replicator_rate_limiter_tests

Use the TDEF_FE macro and cleanup ?_test(begin...end) instances.

2 min agoUpdate couch_replicator_proxy_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:34:26 +0000 (21:34 -0400)] 
Update couch_replicator_proxy_tests

Use the TDEF_FE macro and remove the ugly ?_test(begin...end) construct.

2 min agoUpdate couch_replicator_missing_stubs_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:31:46 +0000 (21:31 -0400)] 
Update couch_replicator_missing_stubs_tests

Use common setup and teardown helpers, TDEF_FE macros and remove all the
foreachx nonsense.

2 min agoUpdate couch_replicator_many_leaves_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:28:13 +0000 (21:28 -0400)] 
Update couch_replicator_many_leaves_tests

Use comon setup and teardown function and the TDEF_FE macros.

Also, remove quite a bit of foreachx and remote boilerplate which is not needed
any longer.

Most of the changes however consisted in update all the db operations to use
fabric instead of couch. Luckily, most of those have fabric equivalents, and
fabric calls are even shorter as they don't need open, re-open and close

2 min agoUpdate couch_replicator_large_atts_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:25:37 +0000 (21:25 -0400)] 
Update couch_replicator_large_atts_tests

Use commong setup functions and TDEF_FE macro.

Removing the foreachx and the remote vs local junk really trimmed down the
size. The test content was tiny compared to the clunky EUnit setup logic.

2 min agoUpdate couch_replicator_id_too_long_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:23:47 +0000 (21:23 -0400)] 
Update couch_replicator_id_too_long_tests

Use common setup and teardown helpers along with some local replicate/2 and db_url/2 functions.

Remove foreachx goop and use TDEF_FE for consistency with other tests.

2 min agoUpdate couch_replicator_httpc_pool_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:22:30 +0000 (21:22 -0400)] 
Update couch_replicator_httpc_pool_tests

Use the TDEF_FE macro and remove the ?_test(begin...end) construct.

2 min agoUpdate couch_replicator_filtered_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:20:15 +0000 (21:20 -0400)] 
Update couch_replicator_filtered_tests

Take advantage of the helper setup and teardown functions.

Switching to a simpler TDEF_FE macro instead of foreachx and inorder setup
cruft also saves some lines of code.

2 min agoUpdate couch_replicator_error_reporting_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:17:37 +0000 (21:17 -0400)] 
Update couch_replicator_error_reporting_tests

Use the TDEF_FE macro and clean up ?_test(begin...end) cruft.

2 min agoUpdate couch_replicator_create_target_with_options_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:15:02 +0000 (21:15 -0400)] 
Update couch_replicator_create_target_with_options_tests

Start using the common setup and tear down functions from the test helper.

Also using the test definitions to use the TDEF_FE macro.

Since the setup function already creates a target endpoint database and the
test is also in charge of creating test database, we just remove the target db
before the replication jobs start.

2 min agoUpdate couch_replicator_connection_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:13:44 +0000 (21:13 -0400)] 
Update couch_replicator_connection_tests

The main changes are just using the TDEF_FE macros and removng the
_test(begin...end) silliness.

2 min agoUpdate couch_replicator_compact_tests
Nick Vatamaniuc [Fri, 19 Aug 2022 01:10:15 +0000 (21:10 -0400)] 
Update couch_replicator_compact_tests

Compactor tests are the only tests which continue using the local ports since
they deals with triggering and managing low level compaction processes.

However, it was still possible to improve the tests somewhat by using the
TDEF_FE macros and removing some left-over foreachx cruft.

2 min agoUpdate couch_replicator_attachments_too_large to use fabric
Nick Vatamaniuc [Fri, 19 Aug 2022 01:08:17 +0000 (21:08 -0400)] 
Update couch_replicator_attachments_too_large to use fabric

Switch test module to use the clustered enpdpoints and TDEF_FE test

2 min agoAdd some utility functions to couch_replicator_test_helper
Nick Vatamaniuc [Fri, 19 Aug 2022 01:03:22 +0000 (21:03 -0400)] 
Add some utility functions to couch_replicator_test_helper

In preparation to start using chttpd (fabric) endpoints add some common utility
functions to the replication test helper module.

Since `couch_db:fold_docs/4` doesn't exit for fabric, use the changes feed to
get all the revision leafs. That is used when comparing database endpoints.

It turns our the majority replication tests can use the exact same setup,
teardown and db_url functions so make sure those are also available in the
helper module.

2 days agoUpdate couchdb-mochiweb to v3.1.0
Ronny Berndt [Tue, 16 Aug 2022 07:29:16 +0000 (09:29 +0200)] 
Update couchdb-mochiweb to v3.1.0

3 days agoUpdate couchdb-config to 2.2.0
Ronny Berndt [Tue, 16 Aug 2022 07:25:44 +0000 (09:25 +0200)] 
Update couchdb-config to 2.2.0

9 days agoFix purge request timeouts
Nick Vatamaniuc [Tue, 9 Aug 2022 22:00:42 +0000 (18:00 -0400)] 
Fix purge request timeouts

Don't rely on the default gen_server 5 second timeout.

Use `infinity` as that's also effectively used for doc updates.


11 days agoFix proxyauth_test and remove it from skipping tests (#4129)
Ronny [Mon, 8 Aug 2022 14:56:26 +0000 (16:56 +0200)] 
Fix proxyauth_test and remove it from skipping tests (#4129)

After reverting #4094, bringing this back as a seperate fix.

3 weeks agoMerge pull request #4133 from noahshaw11/add-editors-magic-lines
iilyak [Fri, 29 Jul 2022 10:42:22 +0000 (03:42 -0700)] 
Merge pull request #4133 from noahshaw11/add-editors-magic-lines

Add editors magic lines

3 weeks agoAdd editors magic lines 4133/head
Noah Shaw [Mon, 25 Jul 2022 23:23:11 +0000 (18:23 -0500)] 
Add editors magic lines

3 weeks agoFix elixir :logger warnings
Ronny [Thu, 28 Jul 2022 07:23:34 +0000 (09:23 +0200)] 
Fix elixir :logger warnings

Migrate deprecated config option variable to new config option for :logger application.
Fixed `warning: :compile_time_purge_level option for the :logger application is deprecated,
use :compile_time_purge_matching instead`.

3 weeks agoMerge pull request #4135 from apache/dedicated-ioq-search-function
Russell Branca [Wed, 27 Jul 2022 22:02:51 +0000 (15:02 -0700)] 
Merge pull request #4135 from apache/dedicated-ioq-search-function

Add ioq:call_search

3 weeks agoAdd ioq:call_search dedicated-ioq-search-function 4135/head
Russell Branca [Tue, 26 Jul 2022 18:22:32 +0000 (11:22 -0700)] 
Add ioq:call_search

3 weeks agoMerge pull request #4106 from apache/4101-add-io-priority
Russell Branca [Mon, 25 Jul 2022 23:27:02 +0000 (16:27 -0700)] 
Merge pull request #4106 from apache/4101-add-io-priority

4101 add io priority

3 weeks agoAdd ioq io_priority functions and system class 4101-add-io-priority 4106/head
Russell Branca [Wed, 13 Jul 2022 22:20:54 +0000 (15:20 -0700)] 
Add ioq io_priority functions and system class

3 weeks agoAdd io_priority classes
Sean Hudgston [Wed, 13 Jul 2022 21:03:59 +0000 (17:03 -0400)] 
Add io_priority classes

- chhtpd_auth_cache
- couch_index

3 weeks agoMerge pull request #4033 from noahshaw11/implement-view_report-function
iilyak [Mon, 25 Jul 2022 19:35:09 +0000 (12:35 -0700)] 
Merge pull request #4033 from noahshaw11/implement-view_report-function

Implement index report functions

3 weeks agoImplement view_report function 4033/head
ncshaw [Fri, 20 May 2022 21:21:44 +0000 (16:21 -0500)] 
Implement view_report function

4 weeks agoMerge pull request #4128 from apache/revert-4094-for-now
Robert Newson [Fri, 22 Jul 2022 17:01:23 +0000 (18:01 +0100)] 
Merge pull request #4128 from apache/revert-4094-for-now

Revert "Replace SHA-1 with SHA-256 for cookie authentication (#4094)"

4 weeks agoRevert "Replace SHA-1 with SHA-256 for cookie authentication (#4094)" 4128/head
Robert Newson [Fri, 22 Jul 2022 16:04:17 +0000 (17:04 +0100)] 
Revert "Replace SHA-1 with SHA-256 for cookie authentication (#4094)"

This reverts commit daff65d8c8772c637eb60365d6e672543163a740.

4 weeks agoMerge pull request #4125 from jiahuili430/couch-tests
iilyak [Thu, 21 Jul 2022 12:40:45 +0000 (05:40 -0700)] 
Merge pull request #4125 from jiahuili430/couch-tests

Remove `couch_tests` from reltool

4 weeks agoRemove `couch_tests` 4125/head
jiahuili [Wed, 20 Jul 2022 21:06:06 +0000 (16:06 -0500)] 
Remove `couch_tests`

`couch_tests` should exist in the tarball but not in the release.

4 weeks agoReturn a 400 response for a single new_edits=false doc update without revisions
Nick Vatamaniuc [Wed, 20 Jul 2022 22:02:12 +0000 (18:02 -0400)] 
Return a 400 response for a single new_edits=false doc update without revisions

We are already doing that for _bulk_docs [1] but forgot to do it for individual
doc updates.



4 weeks agoTurn document update mode atoms into defines
Nick Vatamaniuc [Wed, 20 Jul 2022 20:22:16 +0000 (16:22 -0400)] 
Turn document update mode atoms into defines

We have made some mistakes there in the past mis-typing those. For example in, so as a preliminary PR to fix, where we'd have to again special
case logic based on the update type, let's switch them to be defines so the
compiler double-check the values for us.

4 weeks agoMerge pull request #4123 from apache/dev-run-fix-haproxy-cfg
Robert Newson [Wed, 20 Jul 2022 19:21:35 +0000 (20:21 +0100)] 
Merge pull request #4123 from apache/dev-run-fix-haproxy-cfg

make haproxy config valid again

4 weeks agomake haproxy config valid again 4123/head
Robert Newson [Wed, 20 Jul 2022 17:59:11 +0000 (18:59 +0100)] 
make haproxy config valid again

recent versions of haproxy attempt to discover truncated configs
by insisting on a final blank line.

4 weeks agoFix stats endpoint
ncshaw [Mon, 10 Jan 2022 21:21:05 +0000 (15:21 -0600)] 
Fix stats endpoint

4 weeks agoFix the flaky tests for `create_doc()`
jiahuili [Mon, 18 Jul 2022 20:49:10 +0000 (15:49 -0500)] 
Fix the flaky tests for `create_doc()`

Add `retry_until()` to `create_doc()` to remove unstable tests.

4 weeks agoUpdate application description and dependencies
jiahuili [Mon, 18 Jul 2022 16:06:05 +0000 (11:06 -0500)] 
Update application description and dependencies

1. Update application description and dependencies
2. `couch_uuids_tests.erl` needs to start `config` and `couch_log`
3. Reformat changed files

4 weeks agoUpdate elixir to 1.13
jiahuili [Mon, 18 Jul 2022 16:47:33 +0000 (11:47 -0500)] 
Update elixir to 1.13

1. Update elixir to 1.13
2. Add path() function
3. Removed `hqueue` dependency
4. Removed mix.lock

4 weeks agoTrim X-Auth-CouchDB-Roles header after reading
Ronny [Mon, 18 Jul 2022 18:07:11 +0000 (20:07 +0200)] 
Trim X-Auth-CouchDB-Roles header after reading

4 weeks agoReplace SHA-1 with SHA-256 for cookie authentication (#4094)
Ronny [Mon, 18 Jul 2022 17:01:30 +0000 (19:01 +0200)] 
Replace SHA-1 with SHA-256 for cookie authentication (#4094)

* Replace Sha-1 with Sha-256 for cookie authentication

* Fix proxyauth_test and remove it from skipping tests

4 weeks agoAdd toggle for smoosh queue persistence
Noah Shaw [Tue, 12 Jul 2022 22:11:38 +0000 (17:11 -0500)] 
Add toggle for smoosh queue persistence

5 weeks agoFix not_found error smoosh
Noah Shaw [Tue, 12 Jul 2022 21:21:21 +0000 (16:21 -0500)] 
Fix not_found error smoosh

5 weeks agoFix not calling is_compacting test
Noah Shaw [Thu, 14 Jul 2022 22:27:15 +0000 (17:27 -0500)] 
Fix not calling is_compacting test

5 weeks agoRemove view compaction jobs recovery
Noah Shaw [Tue, 12 Jul 2022 21:08:17 +0000 (16:08 -0500)] 
Remove view compaction jobs recovery

5 weeks agoClean up unused code and invalid spec from replicator
Nick Vatamaniuc [Thu, 14 Jul 2022 15:23:16 +0000 (11:23 -0400)] 
Clean up unused code and invalid spec from replicator

Invalid specs confused dialyzer.

For bulk_docs response handling it detected another left-over clause
from local endpoint support.

It also found un-nessessarily broad case matches in

5 weeks agoRemove some left-over local endpoint clauses in replicator
Nick Vatamaniuc [Wed, 13 Jul 2022 15:05:10 +0000 (11:05 -0400)] 
Remove some left-over local endpoint clauses in replicator

We removed local endpoints a while back and these were left over.

Also remove some unused headers as discovered by erlang_ls and emacs.

5 weeks agoAdd filepath to is_compacting
Noah Shaw [Mon, 11 Jul 2022 20:18:52 +0000 (15:18 -0500)] 
Add filepath to is_compacting

5 weeks agoMerge pull request #4093 from noahshaw11/fix-error-handling-smoosh
Robert Newson [Mon, 11 Jul 2022 15:56:07 +0000 (16:56 +0100)] 
Merge pull request #4093 from noahshaw11/fix-error-handling-smoosh

Improve error handling in smoosh_utils:write_to_file/3

5 weeks agoImprove error handling in smoosh_utils:write_to_file/3 4093/head
Noah Shaw [Thu, 7 Jul 2022 20:33:40 +0000 (15:33 -0500)] 
Improve error handling in smoosh_utils:write_to_file/3

5 weeks agoSkip nightly package uploads since nothing seems to be using them
Nick Vatamaniuc [Sat, 9 Jul 2022 19:07:19 +0000 (15:07 -0400)] 
Skip nightly package uploads since nothing seems to be using them

Since we're not uploading logs skip setting up and fetching creds and
IO log environment variable setup.

5 weeks agoSkip uploading build logs for now
Nick Vatamaniuc [Sat, 9 Jul 2022 19:12:25 +0000 (15:12 -0400)] 
Skip uploading build logs for now

It broke at some point so no need to keep trying to upload them

5 weeks agoFix flaky "validate doc update" elixir test
Nick Vatamaniuc [Fri, 8 Jul 2022 18:15:08 +0000 (14:15 -0400)] 
Fix flaky "validate doc update" elixir test

We saw it fail quite a few times already so adding a retry around it.

Unfortunately it could now fail with a timeout instead of an assert but since
we're updating a document there is no idempotent way to testing it (retry
until, then do the actual assert again below).

6 weeks agoImplement winning_revs_only option for the replicator
Nick Vatamaniuc [Fri, 8 Jul 2022 06:06:59 +0000 (02:06 -0400)] 
Implement winning_revs_only option for the replicator

`winning_revs_only` option replicates only the winning revisions from the
source to the target, effectively discarding conflicts.

This option may be used as an alternative to the custom VDU strategy, or manual
_all_docs read and POST to target scripts, often used by users to discard
conflicted document revisions.

Behind the scenes the implementation simply switches the _changes feed style to
`main_only` from the replication default of `all_docs`.

Additionally, when enabled, the `winning_revs_only` option, will generate a
different replication ID and use a different set of checkpoints than the
default case. This should allows first replicating with `winning_revs_only:
true`, and then later replicate normally between the same endpoints, in
order to "backfill" the rest of the revisions.


6 weeks agoOptimize couch_util:to_hex/1
Nick Vatamaniuc [Wed, 6 Jul 2022 06:49:15 +0000 (02:49 -0400)] 
Optimize couch_util:to_hex/1

When profiling [1] a cluster acting as a replication source, noticed a lot of
time spent in `couch_util:to_hex/1`. That function is used to emit revision ids
amongst other things. When processing a million documents with a 1000 revisions
each, it ends up in the hotpath, so to speak.

Remembering that in Erlang/OTP 24 there is a new `binary:encode_hex/1` [2]
function, decided to benchmark ours vs the OTP implementation. It turns ours is
slower [3] so let's try to use the OTP one.

One difference from the OTP's version is ours emits lower case hex letters,
while the OTP one emits upper case ones. That's why the lookup table had all
the "A"s replaced with "a"s, "B"s with "b"s, etc.

As a bonus, replaced a few calls to `couch_util:to_hex/1` wrapped in `?l2b/1`
or `list_to_binary/1` with just a single call to `couch_util:to_hex_bin/1`.

Existing `couch_util:to_hex/1` version, returning a list ,was left as is and
just calls `to_hex_bin/1` internally and converts the result to a list.

This altered the peak memory usage for the key tree stemming test so had to
alter the magic constants there a bit to avoid flakiness.

> ... eprof:analyze(total,[{sort, time}, {filter, [{time, 1000}, {calls, 100}]}]).

FUNCTION                                                            CALLS        %      TIME  [uS / CALLS]
--------                                                            -----  -------      ----  [----------]
couch_doc:revid_to_str/1                                          1165641     1.71    402860  [      0.35]
couch_key_tree:get_key_leafs_simple/4                             1304102     1.85    435700  [      0.33]
erlang:list_to_integer/2                                           873172     2.00    471140  [      0.54]
gen_server:loop/7                                                   62209     2.11    496932  [      7.99]
couch_key_tree:map_simple/3                                       1829235     2.36    554429  [      0.30]
couch_util:nibble_to_hex/1                                       37334650    16.67   3917127  [      0.10]
couch_util:to_hex/1                                              19834192    34.37   8077050  [      0.41]
---------------------------------------------------------------  --------  -------  --------  [----------]
Total:                                                           91072005  100.00%  23503072  [      0.26]



% ~/src/erlperf/erlperf 'hex:to_hex1(<<210,90,95,232,68,185,66,248,160,33,184,103,181,221,158,96>>).' 'hex:to_hex3(<<210,90,95,232,68,185,66,248,160,33,184,103,181,221,158,96>>).'
Code                                                                                ||        QPS       Time     Rel
hex:to_hex3(<<210,90,95,232,68,185,66,248,160,33,184,103,181,221,158,96>>).          1    2746 Ki     364 ns    100%
hex:to_hex1(<<210,90,95,232,68,185,66,248,160,33,184,103,181,221,158,96>>).          1    1593 Ki     627 ns     58%

(`to_hex1/1` is the existing version and `to_hex3/1` is the OTP version).

6 weeks agoAdd an option to let custodian always use [cluster] n value
Nick Vatamaniuc [Fri, 1 Jul 2022 21:00:05 +0000 (17:00 -0400)] 
Add an option to let custodian always use [cluster] n value

In cases when the application prevents creating dbs with non-default n values
(n=1, n=2) setting `[custodian] use_cluster_n_as_expected_n = true` can help
detect cases when the shard map itself was manpulated and by mistake ended up
with less than `[cluster] n` copies for some ranges.

By default the behavior is unchanged and n value will be deduced from the
number of copies indicated in the shard map documents.

6 weeks agoAdd Erlang 25 to PR CI pipeline and Ubuntu Jammy to full CI
Nick Vatamaniuc [Sat, 2 Jul 2022 22:25:45 +0000 (18:25 -0400)] 
Add Erlang 25 to PR CI pipeline and Ubuntu Jammy to full CI

7 weeks agoBring back POWER full builds
Nick Vatamaniuc [Mon, 27 Jun 2022 17:07:41 +0000 (13:07 -0400)] 
Bring back POWER full builds

7 weeks agoUpdate vm.args for Erlang 23+
Nick Vatamaniuc [Thu, 30 Jun 2022 19:41:33 +0000 (15:41 -0400)] 
Update vm.args for Erlang 23+

 * file IO is using dirty IO schedulers and not the async IO thread pool

 * `-smp enable` does nothing, it's always enalbed

 * kernel polling is always enabled

% erl +SDio 16
Erlang/OTP 23 [erts-] [source] [64-bit] [smp:12:12] [ds:12:12:16] [async-threads:1]

> erlang:system_info(kernel_poll).

> erlang:system_info(dirty_io_schedulers).

7 weeks agoFix Elixir 13 compatibility
Nick Vatamaniuc [Mon, 27 Jun 2022 21:25:46 +0000 (17:25 -0400)] 
Fix Elixir 13 compatibility

Elixir 13 is the only one which seems to work with Erlang 25.

But we can't use Elixir 13 because of a few compatibility issues so have to fix
this. That involves upgrading credo. However, as soon as credo was upgraded it
noticed a bunch of issues which also had to be fixed. Some of those were:

 * + join should be Enum.map_join
 * Invalid module name in teardown `%___MODULE__` -> `%__MODULE__`
 * `Mix.Config` is deprecated, use `Config`
 * `db_test.ex` looks like a test module to credo, so renamed it, otherwise it has to have an .exs extension
 * ibrowse had to be compiled so it could be properly analysed

Tested on macos (intel) with asdf installed elixir and erlang:
  25.0.1    / 1.13.4-otp-25  / 1.13.4-otp-24 / 1.10.4-otp-23

8 weeks agoMerge pull request #4080 from apache/change-irc-url
Ronny [Fri, 24 Jun 2022 12:49:58 +0000 (14:49 +0200)] 
Merge pull request #4080 from apache/change-irc-url

Small url fixes

8 weeks agoSmall url fixes 4080/head
Ronny Berndt [Fri, 24 Jun 2022 11:04:08 +0000 (13:04 +0200)] 
Small url fixes

Fix IRC url
Renaming master -> main for contributing template

8 weeks agoMerge pull request #4079 from apache/backport-from-fdbmain
Ronny [Thu, 23 Jun 2022 14:20:05 +0000 (16:20 +0200)] 
Merge pull request #4079 from apache/backport-from-fdbmain

Backport commits from fdbmain into main (old 3.x)

8 weeks agoBackport commits from fdbmain into main (old 3.x) 4079/head
Ronny Berndt [Thu, 23 Jun 2022 12:16:54 +0000 (14:16 +0200)] 
Backport commits from fdbmain into main (old 3.x)

Cherry-picked commits from 0156a55012b76adb652c11032596d9801c71665e
Thx @kianmeng

8 weeks agoUse https as default protocol for links
Ronny [Wed, 9 Feb 2022 18:59:32 +0000 (19:59 +0100)] 
Use https as default protocol for links

8 weeks agoAdding commit message conventions and update links
Ronny [Wed, 9 Feb 2022 12:25:03 +0000 (13:25 +0100)] 
Adding commit message conventions and update links

Added the commit message conventions from the proposal of discussion #3918 and updated all links to use https and moved all external links to the end of the file

8 weeks ago- rename master to main
Ronny [Mon, 31 Jan 2022 08:39:16 +0000 (09:39 +0100)] 
- rename master to main
- fixing links

8 weeks agoPrevent error:function_clause in check_security/3 if roles claim is malformed (#4070)
Ronny [Wed, 22 Jun 2022 13:32:55 +0000 (15:32 +0200)] 
Prevent error:function_clause in check_security/3 if roles claim is malformed (#4070)

If the given UserRoles/Roles/Names aren't lists, ordersets:from_list/1
or lists:member will fail with an error. Prevent this with Erlang
Pattern Matching and the Robot Butt Rule [1].
Thanks @nickva


8 weeks agoFix couch_debug:opened_files* functions
Nick Vatamaniuc [Mon, 20 Jun 2022 21:45:51 +0000 (17:45 -0400)] 
Fix couch_debug:opened_files* functions

They relied on scanning async IO thread pool ports (`efile`) before . That only
worked in Erlang versions < 21. Since then file IO was moved to dirty
schedulers so listing efile ports doesn't yield any useful results.

The partial fix is iterate through `processes()` and pick out the ones for
which couch_file has put anything into the process dictionary. This will skip
non-couch-file files but it's probably still better than nothing.

8 weeks agoBypass macos builder for now while we sort the rebar issue
Nick Vatamaniuc [Mon, 20 Jun 2022 14:32:23 +0000 (10:32 -0400)] 
Bypass macos builder for now while we sort the rebar issue

8 weeks agoFix full CI build
Nick Vatamaniuc [Sun, 19 Jun 2022 03:04:17 +0000 (23:04 -0400)] 
Fix full CI build

We had to set spidermonkey to 78 in the PR CI builds so do the same
for full builds.

2 months agoFix "variable bound multiple times" warning
Jay Doane [Sat, 18 Jun 2022 15:22:28 +0000 (08:22 -0700)] 
Fix "variable bound multiple times" warning

2 months agoReplace include with include_lib
Jay Doane [Sat, 18 Jun 2022 15:19:36 +0000 (08:19 -0700)] 
Replace include with include_lib

2 months agoDelete unused include_lib
Jay Doane [Sat, 18 Jun 2022 15:18:38 +0000 (08:18 -0700)] 
Delete unused include_lib

2 months agoFix deprecated function warning
Jay Doane [Sat, 18 Jun 2022 15:17:45 +0000 (08:17 -0700)] 
Fix deprecated function warning

Function `httpd_util:hexlist_to_integer/1` will be removed in OTP 26.

2 months agoErlang 25 compatibility - http_uri:parse -> uri_string:parse
Nick Vatamaniuc [Fri, 17 Jun 2022 20:47:12 +0000 (16:47 -0400)] 
Erlang 25 compatibility - http_uri:parse -> uri_string:parse

This fixes the replicator eunit failure:

2 months agoLowest CI image version Erlang 23 doesn't have spidermonkey 1.8.5
Nick Vatamaniuc [Fri, 17 Jun 2022 22:13:54 +0000 (18:13 -0400)] 
Lowest CI image version Erlang 23 doesn't have spidermonkey 1.8.5

2 months agoUpdate erlfmt formatting for smoosh_server
Nick Vatamaniuc [Fri, 17 Jun 2022 22:13:24 +0000 (18:13 -0400)] 
Update erlfmt formatting for smoosh_server

Since the ?STACKTRACE macro is gone now we have actual Erlang syntax in there
and erlfmt knows how to interpret it.

2 months agoRemove Erlang < 23 ifdefs and other macros
Nick Vatamaniuc [Fri, 17 Jun 2022 21:42:27 +0000 (17:42 -0400)] 
Remove Erlang < 23 ifdefs and other macros

2 months agoSet low erlang version in to 23 instead of 20
Nick Vatamaniuc [Fri, 17 Jun 2022 21:41:12 +0000 (17:41 -0400)] 
Set low erlang version in to 23 instead of 20

2 months agoSkip Erlang version != 20 in configure
Nick Vatamaniuc [Fri, 17 Jun 2022 21:40:12 +0000 (17:40 -0400)] 
Skip Erlang version != 20 in configure

They are all != 20 now

2 months agoUpdate rebar config to remove Erlang < 23 support
Nick Vatamaniuc [Fri, 17 Jun 2022 21:23:35 +0000 (17:23 -0400)] 
Update rebar config to remove Erlang < 23 support

2 months agoUpdate CI to remove Erlang < 23
Nick Vatamaniuc [Fri, 17 Jun 2022 21:20:50 +0000 (17:20 -0400)] 
Update CI to remove Erlang < 23

2 months agoIgnore repeated interactive purges
Nick Vatamaniuc [Tue, 14 Jun 2022 19:47:44 +0000 (15:47 -0400)] 
Ignore repeated interactive purges

After enabling default purge request replication, it was possible to get a
`badreq` error because the internal replicator would race with the interactive
worker updates. Since we're dealing with cluster-generated UUIDs if we see the
same UUID, DocId, and Revs again, assume it was applied already and ignore it.

For belt-and-suspenders protection ensure that revs lists don't have duplicates
and are sorted. Do it both in fabric and couch_db in case an operator wants to
do local purges. For extra safety also keep sorting the revs in the duplicate
check for the rare case that the existing purge btree already contain duplicate
revs. Eventually that check could be removed as all the new purge revs lists
would be de-duplicated and sorted.

As a bonus, if it turns out all the purges were applied simply return and avoid
calling the updater process which can be an expensive bottleneck.

2 months agoEnable replicating purge requests between nodes
Nick Vatamaniuc [Mon, 13 Jun 2022 21:21:27 +0000 (17:21 -0400)] 
Enable replicating purge requests between nodes

It seems we had forgotten to enable it and in the case a node if offline when
a clustered purge request is issued, and then re-join they won't know to
process any purge requests issued to other nodes.

2 months agoMerge pull request #4060 from apache/big-r81-add-erlang-25-support
Ronny [Mon, 13 Jun 2022 05:25:45 +0000 (07:25 +0200)] 
Merge pull request #4060 from apache/big-r81-add-erlang-25-support

Add Erlang 25 support

2 months agoAdd Erlang 25 support 4060/head
Ronny Berndt [Sat, 11 Jun 2022 12:18:54 +0000 (14:18 +0200)] 
Add Erlang 25 support

Before merging this PR, PR #16 in couchdb-snappy needs to get merged and tagged with CouchDB-1.0.8

2 months agoMerge pull request #4063 from apache/big-r81-patch-1
Ronny [Sun, 12 Jun 2022 16:01:53 +0000 (18:01 +0200)] 
Merge pull request #4063 from apache/big-r81-patch-1

Display main build status in README

2 months agoDisplay main build status in README 4063/head
Ronny [Sun, 12 Jun 2022 12:39:34 +0000 (14:39 +0200)] 
Display main build status in README

After changing branch 3.x -> main, show status in README

2 months agoReduce complexity of "possible_ancestors" from quadratic to linear.
Nick Vatamaniuc [Fri, 10 Jun 2022 20:33:04 +0000 (16:33 -0400)] 
Reduce complexity of "possible_ancestors" from quadratic to linear.

"Possible ancestors" are all the leaf revisions with a position less than any
of the missing revisions. Previously, for every leaf revision we potentially
checked every missing revision. And with missing revisions list being sorted
from smallest to largest, that meant often traversing most of the missing
revisions list. In general, it meant performing O(R * M) operations, where R is
the number of leaf revision, and M is the number of missing revisions.

The optimization is instead of comparing against every single missing revision
we can compare only against the highest one. If a leaf revision is less than
highest missing revison, then it is already a possible ancestor, so we can stop
checking. Thus, we can go from a quadratic O(R * M) to an O(R) + O(M)
complexity, which, when merging large conflicting documents could give us a
nice performance boost.

2 months agoSkip PPC64 CI builds
Nick Vatamaniuc [Sat, 11 Jun 2022 14:58:21 +0000 (10:58 -0400)] 
Skip PPC64 CI builds

Owner indicated the instance has gone away

2 months agoAdd two larger couch_key_tree:find_missing/2 tests
Nick Vatamaniuc [Thu, 9 Jun 2022 22:52:56 +0000 (18:52 -0400)] 
Add two larger couch_key_tree:find_missing/2 tests

These were used to generate the random rev trees used for bechmarking. But they
also assert useful properties for the function so let's add them to test_util.

Since we added a shuffle/1 function to test_util, DRY some extra shuffle/1
functions as we seem to have a bunch of them sprinkled here and there.

Also, since the rev tree generation code was changed, need to adjust the
`excessive_memory_when_stemming` test as well. To do it, reverted the commit
indicated in the test and then adjusted parameters until the test failed. Then
reset the revert and it failed.

2 months agoOptimize couch_key_tree:find_missing/2
Nick Vatamaniuc [Wed, 8 Jun 2022 16:25:24 +0000 (12:25 -0400)] 
Optimize couch_key_tree:find_missing/2

Previously we always fully traversed the search keys list, when we partitioned
it into possible and impossible sublists. The optimization is we can stop
traversing early if the search keys are sorted as soon as we hit a key that is
>= than the current position in the tree. To keep things simple we can use the
lists:splitwith/2 function which seems to do what we want [1].


2 months agoAvoid using length/1 guard for >0 or ==0 tests in couch_key_tree
Nick Vatamaniuc [Tue, 7 Jun 2022 21:44:26 +0000 (17:44 -0400)] 
Avoid using length/1 guard for >0 or ==0 tests in couch_key_tree

Issue originally discovered by @iilyak using

2 months agoRevert protection for main branch after 3.x -> main move
Nick Vatamaniuc [Tue, 7 Jun 2022 20:22:57 +0000 (16:22 -0400)] 
Revert protection for main branch after 3.x -> main move

See dev list discussion:

Previous PR to disable protection on main so we can replace it: