2 months agoFix typos main
Kian-Meng Ang [Fri, 18 Mar 2022 15:46:39 +0000 (23:46 +0800)] 
Fix typos

3 months agoUse https as default protocol for links
Ronny [Wed, 9 Feb 2022 18:59:32 +0000 (19:59 +0100)] 
Use https as default protocol for links

3 months agoAdding commit message conventions and update links
Ronny [Wed, 9 Feb 2022 12:25:03 +0000 (13:25 +0100)] 
Adding commit message conventions and update links

Added the commit message conventions from the proposal of discussion #3918 and updated all links to use https and moved all external links to the end of the file

3 months ago- rename master to main
Ronny [Mon, 31 Jan 2022 08:39:16 +0000 (09:39 +0100)] 
- rename master to main
- fixing links

3 months agoFix publication of nightly packages (#3926)
Adam Kocoloski [Sat, 5 Feb 2022 22:23:53 +0000 (17:23 -0500)] 
Fix publication of nightly packages (#3926)

3 months agoRandom bits of cleanup (#3914)
Adam Kocoloski [Tue, 1 Feb 2022 16:12:08 +0000 (11:12 -0500)] 
Random bits of cleanup (#3914)

* Remove emilio-related Python script

The Emilio style checker was removed in #3674.

* Remove unused scripts from autotools days

* Update credo to support Elixir v1.12

* Ensure the bin directory sticks around

3 months agoFix typos (#3916)
Ronny [Sat, 29 Jan 2022 16:51:18 +0000 (17:51 +0100)] 
Fix typos (#3916)

Fix typos

3 months agoApply new formatting from erlfmt Jenkins
Adam Kocoloski [Wed, 19 Jan 2022 04:02:32 +0000 (23:02 -0500)] 
Apply new formatting from erlfmt

3 months agoForward port erlfmt improvements from #3837
Adam Kocoloski [Wed, 19 Jan 2022 03:47:31 +0000 (22:47 -0500)] 
Forward port erlfmt improvements from #3837

Credit to @nickva for the original improvements. The main branch is
already Erlang 21+ so the minimum version check is less essential, but
the performance improvements are greatly appreciated!

3 months agoRefactor Jenkins build to dynamically generate test stages (#3903)
Adam Kocoloski [Thu, 20 Jan 2022 03:37:44 +0000 (22:37 -0500)] 
Refactor Jenkins build to dynamically generate test stages (#3903)

This is one of those situations where you go in to make a small change,
see an opportunity for some refactoring, and get sucked into a rabbit
hole that leaves you wondering if you have any idea how computers
actually work. My initial goal was simply to update the Erlang version
used in our binary packages to a modern supported release. Along the
way I decided I wanted to figure out how to eliminate all the copypasta
we generate for making any change to this file, and after a few days of
hacking here we are. This rewrite has the following features:

* Updates to use Debian 11 (current stable) as the base image for
  building releases and packaging repos.

* Defaults to Erlang 24.2 as the embedded Erlang version in packages.

* Dynamically generates the parallel build stages used to test and
  package CouchDB on various OSes. This is accomplished through a bit
  of scripted pipeline code that relies on two new methods defined at
  the beginning of the Jenkinsfile, one for "native" builds on macOS
  and FreeBSD and one for container-based builds. See comments in the
  Jenkinsfile for additional details.

* Expands commands like `make check` into a series of steps to improve
  visibility. The Jenkins UI will now show the time spent in each step
  of the build process, and if a step (e.g. `make eunit`) fails it will
  only expand the logs for that step by default instead of showing the
  logs for the entire build stage. The downside is that if we do make
  changes to the series of targets underneath `check` we need to
  remember to update the Jenkinsfile as well.

* Starts per-stage timer _after_ agent is acquired. Previously builds could
  fail with a 15m timeout when all they did was sit in the build queue.

3 months agoTweak Elixir formatting
Adam Kocoloski [Sat, 15 Jan 2022 21:02:05 +0000 (21:02 +0000)] 
Tweak Elixir formatting

It seems different versions of mix cannot agree on how these lines
ought to be formatted, so let's try to help out.

3 months agoDecompose scripts into specific steps
Adam Kocoloski [Wed, 19 Jan 2022 02:26:11 +0000 (21:26 -0500)] 
Decompose scripts into specific steps

This makes it easier to observe the pipeline progres in the UI. We get
timings for each step in the build, and if one of the steps fails the
logs for that step will be the only ones expanded by default. We can
also label each of the steps to provide a bit more context to the
developer about what the CI pipeline is actually doing.

3 months agoStart stage-specific timers after acquiring agent
Adam Kocoloski [Wed, 19 Jan 2022 02:08:29 +0000 (21:08 -0500)] 
Start stage-specific timers after acquiring agent

This avoids the situation where a build fails with a timeout because
all the docker-based agents were busy running other jobs. Jenkins'
semantics for options.timeout is that the stage-specific timeout starts
the countdown even while waiting for an agent matching the selected
label to become available. We see occasional spurious job failures as a

3 months agoBump Credo to 1.5.6 for Elixir 1.12 support
Adam Kocoloski [Wed, 19 Jan 2022 02:08:01 +0000 (21:08 -0500)] 
Bump Credo to 1.5.6 for Elixir 1.12 support

3 months agoUse specific images for each Erlang version
Adam Kocoloski [Fri, 14 Jan 2022 17:49:35 +0000 (12:49 -0500)] 
Use specific images for each Erlang version

Instead of building one image with all supported Erlang versions through
kerl, this configuration looks for a specific container image for each
Erlang version. Decoupling it like this enables us to more easily adopt
newer distros for newer Erlang versions, and to build new images with
patch releases of Erlang without needing a simultaneous PR to the
CouchDB repo to pick them up in CI (although some change to Jenkins
might be needed to avoid images being cached for too long when a stable
tag changes).

4 months agoRemove CI support for Ubuntu 16.04
Adam Kocoloski [Sat, 15 Jan 2022 12:40:12 +0000 (07:40 -0500)] 
Remove CI support for Ubuntu 16.04

4 months agoUpdate Jenkins Erlang versions, add 24 (#3892)
Adam Kocoloski [Thu, 13 Jan 2022 00:18:46 +0000 (19:18 -0500)] 
Update Jenkins Erlang versions, add 24 (#3892)

4 months agoFeat/support sm91 4.x from lostnet (#3884)
Will Young [Wed, 12 Jan 2022 17:16:01 +0000 (17:16 +0000)] 
Feat/support sm91 4.x from lostnet (#3884)

* Win32-SM91 support and fixes

* spidermonkey_68 identified as spidermonkey_60 and erroneously(?) blocked by configure on aarch64 #3149

* remove unnecessary shell when setting ERL_AFLAGS

* fix foundationdb urls in github workflow

* quote AFLAGS like win echo and fix references to pwd

Co-authored-by: Will <>
4 months agoRemove ERL_OPTS
ncshaw [Fri, 17 Dec 2021 18:46:53 +0000 (13:46 -0500)] 

5 months agoAdd rebar3 and erlfmt install commands to configure.ps1 #3873
Will [Tue, 14 Dec 2021 13:09:47 +0000 (13:09 +0000)] 
Add rebar3 and erlfmt install commands to configure.ps1 #3873

5 months agoMerge pull request #3870 from apache/always-send-all-cookie-attributes-main
Robert Newson [Sat, 11 Dec 2021 10:46:00 +0000 (10:46 +0000)] 
Merge pull request #3870 from apache/always-send-all-cookie-attributes-main

Always send all cookie attributes

5 months agoAlways send all cookie attributes 3870/head
Robert Newson [Thu, 9 Dec 2021 19:17:59 +0000 (19:17 +0000)] 
Always send all cookie attributes

5 months agowget pull cleanup and downgrade fdb
Will Young [Thu, 9 Dec 2021 17:43:38 +0000 (18:43 +0100)] 
wget pull cleanup and downgrade fdb

5 months agoMerge pull request #3862 from apache/delete-session-cookie-samesite-main
Robert Newson [Mon, 6 Dec 2021 15:02:29 +0000 (15:02 +0000)] 
Merge pull request #3862 from apache/delete-session-cookie-samesite-main

Add SameSite setting when clearing session cookie

5 months agoAdd SameSite setting when clearing session cookie delete-session-cookie-samesite-main 3862/head
Robert Newson [Mon, 6 Dec 2021 13:55:34 +0000 (13:55 +0000)] 
Add SameSite setting when clearing session cookie

5 months agofeat(couchjs): add support for SpiderMonkey 91esr (#3842)
Will Young [Thu, 2 Dec 2021 14:42:03 +0000 (14:42 +0000)] 
feat(couchjs): add support for SpiderMonkey 91esr (#3842)

5 months agoFix Jenkins email quoting failures
Nick Vatamaniuc [Tue, 23 Nov 2021 16:13:42 +0000 (11:13 -0500)] 
Fix Jenkins email quoting failures

Should fix:
Warning: A secret was passed to "mail" using Groovy String interpolation, which is insecure.

6 months agoPort changes from Makefile to
Adam Kocoloski [Fri, 12 Nov 2021 20:28:05 +0000 (15:28 -0500)] 
Port changes from Makefile to

I didn't touch the mango-test invocation because it confused me. I'm
not sure how Python.exe and dev\run are interacting there.

I'm also not sure about forward vs. backslash in these quoted mix
invocations. I stuck with the convention I saw already in the Makefile;
i.e. forward slashes for regular arguments to dev\run, backslashes for
quoted arguments to the mix invocation issued by dev\run.

6 months agoClean up mango-test
Adam Kocoloski [Fri, 12 Nov 2021 20:19:16 +0000 (15:19 -0500)] 
Clean up mango-test

* Eliminate remaining use of TEST_OPTS
* Standardize dev/run invocation, matching elixir-integration

6 months agoFix Python formatting, restore python-black target
Adam Kocoloski [Fri, 12 Nov 2021 19:49:15 +0000 (14:49 -0500)] 
Fix Python formatting, restore python-black target

This also allows us to eliminate the `check-all-tests` target, since
we've re-enabled all the test suites (and excluded specific tests as

6 months agoRemove config.erl target dependency
Adam Kocoloski [Fri, 12 Nov 2021 19:38:50 +0000 (14:38 -0500)] 
Remove config.erl target dependency

Frankly I have no idea what this does, but it doesn't seem relevant.

6 months agoReorder tests for clarity, reduce dependencies
Adam Kocoloski [Fri, 12 Nov 2021 19:32:29 +0000 (14:32 -0500)] 
Reorder tests for clarity, reduce dependencies

I removed the `elixir-init` dependency from everything except the
`elixir-suite` target. The other targets are likely to be run
interactively, in which case mix will ask if the local rebar / rebar3
etc. should be installed if they're not already (and they almost always
will be). A little less log cruft, a little more speed.

I also removed `MIX_ENV` exports where their inclusion didn't seem to
matter, e.g. on the code formatting and static analysis checks.

6 months agoConsolidate ExUnit integration test configuration
Adam Kocoloski [Fri, 12 Nov 2021 19:15:48 +0000 (14:15 -0500)] 
Consolidate ExUnit integration test configuration

Two main pieces of work here:

1) Consoidate elixir / elixir-only / elixir-suite into two targets:

- elixir-integration, which only runs ExUnit integration tests
- elixir-suite, which depends on elixir-integration and also calls
  format + credo

This preserves the flexibility of the previous targets while reducing
code duplication and removing some obsolete configurations

2) Limit the number of places where we define test configurations

Previously we were relying on dev/run to inject config entries in
addition to specifiying a .ini file on the command-line. We sometimes
also write config settings in the test definitions themselves. This is
at least an initial attempt to just define the config settings in one

6 months agoRegenerate suite.elixir
Adam Kocoloski [Fri, 12 Nov 2021 13:51:44 +0000 (08:51 -0500)] 
Regenerate suite.elixir

This is the result of running

MIX_ENV=integration mix suite > test/elixir/test/config/suite.elixir

6 months agoDo not start test name with "test "
Adam Kocoloski [Fri, 12 Nov 2021 17:03:28 +0000 (12:03 -0500)] 
Do not start test name with "test "

The code that generates suite.elixir will repeatedly strip the "test "
from the name of the test when writing the file, resulting in a mismatch
between the actual test name and what's in suite.elixir. You can see
this by searching for e.g. COUCHDB-497 in the suite file.

I tried using String.replace_prefix instead of String.replace_leading in
test_name() but that function seems to get called multiple times during
the test grouping. Simpler to just avoid naming the tests that way for

6 months agoUse tags in .exs file as exclusion source of truth
Adam Kocoloski [Fri, 12 Nov 2021 14:24:52 +0000 (09:24 -0500)] 
Use tags in .exs file as exclusion source of truth

We have been redundantly excluding tests by listing them in the
skip.elixir file and also adding a `:pending` or `:skip` tag. Output of
`elixir-suite` before:

Finished in 88.4 seconds
306 tests, 0 failures, 16 excluded, 1 skipped

and after:

Finished in 85.8 seconds
438 tests, 0 failures, 135 excluded, 1 skipped

6 months agoProtect release branches with .asf.yaml config (#3285)
Glynn Bird [Wed, 10 Nov 2021 12:18:18 +0000 (12:18 +0000)] 
Protect release branches with .asf.yaml config (#3285)

The intent here is to require "branch must be up to date
before merge / squash / rebase", and to further require a clean bill of
health from Jenkins for `main` and `3.x`. I'm not so confident about
the CI health of older branches to enable that protection there, and at
any rate it's unlikely we'd be committing anything to those branches

Co-authored-by: Glynn Bird <>
Co-authored-by: Adam Kocoloski <>
6 months agoTemporarily exclude failing tests
Adam Kocoloski [Fri, 5 Nov 2021 03:30:46 +0000 (23:30 -0400)] 
Temporarily exclude failing tests

6 months agoExclude _show/_list/_rewrite tests
Adam Kocoloski [Fri, 5 Nov 2021 03:30:23 +0000 (23:30 -0400)] 
Exclude _show/_list/_rewrite tests

6 months agoRe-enable most elixir tests
Paul J. Davis [Thu, 24 Sep 2020 20:06:13 +0000 (15:06 -0500)] 
Re-enable most elixir tests

This change tags all tests that are currently failing against main with
a `pending` tag that is then excluded.

6 months agoRemove quorum tests
Paul J. Davis [Thu, 24 Sep 2020 18:36:10 +0000 (13:36 -0500)] 
Remove quorum tests

Quorum is no longer a thing now that we rely on FoundationDB.

7 months agoMerge pull request #3764 from apache/changes_duration
Robert Newson [Mon, 27 Sep 2021 08:57:07 +0000 (09:57 +0100)] 
Merge pull request #3764 from apache/changes_duration

support maximum changes_duration configuration option

7 months agosupport maximum changes_duration configuration option 3764/head
Robert Newson [Fri, 24 Sep 2021 13:03:35 +0000 (14:03 +0100)] 
support maximum changes_duration configuration option

8 months agoMerge pull request #3753 from cloudant/fix-ensure_exunit_started
iilyak [Thu, 16 Sep 2021 11:17:49 +0000 (04:17 -0700)] 
Merge pull request #3753 from cloudant/fix-ensure_exunit_started

Fix logic in ensure_exunit_started

8 months agoFix logic in ensure_exunit_started 3753/head
ILYA Khlopotov [Wed, 15 Sep 2021 14:40:29 +0000 (07:40 -0700)] 
Fix logic in ensure_exunit_started

8 months agoSeparate bad_config_test into multiple tests
Jay Doane [Thu, 9 Sep 2021 02:43:44 +0000 (19:43 -0700)] 
Separate bad_config_test into multiple tests

Instead of running as a single test, separate each corresponding field
and error into its own test, with its own timeout. A typical run now
looks like:

make eunit apps=couch_views suites=couch_views_batch_impl tests=bad_config_test_

======================== EUnit ========================
couch_views_batch_impl:186: bad_config_test_ (batch_initial_size)...[0.104 s] ok
couch_views_batch_impl:186: bad_config_test_ (batch_search_increment)...[0.077 s] ok
couch_views_batch_impl:186: bad_config_test_ (batch_sense_increment)...[0.074 s] ok
couch_views_batch_impl:186: bad_config_test_ (batch_max_tx_size_bytes)...[0.077 s] ok
couch_views_batch_impl:186: bad_config_test_ (batch_max_tx_time_msec)...[0.076 s] ok
couch_views_batch_impl:186: bad_config_test_ (batch_threshold_penalty)...[0.074 s] ok
  All 6 tests passed.

Previously, these tests were timing out when CI became resource
constrained. Examples of multiple runs show how prone to timing out
after 5 seconds the old monolithic bad_config_test was:

  couch_views_batch_impl: good_config_test...[0.217 s] ok
  couch_views_batch_impl: bad_config_test...[1.347 s] ok

  couch_views_batch_impl: good_config_test...[0.393 s] ok
  couch_views_batch_impl: bad_config_test...[2.332 s] ok

  couch_views_batch_impl: good_config_test...[0.358 s] ok
  couch_views_batch_impl: bad_config_test...[2.211 s] ok

  couch_views_batch_impl: good_config_test...[0.942 s] ok
  couch_views_batch_impl: bad_config_test...*timed out*

  couch_views_batch_impl: good_config_test...[1.179 s] ok
  couch_views_batch_impl: bad_config_test...*timed out*

  couch_views_batch_impl: good_config_test...[0.497 s] ok
  couch_views_batch_impl: bad_config_test...*timed out*

8 months agoIncrease timeout for couch views active tasks tests
Jay Doane [Wed, 8 Sep 2021 18:06:02 +0000 (11:06 -0700)] 
Increase timeout for couch views active tasks tests

This doubles the default timeout to 10 seconds for these two tests.

They typically take a long time to run:

    couch_views_active_tasks_test:69: active_tasks_test_ (verify_basic_active_tasks)...[2.444 s] ok
    couch_views_active_tasks_test:70: active_tasks_test_ (verify_muliple_active_tasks)...[2.678 s] ok

    couch_views_active_tasks_test:69: active_tasks_test_ (verify_basic_active_tasks)...[2.646 s] ok
    couch_views_active_tasks_test:70: active_tasks_test_ (verify_muliple_active_tasks)...[2.743 s] ok

    couch_views_active_tasks_test:69: active_tasks_test_ (verify_basic_active_tasks)...[2.910 s] ok
    couch_views_active_tasks_test:70: active_tasks_test_ (verify_muliple_active_tasks)...[3.063 s] ok

and when resources are constrained, often time out:

    couch_views_active_tasks_test:69: active_tasks_test_ (verify_basic_active_tasks)...[2.044 s] ok
    couch_views_active_tasks_test:70: active_tasks_test_ (verify_muliple_active_tasks)...*timed out*

    couch_views_active_tasks_test:69: active_tasks_test_ (verify_basic_active_tasks)...[1.777 s] ok
    couch_views_active_tasks_test:70: active_tasks_test_ (verify_muliple_active_tasks)...*timed out*

    couch_views_active_tasks_test:69: active_tasks_test_ (verify_basic_active_tasks)...*timed out*
    couch_views_active_tasks_test:70: active_tasks_test_ (verify_muliple_active_tasks)...*timed out*

8 months agoSet structured logger to info level for exunit tests
Jay Doane [Fri, 3 Sep 2021 23:58:17 +0000 (16:58 -0700)] 
Set structured logger to info level for exunit tests

Suppress excessive debug logging of the form:

mix test --cover --trace src/chttpd/test/exunit/pagination_test.exs:232

  * test Legacy API (10 docs) : _all_docs/queries
12:40:01.895 [debug] [user: "adm", what: :login_attempt]
12:40:01.906 [debug] [username: 'adm', what: :successful_cookie_auth]
12:40:01.925 [debug] [username: 'adm', what: :successful_cookie_auth]
12:40:01.934 [debug] [username: 'adm', what: :successful_cookie_auth]
12:40:01.944 [debug] [username: 'adm', what: :successful_cookie_auth]
12:40:01.952 [debug] [username: 'adm', what: :successful_cookie_auth]
12:40:01.960 [debug] [username: 'adm', what: :successful_cookie_auth]
12:40:01.968 [debug] [username: 'adm', what: :successful_cookie_auth]
12:40:01.977 [debug] [username: 'adm', what: :successful_cookie_auth]
12:40:01.985 [debug] [username: 'adm', what: :successful_cookie_auth]
12:40:01.995 [debug] [username: 'adm', what: :successful_cookie_auth]
12:40:02.004 [debug] [username: 'adm', what: :successful_cookie_auth]
12:40:02.013 [debug] [username: 'adm', what: :successful_cookie_auth]
12:40:02.019 [debug] [username: 'adm', what: :successful_cookie_auth]
  * test Legacy API (10 docs) : _all_docs/queries (152.2ms)

8 months agoSuppress compiler warnings
Jay Doane [Fri, 3 Sep 2021 06:37:43 +0000 (23:37 -0700)] 
Suppress compiler warnings

8 months agoReplace couch_js with js_engine in exunit pagination test
Jay Doane [Sat, 4 Sep 2021 20:36:54 +0000 (13:36 -0700)] 
Replace couch_js with js_engine in exunit pagination test


8 months agoIncrease timeout for multiple ddoc indexer test
Jay Doane [Sat, 4 Sep 2021 00:06:52 +0000 (17:06 -0700)] 
Increase timeout for multiple ddoc indexer test

8 months agoFix 'clause cannot match' compiler warning
Jay Doane [Wed, 1 Sep 2021 23:46:21 +0000 (16:46 -0700)] 
Fix 'clause cannot match' compiler warning

Fixes this warning:

couchdb/src/couch_eval/src/couch_eval.erl:168: Warning: this clause cannot match because a previous clause at line 167 always matches

9 months agoUse couch_eval for changes filtering
garren smith [Mon, 9 Aug 2021 11:30:35 +0000 (13:30 +0200)] 
Use couch_eval for changes filtering

Use couch_eval when filtering docs with a filter function or a map
function. This allows CouchDB to configured to use different engines
through couch_eval.

9 months agoImprove handling of + in URLs
ncshaw [Wed, 23 Jun 2021 19:48:40 +0000 (15:48 -0400)] 
Improve handling of + in URLs

9 months agoBump mochiweb version
ncshaw [Thu, 5 Aug 2021 19:07:22 +0000 (15:07 -0400)] 
Bump mochiweb version

9 months agoRemove emilio
ncshaw [Mon, 19 Jul 2021 17:57:48 +0000 (13:57 -0400)] 
Remove emilio

9 months agoPort attachment deletion fix from 3.x
Nick Vatamaniuc [Fri, 30 Jul 2021 21:05:29 +0000 (17:05 -0400)] 
Port attachment deletion fix from 3.x

Port ba638783b5d87b855939dae69fd63ffd41cb5ed7 from 3.x

This should fix the edge case where deletion with a bad rev was returning a
404. Now it should returna 409.


9 months agoUse correct fabric function names in elixir tests
Jay Doane [Wed, 28 Jul 2021 19:20:42 +0000 (12:20 -0700)] 
Use correct fabric function names in elixir tests

This eliminates several warnings when running `make exunit`.

9 months agoAdd in option to run different js engine for tests
garren smith [Thu, 22 Jul 2021 10:03:44 +0000 (12:03 +0200)] 
Add in option to run different js engine for tests

Add a `js_engine` field when starting test applications.
When the couch apps are loaded it will load the js engine defined
in the ini files.

This allows us to swop out and use different javascript engines for
`couch_eval` for eunit tests.

9 months agomove validate doc update to go through couch_eval
garren smith [Thu, 15 Jul 2021 14:34:31 +0000 (16:34 +0200)] 
move validate doc update to go through couch_eval

10 months agoFix response code for nonexistent attachment
ncshaw [Wed, 14 Jul 2021 19:37:26 +0000 (15:37 -0400)] 
Fix response code for nonexistent attachment

10 months agoFix "error":"case_clause" using latest=true
Nick Vatamaniuc [Tue, 13 Jul 2021 07:18:43 +0000 (03:18 -0400)] 
Fix "error":"case_clause" using latest=true

This is a port of
to main

Issue #3424

10 months agoFix __MODULE__ compiler variable warning
Jay Doane [Sat, 10 Jul 2021 18:40:05 +0000 (11:40 -0700)] 
Fix __MODULE__ compiler variable warning

warning: unknown compiler variable "___MODULE__" (expected one of __MODULE__, __ENV__, __DIR__, __CALLER__, __STACKTRACE__)
  src/couchdb/test/elixir/lib/step/start.ex:62: Couch.Test.Setup.Step.Start.teardown/2

10 months agocomment out max_document_size
jiahuili [Thu, 1 Jul 2021 19:59:50 +0000 (14:59 -0500)] 
comment out max_document_size

10 months agoMerge pull request #3637 from noahshaw11/remove-case-sensitivity-for-basic-auth
iilyak [Fri, 2 Jul 2021 11:21:01 +0000 (04:21 -0700)] 
Merge pull request #3637 from noahshaw11/remove-case-sensitivity-for-basic-auth

Remove case sensitivity for basic auth

10 months agoFix FDB max_document_size, binary_chunk_size, and max_attachment_size checks
ncshaw [Wed, 16 Jun 2021 20:02:09 +0000 (16:02 -0400)] 
Fix FDB max_document_size, binary_chunk_size, and max_attachment_size checks

10 months agoFix CI (#3650)
Bessenyei Balázs Donát [Thu, 1 Jul 2021 14:55:26 +0000 (16:55 +0200)] 
Fix CI (#3650)

10 months agoUse correct socks5 proxy ibrowse options
Nick Vatamaniuc [Wed, 30 Jun 2021 07:32:39 +0000 (03:32 -0400)] 
Use correct socks5 proxy ibrowse options

This is a port of the 3.x PR to main

10 months agoRemove case sensitivity for basic auth and modify tests 3637/head
ncshaw [Tue, 22 Jun 2021 20:23:18 +0000 (16:23 -0400)] 
Remove case sensitivity for basic auth and modify tests

10 months agoAdd rebar3 to .gitignore
ncshaw [Mon, 28 Jun 2021 18:05:41 +0000 (14:05 -0400)] 
Add rebar3 to .gitignore

10 months agoFormat all erl files (#3639)
Bessenyei Balázs Donát [Mon, 28 Jun 2021 12:05:57 +0000 (14:05 +0200)] 
Format all erl files (#3639)

* Format all src files and remove beam comparison

* Apply make erlfmt-format

10 months agoFix formatter python compatibility (#3640)
Bessenyei Balázs Donát [Mon, 28 Jun 2021 11:03:15 +0000 (13:03 +0200)] 
Fix formatter python compatibility (#3640)

10 months agoReformat src files with erlfmt (#3568)
Bessenyei Balázs Donát [Thu, 24 Jun 2021 14:46:03 +0000 (16:46 +0200)] 
Reformat src files with erlfmt (#3568)

10 months agoMerge pull request #3636 from jiahuili430/normalize-config-options-main
iilyak [Thu, 24 Jun 2021 13:26:14 +0000 (06:26 -0700)] 
Merge pull request #3636 from jiahuili430/normalize-config-options-main

Normalize config options main

10 months agoComment out some config options in the main branch 3636/head
jiahuili [Mon, 21 Jun 2021 20:13:59 +0000 (15:13 -0500)] 
Comment out some config options in the main branch

11 months agoRemove attempt to download GeoTrust CA (#3616)
Adam Kocoloski [Thu, 10 Jun 2021 15:29:09 +0000 (11:29 -0400)] 
Remove attempt to download GeoTrust CA (#3616)

This URL is a 404 now, but fortunately we don't care as is no longer dependent on GeoTrust at all in its
root of trust.

11 months agoBump ibrowse to 4.4.2-5
Nick Vatamaniuc [Mon, 7 Jun 2021 22:57:24 +0000 (18:57 -0400)] 
Bump ibrowse to 4.4.2-5

Previously, in 4.4.2-4 ibrowse upstream rebase also included the commit which
unconditionally unquoted userinfo credentials. Since we know have a better way
of handing basic auth creds bump ibrowse with a rebase which doesn't include
that commit.

11 months agoBump ibrowse to 4.4.2-4 and mochiweb to v2.21.0
Nick Vatamaniuc [Mon, 7 Jun 2021 17:41:04 +0000 (13:41 -0400)] 
Bump ibrowse to 4.4.2-4 and mochiweb to v2.21.0

* mochiweb : upgrade crypto functions to support OTP 23+
* ibrowse : update time functions and fix flaky unit test

11 months agoBump ibrowse to 4.4.2-3
Nick Vatamaniuc [Fri, 4 Jun 2021 04:29:07 +0000 (00:29 -0400)] 
Bump ibrowse to 4.4.2-3

This reverts the update to ibrowse to always unquote userinfo
(username and password) in endpoint URLs, as it has issues with
compatibility since it would unqoute passwords which had a literal '+'
as ' ' for example.

There is a new recommended way of specifying passwords which may
contain '@', ':' and other such symbols in


11 months agoImprove basic auth credentials handling in replicator
Nick Vatamaniuc [Thu, 3 Jun 2021 18:09:43 +0000 (14:09 -0400)] 
Improve basic auth credentials handling in replicator

This is a port of commit ecd266b0e87f44e1080cabdb4c28e4758f5a4406 from 3.x to
main. Including the same commit message from there for completeness and then
towards the end, there is a description of changes required to port the PR to

Previously, there were two ways to pass in basic auth credentials for endpoints
-- using URL's userinfo part, and encoding them in an `"Authorization": "basic
..."` header. Neither one is ideal for these reasons:

 * Passwords in userinfo don't allow using ":", "@" and other characters.
   However, even after switching to always unquoting them like we did recently
   [1], it could break authentication for usernames or passwords previously
   containing "+" or "%HH" patterns, as "+" might now be decoded to a " ".

 * Base64 encoded headers need an extra step to encode them. Also, quite often
   these encoded headers are confused as being "encrypted" and shared in a
   clear channel.

To improve this, revert the recent commit to unquote URL userinfo parts to
restore backwards compatibility, and introduce a way to pass in basic auth
credentials in the "auth" object. The "auth" object was already added a while
back to allow authentication plugins to store their credentials in it. The
format is:

   "source": {
       "url": "https://host/db",
       "auth": {
           "basic": {

{"auth" : "basic" : {...}} object is checked first, and if credentials are
provided, they will be used. If they are not then userinfo and basic auth
header will be parsed.

Internally, there was a good amount duplication related to parsing credentials
from userinfo and headers in replication ID generation logic and in the auth
session plugin. As a cleanup, consolidate that logic in the
`couch_replicator_parse` module.

The commit is quite different from the 3.x one for these two reasons:

 * `main` uses two types of replication endpoint "objects": `#httpdb` records
   and `HttpDb` maps. In most cases it uses maps which can be serialized and
   deserialized to and from json. But in lower level, connection handling code
   in couch_replicator_httpc, it uses `#httpdb` records. This explain the need
   to still handle both representations. Auth session plugin, for instance,
   uses the lower level #httpdb records while replicator ID handling code uses
   the map based one.

 * `main` has all the parsing of replication documents and `_replicate` request
   bodies in a separate `couch_replicator_parse`. So, most of the code which
   handles normalizing basic auth creds is there instead of
   `couch_replicator_docs` or `couch_replicator_utils` like it is in 3.x

11 months agoDocument aegis.cache_deletion_grace_sec
Jay Doane [Wed, 2 Jun 2021 20:07:58 +0000 (13:07 -0700)] 
Document aegis.cache_deletion_grace_sec

11 months agoGrace period before deleting aegis cached items
Jay Doane [Fri, 28 May 2021 05:17:15 +0000 (22:17 -0700)] 
Grace period before deleting aegis cached items

On a heavily loaded system, it is possible for `is_key_fresh/1` to
return true, but subsequently for `lookup/2` to return `{error, not_found}`.

An example occurred during performance testing, where a `badmatch`
error in aegis_server generated this stack trace:

May 6 21:34:48 c-fdbcore-perf-api-5bc54ff569-jkjxx db error [error] 2021-05-06T21:34:41.337533Z dbcore@ <0.18553.273> 6adfd54aa5 req_err(3074846293) {{badmatch,{error,not_found}},
  {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]} : {gen_server,call,
         #{after_doc_read => undefined,before_doc_update => undefined,
           check_current_ts => -576456460633,db_options => [],
           db_prefix => <<21,22,21,15,1,23,5,202,10,0>>,
           db_version => <<"f518950b94a759bb410f86d32126f421">>,
           interactive => true,is_encrypted => true,
           layer_prefix => <<21,22>>,
           md_version => <<0,0,12,155,121,36,204,80,0,0>>,
           name =>
           revs_limit => 1000,
           security_doc => {[]},
           security_fun => undefined,
           tx => {erlfdb_transaction,#Ref<0.4001737934.1270480897.28885>},
           user_ctx =>
           uuid => <<"97e681cfd7224680f72213b8bf59a25c">>,
           validate_doc_update_funs => []},

    [<<"gen_server:call/2 L206">>,<<"aegis_server:encrypt/3 L93">>,<<"fabric2_fdb:-write_doc_body/2-fun-0-/3 L1430">>,<<"lists:foreach/2 L1338">>,<<"fabric2_fdb:write_doc/6 L851">>,<<"fabric2_db:update_doc_interactive/4 L1944">>,<<"fabric2_db:update_docs_interactive/5 L1808">>,<<"fabric2_db:batch_update_interactive_tx/1 L1752">>]

The crash occurred in aegis_server [1] (now at a slightly different
line from the trace) because `lookup/2` returned `{error, not_found}`
[2] when unable to find the UUID in the cache.

This commit introduces a configurable grace period, defaulting to 5
seconds, which expired entries remain in the cache, so that such a
race is less likely to occur. It also handles the error case,
preventing a badmatch, and finally DRYs out the code by factoring a
`handle_crypto_call/6` function for both encryption operations.


11 months agoImplement aegis:key_manager/0
Jay Doane [Sun, 30 May 2021 01:09:35 +0000 (18:09 -0700)] 
Implement aegis:key_manager/0

A convenience function for configuration debugging.

11 months agoFix get_db_info aegis server test for operational key manager
Jay Doane [Sat, 29 May 2021 23:46:53 +0000 (16:46 -0700)] 
Fix get_db_info aegis server test for operational key manager

When aegis is configured to use an operational key manager, the
get_db_info test can fail since the `passthrough` option is used, but
`get_db_info/1` doesn't get mocked, resulting in the operational key
manager's implementation unintentially being used in the test.

This replaces the `passthrough` option with `non_strict`, and mocks
`get_db_info/1` so that the same mocked functions are used in tests
regardless of whether `AEGIS_KEY_MANAGER` is `aegis_noop_key_manager`,
or an operational key manager.

11 months agoEnable aegis eunit coverage
Jay Doane [Mon, 31 May 2021 02:46:52 +0000 (19:46 -0700)] 
Enable aegis eunit coverage

11 months agoImprove stability of couch_js_proc_manager_tests
Jay Doane [Tue, 1 Jun 2021 17:31:32 +0000 (10:31 -0700)] 
Improve stability of couch_js_proc_manager_tests

11 months agoDelete unused chttpd_show:apply_etag/2 function
Jay Doane [Mon, 31 May 2021 19:58:25 +0000 (12:58 -0700)] 
Delete unused chttpd_show:apply_etag/2 function

This code is generating an unused function compilation warning.

11 months agoDo not warn for invalid offset with spidermonkey 60 on mac
Jay Doane [Thu, 27 May 2021 05:31:54 +0000 (22:31 -0700)] 
Do not warn for invalid offset with spidermonkey 60 on mac

11 months agomove couch_httpd_auth options to chttpd_auth main
Jiahui Li [Tue, 25 May 2021 22:18:24 +0000 (17:18 -0500)] 
move couch_httpd_auth options to chttpd_auth main

Solved conflicts from "cherry-pick" 3.x commit with kdiff3 merge tool.

11 months agoRe-enable more elixir tests (#3572)
Bessenyei Balázs Donát [Thu, 27 May 2021 09:26:09 +0000 (11:26 +0200)] 
Re-enable more elixir tests (#3572)

* Re-enable cookie auth elixir test

* Re-enable copy elixir test

* Add retry to cookie auth test

11 months agoBump hyper dependency to 2.2.0-7
Nick Vatamaniuc [Wed, 26 May 2021 15:02:35 +0000 (11:02 -0400)] 
Bump hyper dependency to 2.2.0-7

Upgrade random -> rand

11 months agoDelete unused variables
Jay Doane [Fri, 21 May 2021 02:19:24 +0000 (19:19 -0700)] 
Delete unused variables

Eliminate compiler warnings

11 months agoMoved some config options from httpd to chttpd
Jiahui Li [Mon, 17 May 2021 12:41:45 +0000 (07:41 -0500)] 
Moved some config options from httpd to chttpd

Solved conflicts from "cherry-pick" 3.x commit

11 months agoUse the last commit result even when there are intermediate retries
Nick Vatamaniuc [Tue, 18 May 2021 18:21:30 +0000 (14:21 -0400)] 
Use the last commit result even when there are intermediate retries

Previously, if a transaction got a `commit_unknown_result`, during the next
successful attempt that result would be returned to the user. However, if the
next attempt was another retryable error, then the committed result was ignored
and the whole transaction would be applied again. This could result in document
update transactions conflicting with themselves as described in issue

To prevent that from happening we remember that there was an
`commit_unknown_result` error during the course of any previous retries and try
to return that result.

12 months agoRe-enable skipped ConfigTest elixir tests (#3564)
Bessenyei Balázs Donát [Sat, 15 May 2021 17:06:58 +0000 (19:06 +0200)] 
Re-enable skipped ConfigTest elixir tests (#3564)

12 months agoAdd error propagation to couch_eval and couch_views_indexer (#3553)
Bessenyei Balázs Donát [Sat, 15 May 2021 07:08:18 +0000 (09:08 +0200)] 
Add error propagation to couch_eval and couch_views_indexer (#3553)

When ApiMod:acquire_map_context returns with {error, any()}, the caller(s) fail with badmatch. This commit changes the caller hierarchy so that those {error, _} are propagated back and a failing update gets retried (up to retry_limit times).

12 months agoErlang 24 support
Nick Vatamaniuc [Thu, 13 May 2021 06:21:36 +0000 (02:21 -0400)] 
Erlang 24 support

Switching crypto functions to use the new ones such as:

crypto:hmac(Alg, Key, Message) -> crypto:mac(hmac, Alg, Key, Message)

To simplify Erlang 24 support, in which some crypto functions have
been removed, bump the minimum version to 22.

Other fixes were in dependencies:

  * Bumped meck to 0.9.2. New meck from upstream supports Erlang
    24. Also required bumping folsom since it depends on meck

  * Example in passage module would not compile, so commented out
  the parse transform. Required bumping jaeger passage since it
  depends on passage

12 months agoHandle disabled retries in restart_tx logic
Nick Vatamaniuc [Tue, 11 May 2021 16:22:59 +0000 (12:22 -0400)] 
Handle disabled retries in restart_tx logic

During buggify runs we disable max tx retries by setting it to -1. That's FDB's
documented way to of doing it. However, when we re-use that setting to handle
restart_tx logic we don't account for -1, so we that's what this PR fixes.

12 months agoIntroduce buggify integration test mode
Nick Vatamaniuc [Mon, 10 May 2021 05:52:05 +0000 (01:52 -0400)] 
Introduce buggify integration test mode

Add `buggify-elixir-suite` target to run Elixir integration tests
under FoundationDB's client buggify mode [1]. In this mode, the FDB C
client in the `erlfdb` application will periodically throw mostly
retryable errors (`1009`, `1007`, etc). Transaction closures should
properly handle retryable errors without side-effects such as
re-sending response data to the user more than once or, attempt to
re-read data from the socket after it was already read once.

In order to avoid false positives, provide a custom .ini settings file
which disables transaction timeouts (`1031` errors). Those are not
retryable by default, as far as the `on_error` callback is
concerned. Ff we do have timeouts set ( = 60000), it signals the
FoundationDB client that we expect to handle timeouts in buggify mode,
so it starts throwing them [2]. Since we don't handle those everywhere
we get quite a few false positive errors.

Buggify settings I believe are the default -- 25% chance to activate
an error, and 25% chance of firing the error when the code passes over
that section. In most test runs this should result in a pass, but
sometimes, due to lingering bugs, there will be timeouts, 409
conflicts and other failures so we cannot yet turn this into a
reliable integration test step.



12 months agoBump ibrowse to 4.4.2 + couchdb patches
Nick Vatamaniuc [Fri, 7 May 2021 19:47:36 +0000 (15:47 -0400)] 
Bump ibrowse to 4.4.2 + couchdb patches

Set the `worker_trap_exits = false` setting to ensure our replication worker
pool properly cleans up worker processes.