james-project.git
3 days ago[BUILD] Increase ConcurrentTestRunnerTest await time master
Benoit Tellier [Wed, 11 May 2022 09:21:49 +0000 (16:21 +0700)] 
[BUILD] Increase ConcurrentTestRunnerTest await time

Locally 50ms was borderline with random failures, so
I expect the default value of 100 ms to be too low for
being reliable on a busy CI server where context switch
might be made expensive with expensive concurrent builds.

I hereby propose a safer, higher value of 2 seconds.

3 days agoJAMES-3719 Reading inputstream might be blocking
Benoit Tellier [Wed, 11 May 2022 08:15:58 +0000 (15:15 +0700)] 
JAMES-3719 Reading inputstream might be blocking

Subscribes on an elastic scheduler when blocking reads
might be performed.

3 days agoJAMES-3719 Avoid copying mime part content when we are not to extract its content
Benoit Tellier [Wed, 2 Mar 2022 07:45:44 +0000 (14:45 +0700)] 
JAMES-3719 Avoid copying mime part content when we are not to extract its content

3 days agoJAMES-3719 Reactive textual content extraction with Apache Tika
Benoit Tellier [Wed, 2 Mar 2022 07:31:25 +0000 (14:31 +0700)] 
JAMES-3719 Reactive textual content extraction with Apache Tika

Tika was called from reactive code and was doing blocking HTTP calls from within
the MIME parsing code.

This generate:
 - An unneeded thread consumption as we have some threads waiting for Tika
   response
 - Potentially dangerous blocking calls: for instance the InVM event bus was
  doing such calls on the parallel thread pool (where it is critical NOT to
  block)...
 - Also the connection was opened on a per-call basis, not being reused.

 We introduce the following changes:
  - Reactification of the TextExtractor API
  - We re-implement the HTTP calls done by TikaTextExtractor with reactor-netty
  which allows us to pool HTTP connections and do this in a non-blocking
  reactive fashion.
  - We provide a reactive cache using the caffeine caching library - Guava
  caches are blocking thus not an option...
  - We uncouple the text extraction from the MIME parsing phase by introducing
  an intermediate POJO. Doing so requires us to do a post-parsing copy of
  content.

 TODO only do the copy if necessary. We don't want to copy large attachments for whom no text is going to be extracted...

  - Finally we reactify index content generation for ElasticSearch code.

4 days ago[FIX] Filter non error logs as done previously
Benoit Tellier [Wed, 11 May 2022 07:15:43 +0000 (14:15 +0700)] 
[FIX] Filter non error logs as done previously

Mistakenly commited change need to be reverted

4 days agoJAMES-3737 ImapRequestFrameDecoder: remove buggy code
Benoit Tellier [Wed, 11 May 2022 07:13:40 +0000 (14:13 +0700)] 
JAMES-3737 ImapRequestFrameDecoder: remove buggy code

Unneeded, the cumulation cary over when disabling framing do not need it.

Buggy: next handler do not expect a ByteBuf

java.lang.ClassCastException: class io.netty.buffer.PooledDuplicatedByteBuf cannot be cast to class org.apache.james.imap.api.ImapMessage (io.netty.buffer.PooledDuplicatedByteBuf and org.apache.james.imap.api.ImapMessage are in unnamed module of loader 'app')

4 days agoJAMES-3737 IMAPServerTest: test cumulation upon framing
Benoit Tellier [Wed, 11 May 2022 07:12:12 +0000 (14:12 +0700)] 
JAMES-3737 IMAPServerTest: test cumulation upon framing

Demonstrate that the buffer is not lost

4 days agoJAMES-3737 ImapRequestFrameDecoder: better manage framing
Benoit Tellier [Wed, 11 May 2022 05:52:52 +0000 (12:52 +0700)] 
JAMES-3737 ImapRequestFrameDecoder: better manage framing

4 days agoJAMES-3737 IMAPServerTest: tests for non synchronized litterals
Benoit Tellier [Wed, 11 May 2022 03:43:29 +0000 (10:43 +0700)] 
JAMES-3737 IMAPServerTest: tests for non synchronized litterals

5 days agoJAMES-3763 BlobStore backed MailRepository
Matthieu Baechler [Tue, 10 May 2022 12:21:57 +0000 (14:21 +0200)] 
JAMES-3763 BlobStore backed MailRepository

Co-authored-by: Jean Helou <jhe@codamens.fr>
5 days agoAdd Baechler Craftsmanship as a professional support company
Matthieu Baechler [Tue, 10 May 2022 08:03:04 +0000 (10:03 +0200)] 
Add Baechler Craftsmanship as a professional support company

6 days agoJAMES-3762 Reworks MailRepository contract to use MailKeys instead of Mails
Jean Helou [Wed, 20 Apr 2022 20:23:23 +0000 (22:23 +0200)] 
JAMES-3762 Reworks MailRepository contract to use MailKeys instead of Mails

Co-Authored-By: Matthieu Baechler <matthieu@apache.org>
7 days agoJAMES-3534 Avoid blocking calls in Identity/set
Benoit Tellier [Fri, 6 May 2022 08:31:26 +0000 (15:31 +0700)] 
JAMES-3534 Avoid blocking calls in Identity/set

7 days ago[REFACTORING] Use CHarMatcher in FetchCommandParser
Benoit Tellier [Fri, 6 May 2022 04:38:07 +0000 (11:38 +0700)] 
[REFACTORING] Use CHarMatcher in FetchCommandParser

7 days ago[REFACTORING] Reaorder declarations in ImapRequestLineReader
Benoit Tellier [Fri, 6 May 2022 04:21:22 +0000 (11:21 +0700)] 
[REFACTORING] Reaorder declarations in ImapRequestLineReader

 - Nested class/interfaces first
 - Then constant
 - Then static methods
 - Then fields
 - Then abstract methods
 - Then regular methods

7 days ago[PERF] Decode UTF-7 only if needed
Benoit Tellier [Fri, 6 May 2022 04:10:19 +0000 (11:10 +0700)] 
[PERF] Decode UTF-7 only if needed

UTF-7 decoding is only needed if some characters
are UTF-7 encoded, which happens if the switching
character `&` is present.

If absent we can skip the UTF-7 decoding step,
which saves ~10% of IMAP command parsing allocation
(string copy and coder charBuffer allocation) and
~6% of IMAP command parsing CPU time.

7 days ago[PERF] IMAP avoid memory allocation when parsing STATUS items
Benoit Tellier [Fri, 6 May 2022 03:00:50 +0000 (10:00 +0700)] 
[PERF] IMAP avoid memory allocation when parsing STATUS items

This represents ~10% of IMAP command decoding memory consumption.

7 days ago[PERF] IMAP improve status items parsing
Benoit Tellier [Fri, 6 May 2022 02:27:29 +0000 (09:27 +0700)] 
[PERF] IMAP improve status items parsing

Better handle closing bracket not to call substring.

7 days ago[PERF] IMAP improve flags parsing
Benoit Tellier [Fri, 6 May 2022 02:27:06 +0000 (09:27 +0700)] 
[PERF] IMAP improve flags parsing

Better handle closing bracket not to call substring.

7 days ago[PERF] IMAP use constants for CharValidator where immutable
Benoit Tellier [Fri, 6 May 2022 02:26:31 +0000 (09:26 +0700)] 
[PERF] IMAP use constants for CharValidator where immutable

This represents 3% of IMAP parsing memory
consumption. Easy win...

7 days agoJAMES-3737 Backport resetReaderIndex fix
Benoit Tellier [Thu, 5 May 2022 11:27:41 +0000 (18:27 +0700)] 
JAMES-3737 Backport resetReaderIndex fix

7 days agoJAMES-3737 AuthenticateProcessor should disable reads when adding a line handler
Benoit Tellier [Thu, 5 May 2022 00:03:01 +0000 (07:03 +0700)] 
JAMES-3737 AuthenticateProcessor should disable reads when adding a line handler

7 days agoJAMES-3737 Avoid a race condition upon STARTTLS
Benoit Tellier [Tue, 3 May 2022 02:24:12 +0000 (09:24 +0700)] 
JAMES-3737 Avoid a race condition upon STARTTLS

STARTTLS response and activation was sent from
the event loop causing them to be sent and
activated prior earlier responses. This causes
a race condition when several commands are
batched.

7 days agoJAMES-3737 Avoid a race condition upon compress
Benoit Tellier [Tue, 3 May 2022 02:04:06 +0000 (09:04 +0700)] 
JAMES-3737 Avoid a race condition upon compress

Compress response and activation was sent from
the event loop causing them to be sent and
activated prior earlier responses. This causes
a race condition when several commands are
batched.

7 days agoJAMES-3737 StartTLS and compress no longer need a distinct writer
Benoit Tellier [Fri, 22 Apr 2022 13:12:49 +0000 (20:12 +0700)] 
JAMES-3737 StartTLS and compress no longer need a distinct writer

They run on an eventLoop thus are immediately applied

7 days agoJAMES-3737 Execute IMAP requests on eventLoop
Benoit Tellier [Mon, 18 Apr 2022 14:12:26 +0000 (21:12 +0700)] 
JAMES-3737 Execute IMAP requests on eventLoop

Avoids context switches on each IMAP requests

7 days agoJAMES-3737 Copy large APPEND to a file asynchronously
Benoit Tellier [Fri, 22 Apr 2022 04:23:21 +0000 (11:23 +0700)] 
JAMES-3737 Copy large APPEND to a file asynchronously

This enable executing all IMAP requests on the event loop

7 days agoJAMES-3737 Extract methods in ImapRequestFrameDecoder: small enhancements
Benoit Tellier [Fri, 22 Apr 2022 03:20:44 +0000 (10:20 +0700)] 
JAMES-3737 Extract methods in ImapRequestFrameDecoder: small enhancements

 - Reduce counts of parameters
 -Also avoid a double map lookup

7 days agoJAMES-3737 Extract methods in ImapRequestFrameDecoder
Benoit Tellier [Fri, 22 Apr 2022 03:09:53 +0000 (10:09 +0700)] 
JAMES-3737 Extract methods in ImapRequestFrameDecoder

7 days agoJAMES-3747 Reactive implementation for RabbitMQ channelPool (#979)
Benoit TELLIER [Mon, 9 May 2022 01:35:14 +0000 (08:35 +0700)] 
JAMES-3747 Reactive implementation for RabbitMQ channelPool (#979)

7 days ago[REFACTORING] Use Splitter::splitToStream (#989)
Benoit TELLIER [Mon, 9 May 2022 01:35:02 +0000 (08:35 +0700)] 
[REFACTORING] Use Splitter::splitToStream (#989)

Avoids an intermediate collection and a second streaming.

7 days agoJAMES-3744 Fixed URI matcher should be more flexible (#991)
Trần Hồng Quân [Mon, 9 May 2022 01:34:46 +0000 (08:34 +0700)] 
JAMES-3744 Fixed URI matcher should be more flexible (#991)

This allows matching routes with query param

10 days agoJAMES-3761 Adds configurable bucketname to MimeMessageStore (#985)
Matthieu Baechler [Fri, 6 May 2022 01:03:59 +0000 (03:03 +0200)] 
JAMES-3761 Adds configurable bucketname to MimeMessageStore (#985)

11 days ago[UPGRADE] OIDC example JWT revoker jwt-go 3.2.0 -> 3.2.1
Benoit Tellier [Wed, 4 May 2022 03:48:33 +0000 (10:48 +0700)] 
[UPGRADE] OIDC example JWT revoker jwt-go 3.2.0 -> 3.2.1

https://github.com/chibenwa/james-project/security/dependabot/1

jwt-go allows attackers to bypass intended access restrictions in situations with []string{} for m["aud"] (which is allowed by the specification). Because the type assertion fails, "" is the value of aud. This is a security problem if the JWT token is presented to a service that lacks its own audience check. There is no patch available and users of jwt-go are advised to migrate to golang-jwt at version 3.2.1

11 days ago[UPGRADE] jackson 2.13.1 -> 2.13.2.2 fixes CVE-2020-36518
Benoit Tellier [Wed, 4 May 2022 03:47:07 +0000 (10:47 +0700)] 
[UPGRADE] jackson 2.13.1 -> 2.13.2.2 fixes CVE-2020-36518

https://security.snyk.io/vuln/SNYK-JAVA-COMFASTERXMLJACKSONCORE-2421244

com.fasterxml.jackson.core:jackson-databind is a library which contains the general-purpose data-binding functionality and tree-model for Jackson Data Processor.

Affected versions of this package are vulnerable to Denial of Service (DoS) via a large depth of nested objects.

11 days agoJAMES-1489 Fix decoding for complex UTF-8 searches (#980)
Benoit TELLIER [Thu, 5 May 2022 02:54:21 +0000 (09:54 +0700)] 
JAMES-1489 Fix decoding for complex UTF-8 searches (#980)

 - Reader index was  not properly reset for
multiple subsequent continuations.
 - Charset was not carried over for all the
 terms of the search.

11 days ago[BUILD] Await FakeSmtp HTTP assertions
Benoit Tellier [Wed, 4 May 2022 05:54:04 +0000 (12:54 +0700)] 
[BUILD] Await FakeSmtp HTTP assertions

The container might not be fully started as the
wait strategy waits for any port to be
responsive.

An await will add the desired test stability.

11 days ago[REFACTORING] Drop FakeSmtpExtension in favor of FakeSmtp
Benoit Tellier [Wed, 4 May 2022 05:52:39 +0000 (12:52 +0700)] 
[REFACTORING] Drop FakeSmtpExtension in favor of FakeSmtp

Strictly duplicated code.

12 days ago[PERF] Cleanup ElasticSearchListeningMessageSearchIndex
Benoit TELLIER [Thu, 28 Apr 2022 07:20:30 +0000 (14:20 +0700)] 
[PERF] Cleanup ElasticSearchListeningMessageSearchIndex

12 days ago[PERF] Constant for Locales DEFAULT
Benoit TELLIER [Thu, 28 Apr 2022 04:42:45 +0000 (11:42 +0700)] 
[PERF] Constant for Locales DEFAULT

12 days ago[PERF] PopulateEmailQueryViewListener only needs to parse headers
Benoit TELLIER [Thu, 28 Apr 2022 04:06:52 +0000 (11:06 +0700)] 
[PERF] PopulateEmailQueryViewListener only needs to parse headers

12 days ago[PERF] ElasticSearchListeningMessageSearchIndex Avoid uneeded Optional boxing
Benoit TELLIER [Thu, 28 Apr 2022 04:06:34 +0000 (11:06 +0700)] 
[PERF] ElasticSearchListeningMessageSearchIndex Avoid uneeded Optional boxing

The boxing represent 0.6% of memory allocation (in bytes) and a very large
count of objects that impact GC. THis can be avoided by inlining the
Flux::handle.

12 days ago[PERF] MessageManager::getMetaData should allow not to read recent
Benoit TELLIER [Thu, 28 Apr 2022 03:59:37 +0000 (10:59 +0700)] 
[PERF] MessageManager::getMetaData should allow not to read recent

IMAP STATUS command is very common and most of the time do not need to
fetch recent items. Always doing so result in unneeded Cassandra queries.

12 days ago[PERF] Avoid reparsing ContentType
Benoit TELLIER [Thu, 28 Apr 2022 03:56:58 +0000 (10:56 +0700)] 
[PERF] Avoid reparsing ContentType

Such a parsing takes up to 1% of memory allocation.

Also the use of lenient parser helps.

12 days ago[PERF] IMAP: limit memory allocation for command name responses
Benoit TELLIER [Tue, 26 Apr 2022 03:44:58 +0000 (10:44 +0700)] 
[PERF] IMAP: limit memory allocation for command name responses

12 days ago[PERF] IMAP CAPABILITY: Limit memory allocation upon responses
Benoit TELLIER [Tue, 26 Apr 2022 03:36:54 +0000 (10:36 +0700)] 
[PERF] IMAP CAPABILITY: Limit memory allocation upon responses

12 days ago[PERF] IMAP STATUS: Fasten MDC
Benoit TELLIER [Tue, 26 Apr 2022 03:34:03 +0000 (10:34 +0700)] 
[PERF] IMAP STATUS: Fasten MDC

12 days ago[PERF] IMAP SEARCH: Reduce overhead for large responses
Loan [Tue, 26 Apr 2022 03:27:04 +0000 (10:27 +0700)] 
[PERF] IMAP SEARCH: Reduce overhead for large responses

 - Limit Long boxing impact
 - Avoid unneeded array copy
 - Limit stream usage (local use, large overhead)

12 days ago[PERF] Avoid parsing needlessly most SearchHit fields
Loan [Tue, 26 Apr 2022 03:18:39 +0000 (10:18 +0700)] 
[PERF] Avoid parsing needlessly most SearchHit fields

12 days ago[PERF] IMAP FETCH: Fasten flags writing
Loan [Tue, 26 Apr 2022 03:28:13 +0000 (10:28 +0700)] 
[PERF] IMAP FETCH: Fasten flags writing

12 days agoJAMES-3737 Allow configuring ElasticSearch connections
Benoit Tellier [Mon, 18 Apr 2022 10:08:12 +0000 (17:08 +0700)] 
JAMES-3737 Allow configuring ElasticSearch connections

To quote the driver itself: default settings for connection
pooling may be too constraining. These limits might be hit
with a few concurrent ElasticSearch request. Exposing these setting is thus interesting.

12 days agoJAMES-3737 Document thread tuning for protocols
Benoit Tellier [Mon, 18 Apr 2022 09:37:34 +0000 (16:37 +0700)] 
JAMES-3737 Document thread tuning for protocols

12 days agoJAMES-3737 Allow configuring boss thread count for protocols
Benoit Tellier [Mon, 18 Apr 2022 09:37:09 +0000 (16:37 +0700)] 
JAMES-3737 Allow configuring boss thread count for protocols

These threads are used for accepting incoming connections.

 - Ability to specify the thread count
 - Ability to instead rely on IO threads

2 weeks agoJAMES-3752 Allow disabling ImapChannelUpstreamHandler heartbeat handler
Tung Van TRAN [Mon, 18 Apr 2022 09:27:34 +0000 (16:27 +0700)] 
JAMES-3752 Allow disabling ImapChannelUpstreamHandler heartbeat handler

2 weeks agoJAMES-3752 - Enable ImapHeartbeatHandler by setting idle time interval
Tung Van TRAN [Mon, 18 Apr 2022 09:32:04 +0000 (16:32 +0700)] 
JAMES-3752 - Enable ImapHeartbeatHandler by setting idle time interval

2 weeks agoJAMES-3753 Rewrite FlowedMessageUtils.flow()
cketti [Fri, 22 Apr 2022 23:05:55 +0000 (01:05 +0200)] 
JAMES-3753 Rewrite FlowedMessageUtils.flow()

This implementation avoids creating temporary strings and fixes the bugs identified via fuzzing.

2 weeks agoJAMES-3753 Add tests for FlowedMessageUtils.flow()
cketti [Fri, 22 Apr 2022 19:40:05 +0000 (21:40 +0200)] 
JAMES-3753 Add tests for FlowedMessageUtils.flow()

2 weeks ago[PERF] Rely on Deque::peekFirst rather than Iterables::getFirst
Benoit Tellier [Thu, 21 Apr 2022 16:06:46 +0000 (23:06 +0700)] 
[PERF] Rely on Deque::peekFirst rather than Iterables::getFirst

2 weeks ago[PERF] Avoid String => bytes conversions for IMAP constants
Benoit Tellier [Thu, 21 Apr 2022 10:37:45 +0000 (17:37 +0700)] 
[PERF] Avoid String => bytes conversions for IMAP constants

2 weeks agoJAMES-3737 IMAP FETCH: More contextual logs
Benoit Tellier [Thu, 21 Apr 2022 07:47:57 +0000 (14:47 +0700)] 
JAMES-3737 IMAP FETCH: More contextual logs

2 weeks agoJAMES-3737 Fetch should not block when using QResync
Benoit Tellier [Thu, 21 Apr 2022 07:47:33 +0000 (14:47 +0700)] 
JAMES-3737 Fetch should not block when using QResync

2 weeks ago[PERF] getSelectedMailboxReactive: avoid duplicating emptiness management
Benoit Tellier [Thu, 21 Apr 2022 07:33:01 +0000 (14:33 +0700)] 
[PERF] getSelectedMailboxReactive: avoid duplicating emptiness management

2 weeks ago[REFACTORING] CassandraMailboxMapper: remove duplicated code
Benoit Tellier [Thu, 21 Apr 2022 06:35:42 +0000 (13:35 +0700)] 
[REFACTORING] CassandraMailboxMapper: remove duplicated code

2 weeks ago[PERF] Remove an uneeded `distinct` call
Benoit Tellier [Thu, 21 Apr 2022 06:22:15 +0000 (13:22 +0700)] 
[PERF] Remove an uneeded `distinct` call

The underlying DAO does not return duplicates.

2 weeks ago[PERF] Allow using MailboxManager::startProcessing/stopProcessing only for implementa...
Benoit Tellier [Wed, 20 Apr 2022 15:47:48 +0000 (22:47 +0700)] 
[PERF] Allow using MailboxManager::startProcessing/stopProcessing only for implementations that needs it

 - Allow reuse of reactive wrapping of such calls
 - Allow simplifying reactor pipeline

2 weeks ago[PERF] ElasticSearchListeningMessageSearchIndex: simplify documentId
Benoit Tellier [Tue, 19 Apr 2022 16:25:03 +0000 (23:25 +0700)] 
[PERF] ElasticSearchListeningMessageSearchIndex: simplify documentId

Do a string concatenation rather than a join

2 weeks ago[PERF] Improve ElasticSearchListeningMessageSearchIndex log
Benoit Tellier [Tue, 19 Apr 2022 16:24:05 +0000 (23:24 +0700)] 
[PERF] Improve ElasticSearchListeningMessageSearchIndex log

 - Avoid useless "MessageUid" marker
 - Serialize mailbox id

2 weeks ago[PERF] Fasten StoreRightManager::isReadWrite
Benoit Tellier [Tue, 19 Apr 2022 04:57:55 +0000 (11:57 +0700)] 
[PERF] Fasten StoreRightManager::isReadWrite

This avoids a right resolution for mailbox owner.

2 weeks ago[PERF] ListingEncodingUtils::getNameAttributes should avoid intermediate builds
Benoit Tellier [Tue, 19 Apr 2022 04:56:00 +0000 (11:56 +0700)] 
[PERF] ListingEncodingUtils::getNameAttributes should avoid intermediate builds

2 weeks ago[PERF] IMAP should allow sending char response
Benoit Tellier [Tue, 19 Apr 2022 04:54:43 +0000 (11:54 +0700)] 
[PERF] IMAP should allow sending char response

This avoids allocating a string for this and also improves
the quoting algorithm.

2 weeks ago[PERF] Improve, and rely on MailboxPath::getParents
Benoit Tellier [Tue, 19 Apr 2022 04:52:38 +0000 (11:52 +0700)] 
[PERF] Improve, and rely on MailboxPath::getParents

This avoids logic to filter out the current entry...

2 weeks ago[PERF] StoreMailboxManager: Get rid of distinct calls on mailbox
Benoit Tellier [Tue, 19 Apr 2022 04:47:22 +0000 (11:47 +0700)] 
[PERF] StoreMailboxManager: Get rid of distinct calls on mailbox

Was taking 0.36% of CPU.

We don't need this, if we ensure that no user mailboxes are
returned as part of delegated mailbox call.

2 weeks ago[PERF] Fasten Mailbox::hashCode
Benoit Tellier [Tue, 19 Apr 2022 04:44:20 +0000 (11:44 +0700)] 
[PERF] Fasten Mailbox::hashCode

Sub-fields should hash their single field directly.

Note that this is used to enforce mailbox unicity in responses.

2 weeks ago[PERF] JsonGenericSerializer: default behaviour for duplicated tree
Benoit Tellier [Mon, 18 Apr 2022 05:18:43 +0000 (12:18 +0700)] 
[PERF] JsonGenericSerializer: default behaviour for duplicated tree

Checking for duplicates is a rather expensive operation,
default behaviour is definitely acceptable.

2 weeks ago[PERF] Reactor RetrySpec can be a field
Benoit Tellier [Mon, 18 Apr 2022 05:16:15 +0000 (12:16 +0700)] 
[PERF] Reactor RetrySpec can be a field

This avoids instantiating it on each call.

2 weeks ago[PERF] Fasten Username and Domain core object parsing
Benoit Tellier [Sun, 17 Apr 2022 15:24:07 +0000 (22:24 +0700)] 
[PERF] Fasten Username and Domain core object parsing

 - Avoid allocation of substring
 - Avoid stream logic
 - Groups charmatcher checks.
 - Avoid needless operations in Username

This delivers a 5 fold performance enhancement on a code
path often called (event deserialization, JMAP auth,
 Cassandra mailbox queries). Memory allocation is also
 greatly reduced.

2 weeks ago[PERF] Fasten IMAP FETCH MDC context computation
Benoit Tellier [Mon, 18 Apr 2022 05:02:49 +0000 (12:02 +0700)] 
[PERF] Fasten IMAP FETCH MDC context computation

 - Avoid reflection calls to resolve class name
 - Offer shortcuts for single id ranges - this avoids
 the stream + reduce logic: building and executing such a
 pipeline can be expensive.
 - Ommit null values
 - Decrease the count of properties without dropping
 information

2 weeks ago[PERF] FetchData should optimize adding single item
Benoit Tellier [Mon, 18 Apr 2022 05:00:42 +0000 (12:00 +0700)] 
[PERF] FetchData should optimize adding single item

Varargs does a array copy and a full collection addition.
This takes 30% of FETCH command parsing (and 5% of its
allocated memory) and can easily be improved.

2 weeks agoAvoid unneeded port binding with ActiveMQ
Benoit TELLIER [Thu, 28 Apr 2022 08:06:25 +0000 (15:06 +0700)] 
Avoid unneeded port binding with ActiveMQ

We use VM protocol, binding port 61616 is unneeded, and might generate
needless port binding conflicts.

2 weeks agoRevert "JAMES-3747 Reactive implementation for RabbitMQ channelPool"
Benoit TELLIER [Tue, 26 Apr 2022 06:21:18 +0000 (13:21 +0700)] 
Revert "JAMES-3747 Reactive implementation for RabbitMQ channelPool"

This reverts commit 8564a2ecf72e0a7ef404fe491799fb8140dea1f6.

2 weeks agoJAMES-3751 Relax modseq conditions
Benoit Tellier [Mon, 18 Apr 2022 07:42:14 +0000 (14:42 +0700)] 
JAMES-3751 Relax modseq conditions

2 weeks agoJAMES-3751 IMAP Test for ESEARCH tag ALL
Benoit Tellier [Sun, 17 Apr 2022 04:52:21 +0000 (11:52 +0700)] 
JAMES-3751 IMAP Test for ESEARCH tag ALL

2 weeks agoJAMES-3751 IMAP SEARCH was ignoring MODSEQ for last range
Benoit Tellier [Sun, 17 Apr 2022 04:28:11 +0000 (11:28 +0700)] 
JAMES-3751 IMAP SEARCH was ignoring MODSEQ for last range

2 weeks agoJAMES-3751 Tests for RFC-4731 (min, max, count aggregations for IMAP SEARCH)
Benoit Tellier [Sun, 17 Apr 2022 04:26:20 +0000 (11:26 +0700)] 
JAMES-3751 Tests for RFC-4731 (min, max, count aggregations for IMAP SEARCH)

2 weeks agoJAMES-3751 Tests for RFC-5182 (referencing the last IMAP search)
Benoit Tellier [Sun, 17 Apr 2022 03:59:10 +0000 (10:59 +0700)] 
JAMES-3751 Tests for RFC-5182 (referencing the last IMAP search)

2 weeks ago[PERF] Avoid execution of MailboxAnnotationListener
Benoit Tellier [Sun, 17 Apr 2022 09:31:14 +0000 (16:31 +0700)] 
[PERF] Avoid execution of MailboxAnnotationListener

p50=0.062207, p75=0.117759, p95=0.358399, p98=0.602111, p99=0.925695

Not a lot to be honest, mostly the cost of the context
switch of the elastic subscription...

2 weeks ago[REACTOR] Avoid subscribing on elastic while we are blocking
Benoit Tellier [Sun, 17 Apr 2022 09:29:42 +0000 (16:29 +0700)] 
[REACTOR] Avoid subscribing on elastic while we are blocking

Current thread can perform those operations while
blocking so no need to mobilize one extra thread for
the task at hand.

3 weeks agoJAMES-3753 Fix FlowedMessageUtils.deflow() (#972)
cketti [Fri, 22 Apr 2022 05:03:30 +0000 (07:03 +0200)] 
JAMES-3753 Fix FlowedMessageUtils.deflow() (#972)

3 weeks agoJAMES-3740 Fix a typo in UidMsnConverter
Benoit Tellier [Fri, 8 Apr 2022 05:12:10 +0000 (12:12 +0700)] 
JAMES-3740 Fix a typo in UidMsnConverter

3 weeks agoJAMES-3740 Improve performance if UidMsnConverter is initially empty
Benoit Tellier [Wed, 30 Mar 2022 11:01:35 +0000 (18:01 +0700)] 
JAMES-3740 Improve performance if UidMsnConverter is initially empty

This is the nominal use case, in the absent of concurrent message operation on the
mailbox.

Sorting the resulting list 'in place' allows to avoid usage of tree like structures
that enforces a copy and require temporary allocations of entry.

This shows:
 - x2.5 performance enhancements
 - Dramatically reduces heap pressure upon UidMsnConverter::addAll

3 weeks agoJAMES-3740 Compact primitive collections for UID <-> MSN mapping
Benoit Tellier [Wed, 30 Mar 2022 03:24:13 +0000 (10:24 +0700)] 
JAMES-3740 Compact primitive collections for UID <-> MSN mapping

This reduces HEAP memory consumption of this use case by a factor 3.

3 weeks agoJAMES-3747 Fix a typo
Benoit Tellier [Mon, 18 Apr 2022 07:43:54 +0000 (14:43 +0700)] 
JAMES-3747 Fix a typo

3 weeks agoJAMES-3747 Fix badly renamed minDelay methods
Benoit Tellier [Fri, 15 Apr 2022 15:33:35 +0000 (22:33 +0700)] 
JAMES-3747 Fix badly renamed minDelay methods

Also increase wait ties as this no longer is the shorter acquire backoff retry but the full acquiring duration.

3 weeks agoJAMES-3749 Allow disable durability, publish confirms
Benoit Tellier [Fri, 15 Apr 2022 05:39:21 +0000 (12:39 +0700)] 
JAMES-3749 Allow disable durability, publish confirms

Doing so enable massive gains in throughtput thus some
operators might be interested in it...

3 weeks agoJAMES-3749 Shorten rabbitmq routing keys
Benoit Tellier [Thu, 14 Apr 2022 07:45:32 +0000 (14:45 +0700)] 
JAMES-3749 Shorten rabbitmq routing keys

3 weeks agoJAMES-3747 Metrics for RabbitMQ channelPool
Benoit Tellier [Wed, 13 Apr 2022 15:28:17 +0000 (22:28 +0700)] 
JAMES-3747 Metrics for RabbitMQ channelPool

 -> Time to acquire a channel
 -> Time to publish something (includes acquire)
 -> Time to bind/unbind exchanges (includes acquire)

3 weeks agoJAMES-3747 Reactive implementation for RabbitMQ channelPool
Benoit Tellier [Thu, 14 Apr 2022 04:29:33 +0000 (11:29 +0700)] 
JAMES-3747 Reactive implementation for RabbitMQ channelPool

3 weeks ago[DOCUMENTATION] Document implemented standards for Distributed server
Benoit Tellier [Sat, 16 Apr 2022 16:06:33 +0000 (23:06 +0700)] 
[DOCUMENTATION] Document implemented standards for Distributed server

3 weeks ago[DOCUMENTATION] Better document databases responsibilities for distributed James
Benoit Tellier [Sat, 16 Apr 2022 16:06:09 +0000 (23:06 +0700)] 
[DOCUMENTATION] Better document databases responsibilities for distributed James

3 weeks agoJAMES-3737 Don't encapsulate a Cassandra query in a lock
Benoit Tellier [Fri, 15 Apr 2022 16:38:58 +0000 (23:38 +0700)] 
JAMES-3737 Don't encapsulate a Cassandra query in a lock