For every update sent to gluon wait and check the error code to see if
an error occurred.
Note: Updates can't be inspect on the call site as it can lead to
deadlocks.
Sub folders with more than 2 levels of depth (e.g.: Folders/first/second)
could not be created since we did not update the known label list we use
to validate the request.
Prevent infinite error loop in event parsing by not returning errors if
we already have or do not have a given address. This occurs since we
sync the latest state at Bridge startup but still receive the events
which contain these changes later.
Do not treat unknown label creation/deletion/update or deletion in Bridge
as an error as the Gluon cache still needs to receive these events to
correct its internal state.
When calculating the hash for the body to match against sent email to
avoid duplicate addition to the sent folder, do not decode the actual
contents of the body.
It is possible that certain attachments are not formed correctly but
can still accepted by the backend. Trimming spaces and \r characters is
enough to hash the message and match it later on.
This also speeds the process up as we no longer have to perform
encoding conversions.
During sync a user may continue to perform operations on the server it
is possible we run into a message which has a labelID we are not aware
of. To counter this we issue `CreateMessage` updates with
`IgnoreUnknownMailboxIDs` set to true. Eventually, after sync the state
will resolve itself with events.
Make sure that we are at least using 16 workers for sync, otherwise
multiply the current sync worker count by 2.
Finally, this patch also logs the duration of the time it takes to
transfer all the messages from the server.
Revise syncing work distribution. Sync time can be reduced by up to 50%.
Rework the sync so that it pipelines better with bigger batch counts at
each stage. We now use 3 separate stages: Download, Updates and Sync.
The Download stage downloads messages in maxBatchSize intervals using
1.5x syncWorkers. Once the current batch has finished downloading it's
forwarded to the Updates stage and we proceed to download the next
batch.
The Update stage converts everything into gluon updates and prepares a
collection of noops that the sync stage can wait on for termination.
Finally the sync stage waits until the updates have been applied in
Gluon so that the vault information can be updated. We allow up to 4
pending wait operations to be queued currently to not block the
pipeline.
When we send a message, the send recorder records the sent message.
When the client then appends an identical message to the sent folder,
the deduplication works and instead returns the message ID of the
existing proton message, rather than creating a new message. Gluon is
expected to notice that it already has this message ID and perform
some deduplication stuff internally.
However, it can happen that gluon doesn't yet have this message ID,
because we haven't yet received the "Message Created" event from the
API. To prevent this, we poll the events after send and wait for all
new events to be applied.
There's still a chance that the event wasn't generated yet on the API
side. Not sure what we can do about this.
