Improve the handling of sync child jobs to ensure it behaves correctly
in all scenarios.
The sync service now uses a isolated context to avoid all the pipeline
stages shutting down before all the sync tasks have had the opportunity
to run their course.
The job waiter now immediately starts with a counter of 1 and waits
until all the child and the parent job finish before considering the
work to be finished.
Finally, we also handle the case where a sync job can't be queued
because the calling context has been cancelled.
Fix incorrect use of `sync.WaitGroup` use to wait on sync jobs that
fail. After calling `WaitGroup.Wait()` it is advised to call
`WaitGroup.Add` until the existing counter has reached 0.
The code has been updated with a different mechanism that achieves the
same behavior which was previously available.
When syncing an account, if the user creates a new address and then
changes it to be the default address in combined address mode we need
to update the connector maps so that the new primary address ID can be
found in that map.
Includes https://github.com/ProtonMail/go-proton-api/pull/130
When editing a draft created by Apple Mail on the web client and then
later sending the draft with Apple Mail, we need to delete the draft
ourselves, or it will remain in the Draft folder.
This patch makes sure that the deletion of said draft only occurs after
the message was successfully sent.
If a SMPT client keeps hammering bridge and triggers multiple successive
errors in quick succession, force that client to wait 20 seconds before
trying again.
If an IMAP client creates a new message as a reply/forward from an
existing draft, that draft will be deleted once the message has been
sent.
Other than not being the correct behavior, the original reason for which
this line of code was added (carried over from v2), seems to be no longer
necessary as in all tests, the message is correctly removed from the
drafts folder after sent.
If we update the address after determining the sender address is
different, we also need to refresh the identity state in order to use
the right encryption keys.
If the IMAP service happened to finish syncing and wanted to reset the
user event service at a time the latter was publishing an event a
deadlock would occur and the user would not receive any new messages.
This change puts the request to revert the event id in a separate
go-routine to avoid this situation from re-occurring. The operational
flow remains unchanged as the event service will only process this
request once the current set of events have been published.
* Ensure IMAP service sync cancel request waits until the sync has
completely cancelled rather than just signaling. It's possible that
due the context reset on `group.Cancel` that something may have not
have been bookmarked correctly in subsequent sync restarts.
* Handle connection lost/restored events in the services. Removes the
need to lock bridge users. Which could conflict with other ongoing
lock operations. Additionally, it ensure that if one service is
blocked it doesn't block the entire bridge.
* Revise access to bridge user locks.
After we detect that the user has suffered the GODT-3003 bug due the
vault corruption not ensuring that a previous sync state would be
erased, we patch the gluon db directly and then reset the sync state.
After the account is added, the sync is automatically triggered and the
account state fixes itself.
This issue is triggered due to the `Service.Close()` call after the
go-routine for the event service exists. It is possible that during this
period a recently added subscriber with `pendingOpAdd` gets cancelled
and closed.
However, the subscriber later also enqueues a `pendingOpRemove` which
gets processed again with a call in `user.eventService.Close()` leading
to the double close panic.
This patch simply removes the `s.Close()` from the service, and leaves
the cleanup to called externally from user.Close() or user.Logout().
This helps the export tool to deal with problems arising from message
assembly after everything has been successfully encrypted.
The original behavior is still available under `DecryptAndBuildRFC822`.
Incoming messages which arrive into labels we know during sync are now
presented to the IMAP clients.
We also allow messages to be deleted while syncing if deleted on other
clients.
Other operations such as moving, marking messages as read and label
operations need to be considered in a follow up patch as they are far
more complex.
The IMAP service no longer blocks the event loop from progressing. When
syncing the event loop continues as usual, allow other parts of the
bridge to continue updating.
Once the sync is complete, it resets the event id back to the last
handled id before sync started and then we play catch up.
* Only call `RemoveOnFail` on error. It was also called on success.
* Add some logging to track send times.
* Fix non-deterministic hash from `rfc8222.GetMessageHash`.
https://github.com/ProtonMail/gluon/pull/395
* Fix wrong context use for message downloads
* Fix delete of sync data failing due ErrNotFound
* Pre-allocate attachment data buffer before download
* Fix calculation of progress if message count is higher than total
Fix tracking of child jobs. The build stage splits the incoming work
even further, but this was not reflected in the wait group counter. This
also fixes an issue where the cache was cleared to late.
Add more debug info for analysis.
Refactor sync state interface in order to have persistent sync rate.
Implementation of the new sync service that interleaves syncing jobs for
all active users.
It also includes improvements to the message downloader. The download
will now auto rate limit the parallel workers based on the server
responses.
Additionally each of the stages is now tested in isolation to ensure the
behavior matches the expectations.
Finally, this patch does not replace the existing IMAP sync. A follow up
patch is necessary to integrate the IMAP bits into the interfaces
required by these changes.
