Commit Graph

62 Commits

Author SHA1 Message Date
4c24c004db fix(GODT-2371): Continue, not return, when handling draft 2023-02-16 17:23:22 +01:00
31dce41276 test: Fix race condition with initialization of messageIDs
Need to make sure the messageIDs slice is created properly before
it is used async in a different goroutine.
2023-02-07 18:03:36 +01:00
f3c5e300cd fix(GODT-2327): Delay event processing until gluon user exists
We don't want to start processing events until those events have
somewhere to be sent to.

Also, to be safe, ensure remove and re-add the gluon user while
clearing its sync status. This shouldn't be necessary.

fix(GODT-2327): Only start processing events once sync is finished

fix(GODT-2327): avoid windows delete all deadlock

fix(GODT-2327): Clear update channels whenever clearing sync status

fix(GODT-2327): Properly cancel event stream when handling refresh

fix(GODT-2327): Remove unnecessary sync abort call

fix(GODT-2327): Fix lint issue

fix(GODT-2327): Don't retry with abortable context because it's canceled

fix(GODT-2327): Loop to retry until sync has complete

fix(GODT-2327): Better sleep (with context)
2023-02-07 13:41:16 +01:00
34cd611a8b chore: Disable funlen linter 2023-02-06 14:29:13 +00:00
c8065c8092 GODT-2312: used space is properly updated. 2023-01-31 07:22:03 +00:00
e464e11ab9 GODT-2224: Refactor bridge sync to use less memory
Updates go-proton-api and Gluon to includes memory reduction changes and
modify the sync process to take into account how much memory is used
during the sync stage.

The sync process now has an extra stage which first download the message
metada to ensure that we only download up to `syncMaxDownloadRequesMem`
messages or 250 messages total. This allows for scaling the download
request automatically to accommodate many small or few very large
messages.

The IDs are then sent to a download go-routine which downloads the
message and its attachments. The result is then forwarded to another
go-routine which builds the actual message. This stage tries to ensure
that we don't use more than `syncMaxMessageBuildingMem` to build these
messages.

Finally the result is sent to a last go-routine which applies the
changes to Gluon and waits for them to be completed.

The new process is currently limited to 2GB. Dynamic scaling will be
implemented in a follow up. For systems with less than 2GB of memory we
limit the values to a set of values that is known to work.
2023-01-30 15:05:43 +01:00
160489a771 GODT-2223: Testing event processing scenarios. 2023-01-25 13:05:04 +01:00
94703bcf37 GODT-2223: Treat label/message deletion errors as non-critical 2023-01-23 21:25:15 +01:00
59745e6fb6 GODT-2223: Ensure apiLabels map is properly up to date 2023-01-18 14:10:23 +00:00
e35f3b6056 GODT-2223: Fix handling of create/update label events 2023-01-18 14:10:23 +00:00
d68014ec7b GODT-2223: Handle attempting to fetch a message that was just deleted 2023-01-18 14:10:23 +00:00
b63029054d GODT-2223: Fix handling deletion of nonexistent labels 2023-01-18 14:10:23 +00:00
849c8bee78 GODT-2223: Handle bad events by logging user out 2023-01-18 14:10:23 +00:00
93c7552a41 GODT-2202: Report update errors from Gluon
For every update sent to gluon wait and check the error code to see if
an error occurred.

Note: Updates can't be inspect on the call site as it can lead to
deadlocks.
2023-01-13 15:54:31 +01:00
931ed119bb GODT-2226: Fix moving drafts to trash
Only handle draft updates if the event was a message update. Also
includes Gluon update.
2023-01-12 14:25:27 +01:00
6d0a128111 Other: Update copyright year 2023-01-02 11:09:11 +01:00
038b5d1437 GODT-2222: Dot not error on unknown Address Events
Prevent infinite error loop in event parsing by not returning errors if
we already have or do not have a given address. This occurs since we
sync the latest state at Bridge startup but still receive the events
which contain these changes later.
2022-12-21 10:16:02 +01:00
08125e9281 Merge branch 'release/perth_narrows' into devel 2022-12-20 09:00:26 +01:00
47ea4b226a Other: Add sentry reports for event processing failures 2022-12-19 14:38:01 +01:00
00059e6754 Other: Do not fail on label events
Do not treat unknown label creation/deletion/update or deletion in Bridge
as an error as the Gluon cache still needs to receive these events to
correct its internal state.
2022-12-19 14:24:12 +01:00
4b3d4690e8 GODT-2196: Do not generate message updates for unknown labels
During sync a user may continue to perform operations on the server it
is possible we run into a message which has a labelID we are not aware
of. To counter this we issue `CreateMessage` updates with
`IgnoreUnknownMailboxIDs` set to true. Eventually, after sync the state
will resolve itself with events.
2022-12-15 09:37:22 +01:00
6c30a04ac0 Merge branch 'release/perth_narrows' into devel 2022-12-07 09:26:59 +01:00
75c88eaa55 GODT-2187: Handle unbuildable messages in event loop 2022-12-06 19:27:55 +01:00
bd6ae2ac2b GODT-2187: Placeholder for unbuildable messages 2022-12-06 16:35:32 +01:00
58d04f9693 GODT-2187: Skip messages during sync that fail to build/parse 2022-12-06 14:07:13 +00:00
8408a5fdc0 GODT-2170: Improving test server behaviour. 2022-12-02 13:27:19 +00:00
828fe0e86e GODT-2170: Update draft event means delete old and create new message. 2022-12-02 13:27:19 +00:00
71ad1e9939 Other: Only send to necessary update channel 2022-11-30 13:52:42 +00:00
e601245f01 Other(chore): Bump major version to v3 2022-11-23 16:08:27 +01:00
ad1fb47b0d Other: Switch from liteapi to go-proton-api 2022-11-23 15:17:56 +01:00
eb62056755 GODT-2119: Only show supported label IDs to clients 2022-11-21 17:15:51 +00:00
c22037462e Other: Only update event ID in vault once all gluon updates were applied 2022-11-21 09:12:52 +00:00
1e48ab4b9c GODT-2047: Clear last event ID when clearing sync status 2022-11-17 13:18:47 +00:00
ae87d7b236 GODT-1913: pass reporter to gluon, limit restarts, add crash handlers. 2022-11-16 13:48:31 +01:00
59278913ca GODT-2037: Handle and log API refresh event 2022-11-16 13:48:31 +01:00
48cf89b1a6 Other: Use cached labels for label sync rather than refetch 2022-11-16 13:48:31 +01:00
28d78453b7 GODT-2029: Handle deadlock when reordering user addresses 2022-11-16 13:48:31 +01:00
1e29a5210f GODT-1954: Draft message support
Add special case handling for draft messages so that if a Draft is
updated via an event it is correctly updated on the IMAP client via a
the new `imap.MessageUpdated event`.

This patch also updates Gluon to the latest version.
2022-11-16 13:48:30 +01:00
c548ba85fe Other: Add more extensive logging 2022-11-16 13:48:30 +01:00
8f420d728c GODT-1984: Handle permanent message deletion
Only delete messages when unlabeled from trash/spam if they only exists
in All Mail and (spam or trash).

This patch also ports delete_from_trash.feature and use status rather
than fetch to count messages in a mailboxes.
2022-11-16 13:48:30 +01:00
e4c08be28e Other: Don't remove/add gluon user when in combined mode
This used to be necessary because the connectors didn't have access
to the user's emails, but now they do, so we no longer need to do this.
2022-11-16 13:48:30 +01:00
d4da325e57 Other(refactor): Sort safe.Mutex types before locking to prevent deadlocks
This change implements safe.Mutex and safe.RWMutex, which wrap the
sync.Mutex and sync.RWMutex types and are assigned a globally unique
integer ID. The safe.Lock and safe.RLock methods sort the mutexes
by this integer ID before locking to ensure that locks for a given
set of mutexes are always performed in the same order, avoiding
deadlocks.
2022-11-16 13:48:30 +01:00
fd80848fcd Other(refactor): Use normal value + mutex for user.updateCh 2022-11-16 13:48:30 +01:00
cab5ee6752 Other(refactor): Use normal value + mutex for user.apiLabels 2022-11-16 13:48:30 +01:00
0bc99dbd4f Other(refactor): Use normal value + mutex for user.apiAddrs 2022-11-16 13:48:30 +01:00
83339da26c Other(refactor): Use normal value + mutex for user.apiUser 2022-11-16 13:48:30 +01:00
85c0d6f837 Other: Hold user labels in memory
Labels can be held locally and updated in memory. This greatly improves
the responsiveness of IMAP mailbox operations as we don't need to fetch
all a user's labels to find the parent whenever a mailbox is moved.
2022-11-16 13:48:30 +01:00
c7a5b8559c Other: Updating address flags should be handled 2022-11-16 13:48:30 +01:00
828385b049 Other: Fix user sync leaks/race conditions
This fixes various race conditions and leaks related to the user's sync
and API event stream. It was possible for a sync/stream to begin after a
user was already closed; this change prevents that by managing the
goroutines related to sync/stream within cancellable groups.
2022-11-16 13:48:30 +01:00
6bbaf03f1f Other: Fix goroutine leaks in sync tests
Add missing Close calls.

Properly handle nil channel for `user.startSync`.

This patch also updated liteapi and Gluon to latest master and dev
version respectively.
2022-11-16 13:48:30 +01:00