Commit Graph

39 Commits

Author SHA1 Message Date
a6c20f698c feat(GODT-2828): Increase sync progress report frequency
We now report sync progress after a batch completes each stage.
2023-08-29 11:50:50 +00:00
f7252ed40e fix(GODT-2693): Duplicate messages in sent folder
* Only call `RemoveOnFail` on error. It was also called on success.
* Add some logging to track send times.
* Fix non-deterministic hash from `rfc8222.GetMessageHash`.
    https://github.com/ProtonMail/gluon/pull/395
2023-08-29 11:50:50 +00:00
06d7fdf26f fix(GODT-2805): Ignore Contact Group Labels 2023-08-29 07:25:00 +00:00
8aee732fe8 fix(GODT-2866): Add 429/5xx Retry to Event Service 2023-08-29 07:25:00 +00:00
0b9b886039 fix(GODT-2829): Fix new sync service bugs
* Fix wrong context use for message downloads
* Fix delete of sync data failing due ErrNotFound
* Pre-allocate attachment data buffer before download
* Fix calculation of progress if message count is higher than total
2023-08-28 13:37:55 +02:00
efbe84964f feat(GODT-2829): Integrate new sync service
Update imap service to use the new sync service.

The new sync state is stored as simple file on disk to avoid contention
with concurrent vault writes.
2023-08-25 15:21:00 +02:00
aa77a67a1c fix(GODT-2829): Sync Service fixes
Fix tracking of child jobs. The build stage splits the incoming work
even further, but this was not reflected in the wait group counter. This
also fixes an issue where the cache was cleared to late.

Add more debug info for analysis.

Refactor sync state interface in order to have persistent sync rate.
2023-08-25 15:03:27 +02:00
78f7cbdc79 feat(GODT-2829): New Sync Service
Implementation of the new sync service that interleaves syncing jobs for
all active users.

It also includes improvements to the message downloader. The download
will now auto rate limit the parallel workers based on the server
responses.

Additionally each of the stages is now tested in isolation to ensure the
behavior matches the expectations.

Finally, this patch does not replace the existing IMAP sync. A follow up
patch is necessary to integrate the IMAP bits into the interfaces
required by these changes.
2023-08-23 16:12:19 +02:00
a731237701 test(GODT-2871): tests for new telemetry logic. 2023-08-22 11:15:47 +02:00
f557666b4d feat(GODT-2871): is telemetry enabled as service. 2023-08-21 14:08:47 +02:00
e724fafe2f chore: checkout features from trift to devel 2023-08-15 15:40:53 +02:00
cba73997cd fix(GODT-2802): Fix missing key pass input for keyring unlock 2023-08-14 13:09:22 +02:00
29f44fc312 fix(GODT-2814): Server Manager shutdown task cancel 2023-08-14 11:04:53 +02:00
41c125f65e feat(GODT-2803): Gluon IMAP State access
Update to latest Gluon to allow access to the database for bridge. The
cache is placed in a `SharedCache` type to ensure that we respect calls
to `Connector.Close`.

In theory, this should never trigger an error, but this way we can catch
it if it happens.

https://github.com/ProtonMail/gluon/pull/391
2023-08-14 09:51:49 +02:00
24331f9715 fix(GODT-2803): Separate conditions to pause event loop for IMAP
Add two separate toggles to control event loop pausing. This is required
to prevent cases where the bridge requests the event loop to be paused
but a sync process completes and resumes the event loop.

For the loop to resume now both states need to be set to false. This
will be removed once GODT-2848 is implemented.
2023-08-11 14:32:18 +02:00
a5500629e5 feat(GODT-2803): Add Event Poll Waiter
Wait for the current event poll to finish publishing events after a
request to pause the event loop. This is required to change the gluon
cache directory.
2023-08-11 14:32:18 +02:00
4ee6da4baa chore(GODT-2848): Simplify User Event Service
Rather than having the services subscribe to each individual event type,
have only one subscriber for the whole event object. This spreads the
logic to the service through the `EventHandler` type.

This is important for the next stage where the IMAP service will be
required to apply events from the past.
2023-08-09 16:11:09 +02:00
6eb1878f66 fix(GODT-2827): Restore ticker to event poller 2023-08-09 10:17:41 +02:00
20b188368a feat(GODT-2799): SMTP service interacts directly with Server Manager
Bridge no longer needs to manually add and remove accounts from the
service.
2023-08-08 14:14:52 +02:00
9a96588afb feat(GODT-2814): Standalone Server Manager
Convert ServerManger into a standalone service so that it can become a
self contained module.
2023-08-07 16:47:41 +02:00
f0e2688a8e fix(GODT-2800): Ensure 429 does not cause bad event 2023-07-31 13:15:26 +02:00
06639ff6cd fix(GODT-2800): Handle subscribe followed by unsubscribe properly 2023-07-31 13:12:29 +02:00
823ca4d207 feat(GODT-2822): Integrate and activate all service
The bridge now runs on the new architecture.
2023-07-31 12:36:03 +02:00
a187747c7c feat(GODT-2802): IMAP Serivce
Handles all IMAP related tasks. Unlike the previous iteration, this new
service automatically adds and removes users from Gluon by interfacing
with server manager.
2023-07-31 11:06:47 +02:00
11ebb16933 fix(GODT-2822): Update smtp and events service to use ordered cancel 2023-07-31 11:06:47 +02:00
0048767022 fix(GODT-28022): Missing user identity service methods
Add methods to access user identity state from the outside.
2023-07-31 11:06:47 +02:00
86bde91958 feat(GODT-2802): Ordered task cancellation
Add `OrderedCancelGroup` type to enforce the cancellation of go routines
in a reverse order. Additionally this type waits for the go-routine to
finish before proceeding to the next one.

We need this to guarantee that all the user services shut down properly
without deadlocking.
2023-07-28 14:50:42 +02:00
334a256638 fix(GODT-2802): Remove CPC request from Event Service
Prevents deadlocks if the service needs to be paused during an event
loop.
2023-07-26 10:03:48 +02:00
da528f2d9b feat(GODT-2801): Add Key Decryption helpers to identityservice.State 2023-07-25 09:59:26 +02:00
1b0f930471 fix(GODT-2802): Consolidate Address Mode 2023-07-25 09:15:52 +02:00
0b35f41ffd feat(GODT-2799): Integrate SMTP service with User Identity Service 2023-07-24 17:10:05 +02:00
7be1a8ae8a feat(GODT-2801): Identity State Cloning & Auth Check
This patch moves the `user.User.CheckAuth` function into the `State`
type as it contains all the necessary information to perform this check.

A couple of more interfaces are added to abstract the retrieval of the
Bridge and the User's key passwords, as well as telemetry reports.

Finally, adds `State.OnAddressEvents` which other services should use to
update the state.
2023-07-24 17:10:05 +02:00
bdbf1bdd76 fix(GODT-2800): Pending Subscriptions Cleanup
Ensure pending subscriptions are cleaned up with `Service.Close()` or
we can leak go routines.
2023-07-24 17:10:05 +02:00
040ddadb7a feat(GODT-2801): Identity Service
Identity Service contains all the information related to user state,
addresses and keys.

This patch also introduces the `State` type which can be used by other
services to maintain their own copy of this state to avoid lock
contention.

Finally, there are currently no external facing methods via a CPC
interface. Those will added as needed once the refactoring of the
architecture is complete.
2023-07-24 11:22:50 +02:00
11f6f84dd6 fix(GODT-2800): ChanneledSubcriber types 2023-07-21 15:18:49 +02:00
82efa16d65 feat(GODT-2800): User Event Service
This patch adds the User Event Service which is meant to replace the
current event polling flow.

Each user interested in receiving events should register a new
subscriber using the `Service.Subscribe` function and then react on
the incoming events.

The current patch does not hook this up Bridge user as there are no
existing consumers, but it does provide extensive testing for the
expected behavior.
2023-07-21 11:47:43 +02:00
bc66841cdc chore(GODT-2799): Move SMTP backend to SMTP service module 2023-07-19 09:12:33 +02:00
d120bbeffc chore(GODT-2799): Separate account states for SMTP Backend
Rather than accessing the Bridge user list, each user register their
individual SMTP service with the server manager.

Note that some dependencies on the user data are hidden behind the
`UserInterface`. These will be removed in a future patch.
2023-07-19 09:07:02 +02:00
3ef526333a feat(GODT-2799): SMTP Service
Refactor code to isolate the SMTP functionality in a dedicated SMTP
service for each user as discussed in the Bridge Service Architecture
RFC.

Some shared types have been moved from `user` to `usertypes` so that
they can be shared with Service and User Code.

Finally due to lack of recursive imports, the user data SMTP needs
access to is hidden behind an interface until the User Identity service
is implemented.
2023-07-18 11:49:18 +02:00