yesBad/proton-bridge
1
0
Fork 0
forked from Silverfish/proton-bridge
Code Actions Activity

GODT-2224: Refactor bridge sync to use less memory

Browse Source 
Updates go-proton-api and Gluon to includes memory reduction changes and
modify the sync process to take into account how much memory is used
during the sync stage.

The sync process now has an extra stage which first download the message
metada to ensure that we only download up to `syncMaxDownloadRequesMem`
messages or 250 messages total. This allows for scaling the download
request automatically to accommodate many small or few very large
messages.

The IDs are then sent to a download go-routine which downloads the
message and its attachments. The result is then forwarded to another
go-routine which builds the actual message. This stage tries to ensure
that we don't use more than `syncMaxMessageBuildingMem` to build these
messages.

Finally the result is sent to a last go-routine which applies the
changes to Gluon and waits for them to be completed.

The new process is currently limited to 2GB. Dynamic scaling will be
implemented in a follow up. For systems with less than 2GB of memory we
limit the values to a set of values that is known to work.
This commit is contained in:
Leander Beernaert
2023-01-24 13:41:03 +01:00
parent d7ff54d679
commit e464e11ab9
24 changed files with 481 additions and 303 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
go.mod
View File

@ -5,9 +5,9 @@ go 1.18
require (
github.com/0xAX/notificator v0.0.0-20220220101646-ee9b8921e557
github.com/Masterminds/semver/v3 v3.1.1
github.com/ProtonMail/gluon v0.14.2-0.20230127085305-bc2d818d9d13
github.com/ProtonMail/gluon v0.14.2-0.20230130104154-2c64e59b8f54
github.com/ProtonMail/go-autostart v0.0.0-20210130080809-00ed301c8e9a
github.com/ProtonMail/go-proton-api v0.3.1-0.20230126112849-3c1ac277855e
github.com/ProtonMail/go-proton-api v0.3.1-0.20230130093944-dd1190680368
github.com/ProtonMail/go-rfc5322 v0.11.0
github.com/ProtonMail/gopenpgp/v2 v2.4.10
github.com/PuerkitoBio/goquery v1.8.0
@ -35,6 +35,7 @@ require (
github.com/jaytaylor/html2text v0.0.0-20211105163654-bc68cce691ba
github.com/keybase/go-keychain v0.0.0
github.com/miekg/dns v1.1.50
github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58
github.com/pkg/errors v0.9.1
github.com/pkg/profile v1.6.0
github.com/sirupsen/logrus v1.9.0
@ -99,6 +100,7 @@ require (
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/olekukonko/tablewriter v0.0.5 // indirect
github.com/pelletier/go-toml/v2 v2.0.5 // indirect
github.com/pierrec/lz4/v4 v4.1.17 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/procfs v0.8.0 // indirect
github.com/rivo/uniseg v0.4.2 // indirect