yesBad/proton-bridge
1
0
Fork 0
forked from Silverfish/proton-bridge
Code Actions Activity
Files
54c013012e2222b2e09eaafb003a3eaffcd4171f
proton-bridge/COPYING_NOTES.md
Leander Beernaert e464e11ab9 GODT-2224: Refactor bridge sync to use less memory 
Updates go-proton-api and Gluon to includes memory reduction changes and
modify the sync process to take into account how much memory is used
during the sync stage.

The sync process now has an extra stage which first download the message
metada to ensure that we only download up to `syncMaxDownloadRequesMem`
messages or 250 messages total. This allows for scaling the download
request automatically to accommodate many small or few very large
messages.

The IDs are then sent to a download go-routine which downloads the
message and its attachments. The result is then forwarded to another
go-routine which builds the actual message. This stage tries to ensure
that we don't use more than `syncMaxMessageBuildingMem` to build these
messages.

Finally the result is sent to a last go-routine which applies the
changes to Gluon and waits for them to be completed.

The new process is currently limited to 2GB. Dynamic scaling will be
implemented in a follow up. For systems with less than 2GB of memory we
limit the values to a set of values that is known to work.
2023-01-30 15:05:43 +01:00

140 lines
16 KiB
Markdown
Raw Blame History

# Copying
Copyright (c) 2022 Proton AG
Proton Mail Bridge is free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the Free
Software Foundation, either version 3 of the License, or (at your option) any
later version.
Proton Mail Bridge is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with
Proton Mail Bridge. If not, see https://www.gnu.org/licenses.
# Dependencies
Proton Mail Bridge includes the following 3rd party software:
* [The Go Project libraries](https://golang.org/project/) | Available under [BSD license](https://golang.org/LICENSE)
* [Qt](https://www.qt.io/) | Available under [multiple licences](https://www.qt.io/licensing)
<!-- START AUTOGEN -->
* [notificator](https://github.com/0xAX/notificator) available under [license](https://github.com/0xAX/notificator/blob/master/LICENSE)
* [semver](https://github.com/Masterminds/semver/v3) available under [license](https://github.com/Masterminds/semver/v3/blob/master/LICENSE)
* [gluon](https://github.com/ProtonMail/gluon) available under [license](https://github.com/ProtonMail/gluon/blob/master/LICENSE)
* [go-autostart](https://github.com/ProtonMail/go-autostart) available under [license](https://github.com/ProtonMail/go-autostart/blob/master/LICENSE)
* [go-proton-api](https://github.com/ProtonMail/go-proton-api) available under [license](https://github.com/ProtonMail/go-proton-api/blob/master/LICENSE)
* [go-rfc5322](https://github.com/ProtonMail/go-rfc5322) available under [license](https://github.com/ProtonMail/go-rfc5322/blob/master/LICENSE)
* [gopenpgp](https://github.com/ProtonMail/gopenpgp/v2) available under [license](https://github.com/ProtonMail/gopenpgp/v2/blob/master/LICENSE)
* [goquery](https://github.com/PuerkitoBio/goquery) available under [license](https://github.com/PuerkitoBio/goquery/blob/master/LICENSE)
* [ishell](https://github.com/abiosoft/ishell) available under [license](https://github.com/abiosoft/ishell/blob/master/LICENSE)
* [go-singleinstance](https://github.com/allan-simon/go-singleinstance) available under [license](https://github.com/allan-simon/go-singleinstance/blob/master/LICENSE)
* [juniper](https://github.com/bradenaw/juniper) available under [license](https://github.com/bradenaw/juniper/blob/master/LICENSE)
* [godog](https://github.com/cucumber/godog) available under [license](https://github.com/cucumber/godog/blob/master/LICENSE)
* [messages-go](https://github.com/cucumber/messages-go/v16) available under [license](https://github.com/cucumber/messages-go/v16/blob/master/LICENSE)
* [docker-credential-helpers](https://github.com/docker/docker-credential-helpers) available under [license](https://github.com/docker/docker-credential-helpers/blob/master/LICENSE)
* [go-sysinfo](https://github.com/elastic/go-sysinfo) available under [license](https://github.com/elastic/go-sysinfo/blob/master/LICENSE)
* [go-imap](https://github.com/emersion/go-imap) available under [license](https://github.com/emersion/go-imap/blob/master/LICENSE)
* [go-imap-id](https://github.com/emersion/go-imap-id) available under [license](https://github.com/emersion/go-imap-id/blob/master/LICENSE)
* [go-message](https://github.com/emersion/go-message) available under [license](https://github.com/emersion/go-message/blob/master/LICENSE)
* [go-sasl](https://github.com/emersion/go-sasl) available under [license](https://github.com/emersion/go-sasl/blob/master/LICENSE)
* [go-smtp](https://github.com/emersion/go-smtp) available under [license](https://github.com/emersion/go-smtp/blob/master/LICENSE)
* [color](https://github.com/fatih/color) available under [license](https://github.com/fatih/color/blob/master/LICENSE)
* [sentry-go](https://github.com/getsentry/sentry-go) available under [license](https://github.com/getsentry/sentry-go/blob/master/LICENSE)
* [resty](https://github.com/go-resty/resty/v2) available under [license](https://github.com/go-resty/resty/v2/blob/master/LICENSE)
* [go-json](https://github.com/goccy/go-json) available under [license](https://github.com/goccy/go-json/blob/master/LICENSE)
* [dbus](https://github.com/godbus/dbus) available under [license](https://github.com/godbus/dbus/blob/master/LICENSE)
* [mock](https://github.com/golang/mock) available under [license](https://github.com/golang/mock/blob/master/LICENSE)
* [go-cmp](https://github.com/google/go-cmp) available under [license](https://github.com/google/go-cmp/blob/master/LICENSE)
* [uuid](https://github.com/google/uuid) available under [license](https://github.com/google/uuid/blob/master/LICENSE)
* [go-multierror](https://github.com/hashicorp/go-multierror) available under [license](https://github.com/hashicorp/go-multierror/blob/master/LICENSE)
* [html2text](https://github.com/jaytaylor/html2text) available under [license](https://github.com/jaytaylor/html2text/blob/master/LICENSE)
* [go-keychain](https://github.com/keybase/go-keychain) available under [license](https://github.com/keybase/go-keychain/blob/master/LICENSE)
* [dns](https://github.com/miekg/dns) available under [license](https://github.com/miekg/dns/blob/master/LICENSE)
* [memory](https://github.com/pbnjay/memory) available under [license](https://github.com/pbnjay/memory/blob/master/LICENSE)
* [errors](https://github.com/pkg/errors) available under [license](https://github.com/pkg/errors/blob/master/LICENSE)
* [profile](https://github.com/pkg/profile) available under [license](https://github.com/pkg/profile/blob/master/LICENSE)
* [logrus](https://github.com/sirupsen/logrus) available under [license](https://github.com/sirupsen/logrus/blob/master/LICENSE)
* [testify](https://github.com/stretchr/testify) available under [license](https://github.com/stretchr/testify/blob/master/LICENSE)
* [cli](https://github.com/urfave/cli/v2) available under [license](https://github.com/urfave/cli/v2/blob/master/LICENSE)
* [msgpack](https://github.com/vmihailenco/msgpack/v5) available under [license](https://github.com/vmihailenco/msgpack/v5/blob/master/LICENSE)
* [goleak](https://go.uber.org/goleak)
* [exp](https://golang.org/x/exp) available under [license](https://cs.opensource.google/go/x/exp/+/master:LICENSE)
* [net](https://golang.org/x/net) available under [license](https://cs.opensource.google/go/x/net/+/master:LICENSE)
* [sys](https://golang.org/x/sys) available under [license](https://cs.opensource.google/go/x/sys/+/master:LICENSE)
* [text](https://golang.org/x/text) available under [license](https://cs.opensource.google/go/x/text/+/master:LICENSE)
* [grpc](https://google.golang.org/grpc) available under [license](https://github.com/grpc/grpc-go/blob/master/LICENSE)
* [protobuf](https://google.golang.org/protobuf) available under [license](https://github.com/protocolbuffers/protobuf/blob/main/LICENSE)
* [plist](https://howett.net/plist) available under [license](https://github.com/DHowett/go-plist/blob/main/LICENSE)
* [atlas](https://ariga.io/atlas)
* [ent](https://entgo.io/ent)
* [bcrypt](https://github.com/ProtonMail/bcrypt) available under [license](https://github.com/ProtonMail/bcrypt/blob/master/LICENSE)
* [go-crypto](https://github.com/ProtonMail/go-crypto) available under [license](https://github.com/ProtonMail/go-crypto/blob/master/LICENSE)
* [go-mime](https://github.com/ProtonMail/go-mime) available under [license](https://github.com/ProtonMail/go-mime/blob/master/LICENSE)
* [go-srp](https://github.com/ProtonMail/go-srp) available under [license](https://github.com/ProtonMail/go-srp/blob/master/LICENSE)
* [readline](https://github.com/abiosoft/readline) available under [license](https://github.com/abiosoft/readline/blob/master/LICENSE)
* [levenshtein](https://github.com/agext/levenshtein) available under [license](https://github.com/agext/levenshtein/blob/master/LICENSE)
* [cascadia](https://github.com/andybalholm/cascadia) available under [license](https://github.com/andybalholm/cascadia/blob/master/LICENSE)
* [antlr](https://github.com/antlr/antlr4/runtime/Go/antlr) available under [license](https://github.com/antlr/antlr4/runtime/Go/antlr/blob/master/LICENSE)
* [go-textseg](https://github.com/apparentlymart/go-textseg/v13) available under [license](https://github.com/apparentlymart/go-textseg/v13/blob/master/LICENSE)
* [test](https://github.com/chzyer/test) available under [license](https://github.com/chzyer/test/blob/master/LICENSE)
* [circl](https://github.com/cloudflare/circl) available under [license](https://github.com/cloudflare/circl/blob/master/LICENSE)
* [go-md2man](https://github.com/cpuguy83/go-md2man/v2) available under [license](https://github.com/cpuguy83/go-md2man/v2/blob/master/LICENSE)
* [saferith](https://github.com/cronokirby/saferith) available under [license](https://github.com/cronokirby/saferith/blob/master/LICENSE)
* [gherkin-go](https://github.com/cucumber/gherkin-go/v19) available under [license](https://github.com/cucumber/gherkin-go/v19/blob/master/LICENSE)
* [wincred](https://github.com/danieljoos/wincred) available under [license](https://github.com/danieljoos/wincred/blob/master/LICENSE)
* [go-spew](https://github.com/davecgh/go-spew) available under [license](https://github.com/davecgh/go-spew/blob/master/LICENSE)
* [go-windows](https://github.com/elastic/go-windows) available under [license](https://github.com/elastic/go-windows/blob/master/LICENSE)
* [go-textwrapper](https://github.com/emersion/go-textwrapper) available under [license](https://github.com/emersion/go-textwrapper/blob/master/LICENSE)
* [go-vcard](https://github.com/emersion/go-vcard) available under [license](https://github.com/emersion/go-vcard/blob/master/LICENSE)
* [go-shlex](https://github.com/flynn-archive/go-shlex) available under [license](https://github.com/flynn-archive/go-shlex/blob/master/LICENSE)
* [sse](https://github.com/gin-contrib/sse) available under [license](https://github.com/gin-contrib/sse/blob/master/LICENSE)
* [gin](https://github.com/gin-gonic/gin) available under [license](https://github.com/gin-gonic/gin/blob/master/LICENSE)
* [inflect](https://github.com/go-openapi/inflect) available under [license](https://github.com/go-openapi/inflect/blob/master/LICENSE)
* [locales](https://github.com/go-playground/locales) available under [license](https://github.com/go-playground/locales/blob/master/LICENSE)
* [universal-translator](https://github.com/go-playground/universal-translator) available under [license](https://github.com/go-playground/universal-translator/blob/master/LICENSE)
* [validator](https://github.com/go-playground/validator/v10) available under [license](https://github.com/go-playground/validator/v10/blob/master/LICENSE)
* [uuid](https://github.com/gofrs/uuid) available under [license](https://github.com/gofrs/uuid/blob/master/LICENSE)
* [protobuf](https://github.com/golang/protobuf) available under [license](https://github.com/golang/protobuf/blob/master/LICENSE)
* [errwrap](https://github.com/hashicorp/errwrap) available under [license](https://github.com/hashicorp/errwrap/blob/master/LICENSE)
* [go-immutable-radix](https://github.com/hashicorp/go-immutable-radix) available under [license](https://github.com/hashicorp/go-immutable-radix/blob/master/LICENSE)
* [go-memdb](https://github.com/hashicorp/go-memdb) available under [license](https://github.com/hashicorp/go-memdb/blob/master/LICENSE)
* [golang-lru](https://github.com/hashicorp/golang-lru) available under [license](https://github.com/hashicorp/golang-lru/blob/master/LICENSE)
* [hcl](https://github.com/hashicorp/hcl/v2) available under [license](https://github.com/hashicorp/hcl/v2/blob/master/LICENSE)
* [multierror](https://github.com/joeshaw/multierror) available under [license](https://github.com/joeshaw/multierror/blob/master/LICENSE)
* [go](https://github.com/json-iterator/go) available under [license](https://github.com/json-iterator/go/blob/master/LICENSE)
* [go-urn](https://github.com/leodido/go-urn) available under [license](https://github.com/leodido/go-urn/blob/master/LICENSE)
* [go-colorable](https://github.com/mattn/go-colorable) available under [license](https://github.com/mattn/go-colorable/blob/master/LICENSE)
* [go-isatty](https://github.com/mattn/go-isatty) available under [license](https://github.com/mattn/go-isatty/blob/master/LICENSE)
* [go-runewidth](https://github.com/mattn/go-runewidth) available under [license](https://github.com/mattn/go-runewidth/blob/master/LICENSE)
* [go-sqlite3](https://github.com/mattn/go-sqlite3) available under [license](https://github.com/mattn/go-sqlite3/blob/master/LICENSE)
* [go-wordwrap](https://github.com/mitchellh/go-wordwrap) available under [license](https://github.com/mitchellh/go-wordwrap/blob/master/LICENSE)
* [concurrent](https://github.com/modern-go/concurrent) available under [license](https://github.com/modern-go/concurrent/blob/master/LICENSE)
* [reflect2](https://github.com/modern-go/reflect2) available under [license](https://github.com/modern-go/reflect2/blob/master/LICENSE)
* [tablewriter](https://github.com/olekukonko/tablewriter) available under [license](https://github.com/olekukonko/tablewriter/blob/master/LICENSE)
* [go-toml](https://github.com/pelletier/go-toml/v2) available under [license](https://github.com/pelletier/go-toml/v2/blob/master/LICENSE)
* [lz4](https://github.com/pierrec/lz4/v4) available under [license](https://github.com/pierrec/lz4/v4/blob/master/LICENSE)
* [go-difflib](https://github.com/pmezard/go-difflib) available under [license](https://github.com/pmezard/go-difflib/blob/master/LICENSE)
* [procfs](https://github.com/prometheus/procfs) available under [license](https://github.com/prometheus/procfs/blob/master/LICENSE)
* [uniseg](https://github.com/rivo/uniseg) available under [license](https://github.com/rivo/uniseg/blob/master/LICENSE)
* [blackfriday](https://github.com/russross/blackfriday/v2) available under [license](https://github.com/russross/blackfriday/v2/blob/master/LICENSE)
* [pflag](https://github.com/spf13/pflag) available under [license](https://github.com/spf13/pflag/blob/master/LICENSE)
* [bom](https://github.com/ssor/bom) available under [license](https://github.com/ssor/bom/blob/master/LICENSE)
* [codec](https://github.com/ugorji/go/codec) available under [license](https://github.com/ugorji/go/codec/blob/master/LICENSE)
* [tagparser](https://github.com/vmihailenco/tagparser/v2) available under [license](https://github.com/vmihailenco/tagparser/v2/blob/master/LICENSE)
* [smetrics](https://github.com/xrash/smetrics) available under [license](https://github.com/xrash/smetrics/blob/master/LICENSE)
* [go-cty](https://github.com/zclconf/go-cty) available under [license](https://github.com/zclconf/go-cty/blob/master/LICENSE)
* [crypto](https://golang.org/x/crypto) available under [license](https://cs.opensource.google/go/x/crypto/+/master:LICENSE)
* [mod](https://golang.org/x/mod) available under [license](https://cs.opensource.google/go/x/mod/+/master:LICENSE)
* [sync](https://golang.org/x/sync) available under [license](https://cs.opensource.google/go/x/sync/+/master:LICENSE)
* [tools](https://golang.org/x/tools) available under [license](https://cs.opensource.google/go/x/tools/+/master:LICENSE)
* [genproto](https://google.golang.org/genproto)
gopkg.in/yaml.v2
gopkg.in/yaml.v3
* [docker-credential-helpers](https://github.com/ProtonMail/docker-credential-helpers) available under [license](https://github.com/ProtonMail/docker-credential-helpers/blob/master/LICENSE)
* [go-message](https://github.com/ProtonMail/go-message) available under [license](https://github.com/ProtonMail/go-message/blob/master/LICENSE)
* [go-keychain](https://github.com/cuthix/go-keychain) available under [license](https://github.com/cuthix/go-keychain/blob/master/LICENSE)
<!-- END AUTOGEN -->
View Git Blame Copy Permalink