GODT-829 fix apple mail subfolders

.gitlab-ci.yml

@@ -82,9 +82,7 @@ dependency-updates:
   script:
     - make build
   artifacts:
-    # Note: The latest artifacts for refs are locked against deletion, and kept regardless of the expiry time.
+    expire_in: 2 week
-    # Introduced in GitLab 13.0 behind a disabled feature flag, and made the default behavior in GitLab 13.4.
-    expire_in: 1 day
 
 build-linux:
   extends: .build-base

Changelog.md

@@ -2,13 +2,33 @@
 
 Changelog [format](http://keepachangelog.com/en/1.0.0/)
 
-## [IE 1.1.2] Danube (beta 2020-09-xx)
+## Unreleased
 
 ### Fixed
-* GODT-770 Better handling of extraneous end-of-mail indicator.
+* GODT-829 Remove `NoInferior` to display sub-folders in apple mail.
+
+## [Bridge 1.4.4] Forth
+
+### Fixed
+* GODT-798 Replace, don't add, transfer encoding when making body 7-bit clean.
+* Move/Copy duplicate for emails with References in Outlook
+* CSB-247 Cannot update from 1.4.0
+
+
+## [Bridge 1.4.3] Forth
 
 ### Changed
-* Bump crypto version to v0.0.0-20200818122824-ed5d25e28db8
+* Reverted sending IMAP updates to be not blocking again.
+
+### Fixed
+* GODT-783 Settings flags by FLAGS (not using +/-FLAGS) do not change spam state.
+
+
+## [Bridge 1.4.2] Forth
+
+### Changed
+* GODT-761 Use label.Path instead of Name to partially support subfolders for webapp beta release.
+* GODT-765 Improve speed of checking whether message is deleted.
 
 
 ## [IE 1.1.1] Danube (beta 2020-09-xx) [Bridge 1.4.1] Forth (beta 2020-09-xx)
@@ -18,12 +38,14 @@ Changelog [format](http://keepachangelog.com/en/1.0.0/)
 * GODT-752 Parsing non-utf8 multipart/alternative message.
 * GODT-752 Parsing message with duplicate charset parameter.
 
+
 ## [IE 1.1.0] Danube
 
 ### Fixed
 * GODT-703 Import-Export showed always at least one total message.
 * GODT-738 Fix for mbox files with long lines.
 
+
 ## [Bridge 1.4.0] Forth
 
 ### Added

go.mod

@@ -78,5 +78,5 @@ replace (
 	github.com/emersion/go-mbox => github.com/ProtonMail/mbox v0.0.0-20200918064939-909a18c9af45
 	github.com/emersion/go-smtp => github.com/ProtonMail/go-smtp v0.0.0-20181206232543-8261df20d309
 	github.com/jameskeane/bcrypt => github.com/ProtonMail/bcrypt v0.0.0-20170924085257-7509ea014998
-	golang.org/x/crypto => github.com/ProtonMail/crypto v0.0.0-20200818122824-ed5d25e28db8
+	golang.org/x/crypto => github.com/ProtonMail/crypto v0.0.0-20200416114516-1fa7f403fb9c
 )

go.sum

@@ -1,12 +1,13 @@
 github.com/0xAX/notificator v0.0.0-20191016112426-3962a5ea8da1 h1:j9HaafapDbPbGRDku6e/HRs6KBMcKHiWcm1/9Sbxnl4=
 github.com/0xAX/notificator v0.0.0-20191016112426-3962a5ea8da1/go.mod h1:NtXa9WwQsukMHZpjNakTTz0LArxvGYdPA9CjIcUSZ6s=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
+github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww=
 github.com/Masterminds/semver/v3 v3.1.0 h1:Y2lUDsFKVRSYGojLJ1yLxSXdMmMYTYls0rCvoqmMUQk=
 github.com/Masterminds/semver/v3 v3.1.0/go.mod h1:VPu/7SZ7ePZ3QOrcuXROw5FAcLl4a0cBrbBpGY/8hQs=
 github.com/ProtonMail/bcrypt v0.0.0-20170924085257-7509ea014998 h1:YT2uVwQiRQZxCaaahwfcgTq2j3j66w00n/27gb/zubs=
 github.com/ProtonMail/bcrypt v0.0.0-20170924085257-7509ea014998/go.mod h1:HecWFHognK8GfRDGnFQbW/LiV7A3MX3gZVs45vk5h8I=
-github.com/ProtonMail/crypto v0.0.0-20200818122824-ed5d25e28db8 h1:u1j0xLTrCHpNS40B6m4Sv3IVUz5m9jt+AnTIopT3IgM=
+github.com/ProtonMail/crypto v0.0.0-20200416114516-1fa7f403fb9c h1:DAvlgde2Stu18slmjwikiMPs/CKPV35wSvmJS34z0FU=
-github.com/ProtonMail/crypto v0.0.0-20200818122824-ed5d25e28db8/go.mod h1:Pxr7w4gA2ikI4sWyYwEffm+oew1WAJHzG1SiDpQMkrI=
+github.com/ProtonMail/crypto v0.0.0-20200416114516-1fa7f403fb9c/go.mod h1:Pxr7w4gA2ikI4sWyYwEffm+oew1WAJHzG1SiDpQMkrI=
 github.com/ProtonMail/docker-credential-helpers v1.1.0 h1:+kvUIpwWcbtP3WFv5sSvkFn/XLzSqPOB5AAthuk9xPk=
 github.com/ProtonMail/docker-credential-helpers v1.1.0/go.mod h1:mK0aBveCxhnQ756AmaTfXMZDeULvheYVhF/MWMErN5g=
 github.com/ProtonMail/go-appdir v1.1.0 h1:9hdNDlU9kTqRKVNzmoqah8qqrj5QZyLByQdwQNlFWig=
@@ -67,6 +68,8 @@ github.com/emersion/go-imap-specialuse v0.0.0-20200722111535-598ff00e4075 h1:z8T
 github.com/emersion/go-imap-specialuse v0.0.0-20200722111535-598ff00e4075/go.mod h1:/nybxhI8kXom8Tw6BrHMl42usALvka6meORflnnYwe4=
 github.com/emersion/go-imap-unselect v0.0.0-20171113212723-b985794e5f26 h1:FiSb8+XBQQSkcX3ubr+1tAtlRJBYaFmRZqOAweZ9Wy8=
 github.com/emersion/go-imap-unselect v0.0.0-20171113212723-b985794e5f26/go.mod h1:+gnnZx3Mg3MnCzZrv0eZdp5puxXQUgGT/6N6L7ShKfM=
+github.com/emersion/go-mbox v1.0.0 h1:HN6aKbyqmgIfK9fS/gen+NRr2wXLSxZXWfdAIAnzQPc=
+github.com/emersion/go-mbox v1.0.0/go.mod h1:Yp9IVuuOYLEuMv4yjgDHvhb5mHOcYH6x92Oas3QqEZI=
 github.com/emersion/go-message v0.11.1/go.mod h1:C4jnca5HOTo4bGN9YdqNQM9sITuT3Y0K6bSUw9RklvY=
 github.com/emersion/go-message v0.12.1-0.20200903165315-e1abe21f389a h1:3C6qIGgPr1qAT0ikRD5NbyKpME/iHCDeXhpv/JJsFsE=
 github.com/emersion/go-message v0.12.1-0.20200903165315-e1abe21f389a/go.mod h1:kYIioST9GDHte9/BRWgi93rpqbDuFftMjKSMaXS8ABo=
@@ -98,6 +101,8 @@ github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/U
 github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
 github.com/hashicorp/go-multierror v1.1.0 h1:B9UzwGQJehnUY1yNrnwREHc3fGbC2xefo8g4TbElacI=
 github.com/hashicorp/go-multierror v1.1.0/go.mod h1:spPvp8C1qA32ftKqdAHm4hHTbPw+vmowP0z+KUhOZdA=
+github.com/jameshoulahan/go-imap v0.0.0-20200728140727-d57327f48843 h1:suxlO4AC4E4bjueAsL0m+qp8kmkxRWMGj+5bBU/KJ8g=
+github.com/jameshoulahan/go-imap v0.0.0-20200728140727-d57327f48843/go.mod h1:yKASt+C3ZiDAiCSssxg9caIckWF/JG7ZQTO7GAmvicU=
 github.com/jaytaylor/html2text v0.0.0-20200412013138-3577fbdbcff7 h1:g0fAGBisHaEQ0TRq1iBvemFRf+8AEWEmBESSiWB3Vsc=
 github.com/jaytaylor/html2text v0.0.0-20200412013138-3577fbdbcff7/go.mod h1:CVKlgaMiht+LXvHG173ujK6JUhZXKb2u/BQtjPDIvyk=
 github.com/kardianos/osext v0.0.0-20190222173326-2bc1f35cddc0 h1:iQTw/8FWTuc7uiaSepXwyf3o52HaUYcV+Tu66S3F5GA=

internal/bridge/credits.go

@@ -15,8 +15,8 @@
 // You should have received a copy of the GNU General Public License
 // along with ProtonMail Bridge.  If not, see <https://www.gnu.org/licenses/>.
 
-// Code generated by ./credits.sh at Wed Sep 16 16:48:58 CEST 2020. DO NOT EDIT.
+// Code generated by ./credits.sh at Tue Sep 29 14:56:25 CEST 2020. DO NOT EDIT.
 
 package bridge
 
-const Credits = "github.com/0xAX/notificator;github.com/ProtonMail/bcrypt;github.com/ProtonMail/crypto;github.com/ProtonMail/docker-credential-helpers;github.com/ProtonMail/go-appdir;github.com/ProtonMail/go-apple-mobileconfig;github.com/ProtonMail/go-autostart;github.com/ProtonMail/go-imap;github.com/ProtonMail/go-imap-id;github.com/ProtonMail/go-smtp;github.com/ProtonMail/go-vcard;github.com/ProtonMail/gopenpgp/v2;github.com/PuerkitoBio/goquery;github.com/abiosoft/ishell;github.com/abiosoft/readline;github.com/allan-simon/go-singleinstance;github.com/certifi/gocertifi;github.com/chzyer/logex;github.com/chzyer/test;github.com/cucumber/godog;github.com/docker/docker-credential-helpers;github.com/emersion/go-imap;github.com/emersion/go-imap-appendlimit;github.com/emersion/go-imap-idle;github.com/emersion/go-imap-move;github.com/emersion/go-imap-quota;github.com/emersion/go-imap-specialuse;github.com/emersion/go-imap-unselect;github.com/emersion/go-mbox;github.com/emersion/go-message;github.com/emersion/go-sasl;github.com/emersion/go-smtp;github.com/emersion/go-textwrapper;github.com/emersion/go-vcard;github.com/fatih/color;github.com/flynn-archive/go-shlex;github.com/getsentry/raven-go;github.com/go-resty/resty/v2;github.com/golang/mock;github.com/google/go-cmp;github.com/google/uuid;github.com/gopherjs/gopherjs;github.com/hashicorp/go-multierror;github.com/jameskeane/bcrypt;github.com/jaytaylor/html2text;github.com/kardianos/osext;github.com/keybase/go-keychain;github.com/logrusorgru/aurora;github.com/mattn/go-runewidth;github.com/miekg/dns;github.com/myesui/uuid;github.com/nsf/jsondiff;github.com/olekukonko/tablewriter;github.com/pkg/errors;github.com/sirupsen/logrus;github.com/skratchdot/open-golang;github.com/ssor/bom;github.com/stretchr/testify;github.com/therecipe/qt;github.com/twinj/uuid;github.com/urfave/cli;go.etcd.io/bbolt;golang.org/x/crypto;golang.org/x/net;golang.org/x/text;gopkg.in/stretchr/testify.v1;;Font Awesome 4.7.0;;Qt 5.13 by Qt group;"
+const Credits = "github.com/0xAX/notificator;github.com/Masterminds/semver/v3;github.com/ProtonMail/bcrypt;github.com/ProtonMail/crypto;github.com/ProtonMail/docker-credential-helpers;github.com/ProtonMail/go-appdir;github.com/ProtonMail/go-apple-mobileconfig;github.com/ProtonMail/go-autostart;github.com/ProtonMail/go-imap;github.com/ProtonMail/go-imap-id;github.com/ProtonMail/go-smtp;github.com/ProtonMail/go-vcard;github.com/ProtonMail/gopenpgp/v2;github.com/ProtonMail/mbox;github.com/PuerkitoBio/goquery;github.com/abiosoft/ishell;github.com/abiosoft/readline;github.com/allan-simon/go-singleinstance;github.com/certifi/gocertifi;github.com/chzyer/logex;github.com/chzyer/test;github.com/cucumber/godog;github.com/docker/docker-credential-helpers;github.com/emersion/go-imap;github.com/emersion/go-imap-appendlimit;github.com/emersion/go-imap-idle;github.com/emersion/go-imap-move;github.com/emersion/go-imap-quota;github.com/emersion/go-imap-specialuse;github.com/emersion/go-imap-unselect;github.com/emersion/go-mbox;github.com/emersion/go-message;github.com/emersion/go-sasl;github.com/emersion/go-smtp;github.com/emersion/go-textwrapper;github.com/emersion/go-vcard;github.com/fatih/color;github.com/flynn-archive/go-shlex;github.com/getsentry/raven-go;github.com/go-resty/resty/v2;github.com/golang/mock;github.com/google/go-cmp;github.com/google/uuid;github.com/gopherjs/gopherjs;github.com/hashicorp/go-multierror;github.com/jameskeane/bcrypt;github.com/jaytaylor/html2text;github.com/kardianos/osext;github.com/keybase/go-keychain;github.com/logrusorgru/aurora;github.com/mattn/go-runewidth;github.com/miekg/dns;github.com/myesui/uuid;github.com/nsf/jsondiff;github.com/olekukonko/tablewriter;github.com/pkg/errors;github.com/sirupsen/logrus;github.com/skratchdot/open-golang;github.com/ssor/bom;github.com/stretchr/testify;github.com/therecipe/qt;github.com/twinj/uuid;github.com/urfave/cli;go.etcd.io/bbolt;golang.org/x/crypto;golang.org/x/net;golang.org/x/text;gopkg.in/stretchr/testify.v1;;Font Awesome 4.7.0;;Qt 5.13 by Qt group;"

internal/imap/mailbox.go

@@ -80,7 +80,10 @@ func (im *imapMailbox) Info() (*imap.MailboxInfo, error) {
 }
 
 func (im *imapMailbox) getFlags() []string {
-	flags := []string{imap.NoInferiorsAttr} // Subfolders are not yet supported by API.
+	flags := []string{}
+	if !im.storeMailbox.IsFolder() || im.storeMailbox.IsSystem() {
+		flags = append(flags, imap.NoInferiorsAttr) // Subfolders are not supported for System or Label
+	}
 	switch im.storeMailbox.LabelID() {
 	case pmapi.SentLabel:
 		flags = append(flags, specialuse.Sent)

internal/imap/mailbox_message.go

@@ -24,7 +24,6 @@ import (
 	"mime/multipart"
 	"net/mail"
 	"net/textproto"
-	"regexp"
 	"sort"
 	"strings"
 	"time"
@@ -141,18 +140,19 @@ func (im *imapMailbox) CreateMessage(flags []string, date time.Time, body imap.L
 	references := m.Header.Get("References")
 	referenceList := strings.Fields(references)
 
-	if len(referenceList) > 0 {
+	// In case there is a mail client which corrupts headers, try
+	// "References" too.
+	if internalID == "" && len(referenceList) > 0 {
 		lastReference := referenceList[len(referenceList)-1]
-		// In case we are using a mail client which corrupts headers, try "References" too.
+		match := pmapi.RxInternalReferenceFormat.FindStringSubmatch(lastReference)
-		re := regexp.MustCompile(pmapi.InternalReferenceFormat)
+		if len(match) == 2 {
-		match := re.FindStringSubmatch(lastReference)
+			internalID = match[1]
-		if len(match) > 0 {
-			internalID = match[0]
 		}
 	}
 
-	// Avoid appending a message which is already on the server. Apply the new
+	// Avoid appending a message which is already on the server. Apply the
-	// label instead. This sometimes happens which Outlook (it uses APPEND instead of COPY).
+	// new label instead. This always happens with Outlook (it uses APPEND
+	// instead of COPY).
 	if internalID != "" {
 		// Check to see if this belongs to a different address in split mode or another ProtonMail account.
 		msg, err := im.storeMailbox.GetMessage(internalID)

internal/imap/mailbox_messages.go

@@ -57,6 +57,10 @@ func (im *imapMailbox) UpdateMessagesFlags(uid bool, seqSet *imap.SeqSet, operat
 	return im.addOrRemoveFlags(operation, messageIDs, flags)
 }
 
+// setFlags is used for FLAGS command (not +FLAGS or -FLAGS), which means
+// to set flags passed as an argument and unset the rest. For example,
+// if message is not read, is flagged and is not deleted, call FLAGS \Seen
+// should flag message as read, unflagged and keep undeleted.
 func (im *imapMailbox) setFlags(messageIDs, flags []string) error { //nolint
 	seen := false
 	flagged := false
@@ -106,16 +110,17 @@ func (im *imapMailbox) setFlags(messageIDs, flags []string) error { //nolint
 		}
 	}
 
-	spamMailbox, err := im.storeAddress.GetMailbox("Spam")
+	// Spam should not be taken into action here as Outlook is using FLAGS
-	if err != nil {
+	// without preserving junk flag. Probably it's because junk is not standard
-		return err
+	// in the rfc3501 and thus Outlook expects calling FLAGS \Seen will not
-	}
+	// change the state of junk or other non-standard flags.
+	// Still, its safe to label as spam once any client sends the request.
 	if spam {
-		if err := spamMailbox.LabelMessages(messageIDs); err != nil {
+		spamMailbox, err := im.storeAddress.GetMailbox("Spam")
+		if err != nil {
 			return err
 		}
-	} else {
+		if err := spamMailbox.LabelMessages(messageIDs); err != nil {
-		if err := spamMailbox.UnlabelMessages(messageIDs); err != nil {
 			return err
 		}
 	}

internal/importexport/credits.go

@@ -15,8 +15,8 @@
 // You should have received a copy of the GNU General Public License
 // along with ProtonMail Bridge.  If not, see <https://www.gnu.org/licenses/>.
 
-// Code generated by ./credits.sh at Wed Sep 23 01:34:10 PM CEST 2020. DO NOT EDIT.
+// Code generated by ./credits.sh at Tue Sep 29 14:56:25 CEST 2020. DO NOT EDIT.
 
 package importexport
 
-const Credits = "github.com/0xAX/notificator;github.com/abiosoft/ishell;github.com/abiosoft/readline;github.com/allan-simon/go-singleinstance;github.com/certifi/gocertifi;github.com/chzyer/logex;github.com/chzyer/test;github.com/cucumber/godog;github.com/docker/docker-credential-helpers;github.com/emersion/go-imap;github.com/emersion/go-imap-appendlimit;github.com/emersion/go-imap-idle;github.com/emersion/go-imap-move;github.com/emersion/go-imap-quota;github.com/emersion/go-imap-specialuse;github.com/emersion/go-imap-unselect;github.com/emersion/go-mbox;github.com/emersion/go-message;github.com/emersion/go-sasl;github.com/emersion/go-smtp;github.com/emersion/go-textwrapper;github.com/emersion/go-vcard;github.com/fatih/color;github.com/flynn-archive/go-shlex;github.com/getsentry/raven-go;github.com/golang/mock;github.com/google/go-cmp;github.com/google/uuid;github.com/gopherjs/gopherjs;github.com/go-resty/resty/v2;github.com/hashicorp/go-multierror;github.com/jameskeane/bcrypt;github.com/jaytaylor/html2text;github.com/kardianos/osext;github.com/keybase/go-keychain;github.com/logrusorgru/aurora;github.com/Masterminds/semver/v3;github.com/mattn/go-runewidth;github.com/miekg/dns;github.com/myesui/uuid;github.com/nsf/jsondiff;github.com/olekukonko/tablewriter;github.com/pkg/errors;github.com/ProtonMail/bcrypt;github.com/ProtonMail/crypto;github.com/ProtonMail/docker-credential-helpers;github.com/ProtonMail/go-appdir;github.com/ProtonMail/go-apple-mobileconfig;github.com/ProtonMail/go-autostart;github.com/ProtonMail/go-imap;github.com/ProtonMail/go-imap-id;github.com/ProtonMail/gopenpgp/v2;github.com/ProtonMail/go-smtp;github.com/ProtonMail/go-vcard;github.com/ProtonMail/mbox;github.com/PuerkitoBio/goquery;github.com/sirupsen/logrus;github.com/skratchdot/open-golang;github.com/ssor/bom;github.com/stretchr/testify;github.com/therecipe/qt;github.com/twinj/uuid;github.com/urfave/cli;go.etcd.io/bbolt;golang.org/x/crypto;golang.org/x/net;golang.org/x/text;gopkg.in/stretchr/testify.v1;;Font Awesome 4.7.0;;Qt 5.13 by Qt group;"
+const Credits = "github.com/0xAX/notificator;github.com/Masterminds/semver/v3;github.com/ProtonMail/bcrypt;github.com/ProtonMail/crypto;github.com/ProtonMail/docker-credential-helpers;github.com/ProtonMail/go-appdir;github.com/ProtonMail/go-apple-mobileconfig;github.com/ProtonMail/go-autostart;github.com/ProtonMail/go-imap;github.com/ProtonMail/go-imap-id;github.com/ProtonMail/go-smtp;github.com/ProtonMail/go-vcard;github.com/ProtonMail/gopenpgp/v2;github.com/ProtonMail/mbox;github.com/PuerkitoBio/goquery;github.com/abiosoft/ishell;github.com/abiosoft/readline;github.com/allan-simon/go-singleinstance;github.com/certifi/gocertifi;github.com/chzyer/logex;github.com/chzyer/test;github.com/cucumber/godog;github.com/docker/docker-credential-helpers;github.com/emersion/go-imap;github.com/emersion/go-imap-appendlimit;github.com/emersion/go-imap-idle;github.com/emersion/go-imap-move;github.com/emersion/go-imap-quota;github.com/emersion/go-imap-specialuse;github.com/emersion/go-imap-unselect;github.com/emersion/go-mbox;github.com/emersion/go-message;github.com/emersion/go-sasl;github.com/emersion/go-smtp;github.com/emersion/go-textwrapper;github.com/emersion/go-vcard;github.com/fatih/color;github.com/flynn-archive/go-shlex;github.com/getsentry/raven-go;github.com/go-resty/resty/v2;github.com/golang/mock;github.com/google/go-cmp;github.com/google/uuid;github.com/gopherjs/gopherjs;github.com/hashicorp/go-multierror;github.com/jameskeane/bcrypt;github.com/jaytaylor/html2text;github.com/kardianos/osext;github.com/keybase/go-keychain;github.com/logrusorgru/aurora;github.com/mattn/go-runewidth;github.com/miekg/dns;github.com/myesui/uuid;github.com/nsf/jsondiff;github.com/olekukonko/tablewriter;github.com/pkg/errors;github.com/sirupsen/logrus;github.com/skratchdot/open-golang;github.com/ssor/bom;github.com/stretchr/testify;github.com/therecipe/qt;github.com/twinj/uuid;github.com/urfave/cli;go.etcd.io/bbolt;golang.org/x/crypto;golang.org/x/net;golang.org/x/text;gopkg.in/stretchr/testify.v1;;Font Awesome 4.7.0;;Qt 5.13 by Qt group;"

internal/smtp/user.go

@@ -25,7 +25,6 @@ import (
 	"io"
 	"mime"
 	"net/mail"
-	"regexp"
 	"strings"
 	"time"
 
@@ -408,9 +407,9 @@ func (su *smtpUser) handleReferencesHeader(m *pmapi.Message) (draftID, parentID
 		if !strings.Contains(reference, "@"+pmapi.InternalIDDomain) {
 			newReferences = append(newReferences, reference)
 		} else { // internalid is the parentID.
-			idMatch := regexp.MustCompile(pmapi.InternalReferenceFormat).FindStringSubmatch(reference)
+			idMatch := pmapi.RxInternalReferenceFormat.FindStringSubmatch(reference)
-			if len(idMatch) > 0 {
+			if len(idMatch) == 2 {
-				lastID := strings.TrimSuffix(strings.Trim(idMatch[0], "<>"), "@protonmail.internalid")
+				lastID := idMatch[1]
 				filter := &pmapi.MessagesFilter{ID: []string{lastID}}
 				if su.addressID != "" {
 					filter.AddressID = su.addressID

internal/store/address.go

@@ -69,7 +69,7 @@ func (storeAddress *Address) init(foldersAndLabels []*pmapi.Label) (err error) {
 			prefix := getLabelPrefix(label)
 
 			var mailbox *Mailbox
-			if mailbox, err = txNewMailbox(tx, storeAddress, label.ID, prefix, label.Name, label.Color); err != nil {
+			if mailbox, err = txNewMailbox(tx, storeAddress, label.ID, prefix, label.Path, label.Color); err != nil {
 				storeAddress.log.
 					WithError(err).
 					WithField("labelID", label.ID).

internal/store/address_mailbox.go

@@ -73,14 +73,14 @@ func (storeAddress *Address) createOrUpdateMailboxEvent(label *pmapi.Label) erro
 	prefix := getLabelPrefix(label)
 	mailbox, ok := storeAddress.mailboxes[label.ID]
 	if !ok {
-		mailbox, err := newMailbox(storeAddress, label.ID, prefix, label.Name, label.Color)
+		mailbox, err := newMailbox(storeAddress, label.ID, prefix, label.Path, label.Color)
 		if err != nil {
 			return err
 		}
 		storeAddress.mailboxes[label.ID] = mailbox
 		mailbox.store.imapMailboxCreated(storeAddress.address, mailbox.labelName)
 	} else {
-		mailbox.labelName = prefix + label.Name
+		mailbox.labelName = prefix + label.Path
 		mailbox.color = label.Color
 	}
 	return nil

internal/store/change.go

@@ -122,22 +122,10 @@ func (store *Store) imapSendUpdate(update imapBackend.Update) {
 		return
 	}
 
-	done := update.Done()
-	go func() {
-		// This timeout is to not keep running many blocked goroutines.
-		// In case nothing listens to this channel, this thread should stop.
-		select {
-		case store.imapUpdates <- update:
-		case <-time.After(1 * time.Second):
-			store.log.Warn("IMAP update could not be sent (timeout).")
-		}
-	}()
-
-	// This timeout is to not block IMAP backend by wait for IMAP client.
 	select {
-	case <-done:
 	case <-time.After(1 * time.Second):
-		store.log.Warn("IMAP update could not be delivered (timeout).")
+		store.log.Warn("IMAP update could not be sent (timeout)")
 		return
+	case store.imapUpdates <- update:
 	}
 }

internal/store/mailbox.go

@@ -142,6 +142,9 @@ func initMailboxBucket(tx *bolt.Tx, bucketName []byte) error {
 	if _, err := bucket.CreateBucketIfNotExists(apiIDsBucket); err != nil {
 		return err
 	}
+	if _, err := bucket.CreateBucketIfNotExists(deletedIDsBucket); err != nil {
+		return err
+	}
 
 	return nil
 }
@@ -240,13 +243,7 @@ func (storeMailbox *Mailbox) txGetAPIIDsBucket(tx *bolt.Tx) *bolt.Bucket {
 
 // txGetDeletedIDsBucket returns the bucket with messagesID marked as deleted
 func (storeMailbox *Mailbox) txGetDeletedIDsBucket(tx *bolt.Tx) *bolt.Bucket {
-	// There should be no error since it _...returns an error if the bucket
+	return storeMailbox.txGetBucket(tx).Bucket(deletedIDsBucket)
-	// name is blank, or if the bucket name is too long._
-	bucket, err := storeMailbox.txGetBucket(tx).CreateBucketIfNotExists(deletedIDsBucket)
-	if err != nil || bucket == nil {
-		storeMailbox.log.WithError(err).Error("Cannot create or get bucket with deleted IDs.")
-	}
-	return bucket
 }
 
 // txGetBucket returns the bucket of mailbox containing mapping buckets.

internal/store/mailbox_counts.go

@@ -125,6 +125,7 @@ func (mc *mailboxCounts) getPMLabel() *pmapi.Label {
 	return &pmapi.Label{
 		ID:        mc.LabelID,
 		Name:      mc.LabelName,
+		Path:      mc.LabelName,
 		Color:     mc.Color,
 		Order:     mc.Order,
 		Type:      pmapi.LabelTypeMailbox,
@@ -158,7 +159,7 @@ func (store *Store) createOrUpdateMailboxCountsBuckets(labels []*pmapi.Label) er
 			}
 
 			// Update mailbox info, but dont change on-API-counts.
-			mailbox.LabelName = label.Name
+			mailbox.LabelName = label.Path
 			mailbox.Color = label.Color
 			mailbox.Order = label.Order
 			mailbox.IsFolder = label.Exclusive == 1

internal/store/message.go

@@ -66,7 +66,7 @@ func (message *Message) Message() *pmapi.Message {
 // mailbox
 func (message *Message) IsMarkedDeleted() bool {
 	isMarkedAsDeleted := false
-	err := message.storeMailbox.db().Update(func(tx *bolt.Tx) error {
+	err := message.storeMailbox.db().View(func(tx *bolt.Tx) error {
 		isMarkedAsDeleted = message.storeMailbox.txGetDeletedIDsBucket(tx).Get([]byte(message.msg.ID)) != nil
 		return nil
 	})

internal/updates/updates.go

@@ -310,7 +310,9 @@ func (u *Updates) StartUpgrade(currentStatus chan<- Progress) { // nolint[funlen
 	status.UpdateDescription(InfoUpgrading)
 	switch runtime.GOOS {
 	case "windows": //nolint[goconst]
-		installerFile := strings.Split(u.winInstallerFile, "/")[1]
+		// Cannot use filepath.Base on windows it has different delimiter
+		split := strings.Split(u.winInstallerFile, "/")
+		installerFile := split[len(split)-1]
 		cmd := exec.Command("./" + installerFile) // nolint[gosec]
 		cmd.Dir = u.updateTempDir
 		status.Err = cmd.Start()

pkg/message/parser/parser.go

@@ -32,7 +32,7 @@ type Parser struct {
 func New(r io.Reader) (*Parser, error) {
 	p := new(Parser)
 
-	entity, err := message.Read(newEndOfMailTrimmer(r))
+	entity, err := message.Read(r)
 	if err != nil && !message.IsUnknownCharset(err) {
 		return nil, err
 	}

pkg/message/parser/trimmer.go

@@ -1,56 +0,0 @@
-// Copyright (c) 2020 Proton Technologies AG
-//
-// This file is part of ProtonMail Bridge.
-//
-// ProtonMail Bridge is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// ProtonMail Bridge is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-//
-// You should have received a copy of the GNU General Public License
-// along with ProtonMail Bridge.  If not, see <https://www.gnu.org/licenses/>.
-
-package parser
-
-import (
-	"bytes"
-	"io"
-)
-
-const endOfMail = "\r\n.\r\n"
-
-// endOfMailTrimmer wraps a reader to trim the End-Of-Mail indicator at the end
-// of the input, if present.
-//
-// During SMTP sending of a message, the DATA command indicates that you are
-// about to send the text (or body) of the message. The message text must end
-// with "\r\n.\r\n." I'm 99% sure that these 5 bytes should not be considered
-// part of the message body. However, some mail servers keep them as part of
-// the message, which our parser sometimes doesn't like. Therefore, we strip
-// them if we find them.
-type endOfMailTrimmer struct {
-	r   io.Reader
-	buf bytes.Buffer
-}
-
-func newEndOfMailTrimmer(r io.Reader) *endOfMailTrimmer {
-	return &endOfMailTrimmer{r: r}
-}
-
-func (r *endOfMailTrimmer) Read(p []byte) (int, error) {
-	_, err := io.CopyN(&r.buf, r.r, int64(len(p)+len(endOfMail)-r.buf.Len()))
-	if err != nil && err != io.EOF {
-		return 0, err
-	}
-
-	if err == io.EOF && bytes.HasSuffix(r.buf.Bytes(), []byte(endOfMail)) {
-		r.buf.Truncate(r.buf.Len() - len(endOfMail))
-	}
-
-	return r.buf.Read(p)
-}

pkg/message/parser/trimmer_test.go

@@ -1,55 +0,0 @@
-// Copyright (c) 2020 Proton Technologies AG
-//
-// This file is part of ProtonMail Bridge.
-//
-// ProtonMail Bridge is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-//
-// ProtonMail Bridge is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-//
-// You should have received a copy of the GNU General Public License
-// along with ProtonMail Bridge.  If not, see <https://www.gnu.org/licenses/>.
-
-package parser
-
-import (
-	"io"
-	"strings"
-	"testing"
-
-	"github.com/stretchr/testify/assert"
-)
-
-func TestEndOfMailTrimmer(t *testing.T) {
-	var tests = []struct {
-		in  string
-		out string
-	}{
-		{"string without eom", "string without eom"},
-		{"string with eom\r\n.\r\n", "string with eom"},
-		{"string with eom\r\n.\r\nin the middle", "string with eom\r\n.\r\nin the middle"},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.in, func(t *testing.T) {
-			res := dumbRead(newEndOfMailTrimmer(strings.NewReader(tt.in)))
-			assert.Equal(t, tt.out, string(res))
-		})
-	}
-}
-
-func dumbRead(r io.Reader) []byte {
-	out := []byte{}
-
-	b := make([]byte, 1)
-	for _, err := r.Read(b); err == nil; _, err = r.Read(b) {
-		out = append(out, b...)
-	}
-
-	return out
-}

pkg/message/parser/writer.go

@@ -35,7 +35,7 @@ func newWriter(root *Part) *Writer {
 
 func (w *Writer) Write(ww io.Writer) error {
 	if !w.root.is7BitClean() {
-		w.root.Header.Add("Content-Transfer-Encoding", "base64")
+		w.root.Header.Set("Content-Transfer-Encoding", "base64")
 	}
 
 	msgWriter, err := message.CreateWriter(ww, w.root.Header)
@@ -68,7 +68,7 @@ func (w *Writer) write(writer *message.Writer, p *Part) error {
 
 func (w *Writer) writeAsChild(writer *message.Writer, p *Part) error {
 	if !p.is7BitClean() {
-		p.Header.Add("Content-Transfer-Encoding", "base64")
+		p.Header.Set("Content-Transfer-Encoding", "base64")
 	}
 
 	childWriter, err := writer.CreatePart(p.Header)

pkg/message/parser_test.go

@@ -467,19 +467,6 @@ func TestParseMultipartAlternativeLatin1(t *testing.T) {
 	assert.Equal(t, "*aoeuaoeu*\n\n", plainBody)
 }
 
-func TestParseWithTrailingEndOfMailIndicator(t *testing.T) {
-	f := getFileReader("text_html_trailing_end_of_mail.eml")
-
-	m, _, plainBody, _, err := Parse(f, "", "")
-	require.NoError(t, err)
-
-	assert.Equal(t, `"Sender" <sender@sender.com>`, m.Sender.String())
-	assert.Equal(t, `"Receiver" <receiver@receiver.com>`, m.ToList[0].String())
-
-	assert.Equal(t, "<!DOCTYPE html><html><head></head><body>boo!</body></html>", m.Body)
-	assert.Equal(t, "boo!", plainBody)
-}
-
 func getFileReader(filename string) io.Reader {
 	f, err := os.Open(filepath.Join("testdata", filename))
 	if err != nil {

pkg/message/testdata/text_html_trailing_end_of_mail.eml

@@ -1,8 +0,0 @@
-From: "Sender" <sender@sender.com>
-To: "Receiver" <receiver@receiver.com>
-Content-Type: text/html; charset="utf-8"
-Content-Transfer-Encoding: base64
-MIME-Version: 1.0
-
-PCFET0NUWVBFIEhUTUw+CjxodG1sPjxib2R5PmJvbyE8L2JvZHk+PC9odG1sPg==
-.

pkg/pmapi/labels.go

@@ -80,6 +80,7 @@ const (
 type Label struct {
 	ID        string
 	Name      string
+	Path      string
 	Color     string
 	Order     int `json:",omitempty"`
 	Display   int // Not used for now, leave it empty.

pkg/pmapi/messages.go

@@ -29,6 +29,7 @@ import (
 	"net/http"
 	"net/mail"
 	"net/url"
+	"regexp"
 	"strconv"
 	"strings"
 
@@ -149,8 +150,9 @@ const ConversationIDDomain = `protonmail.conversationid`
 // InternalIDDomain is used as a placeholder for reference/message ID headers to improve compatibility with various clients.
 const InternalIDDomain = `protonmail.internalid`
 
-// InternalReferenceFormat describes format of the message ID (as regex) used for parsing reference headers.
+// RxInternalReferenceFormat is compiled regexp which describes the match for
-const InternalReferenceFormat = `(?U)<.*@` + InternalIDDomain + `>`
+// a message ID used in reference headers.
+var RxInternalReferenceFormat = regexp.MustCompile(`(?U)<(.+)@` + regexp.QuoteMeta(InternalIDDomain) + `>`) //nolint[gochecknoglobals]
 
 // Message structure.
 type Message struct {

pkg/pmapi/mocks/mocks.go

@@ -5,12 +5,11 @@
 package mocks
 
 import (
-	io "io"
-	reflect "reflect"
-
 	crypto "github.com/ProtonMail/gopenpgp/v2/crypto"
 	pmapi "github.com/ProtonMail/proton-bridge/pkg/pmapi"
 	gomock "github.com/golang/mock/gomock"
+	io "io"
+	reflect "reflect"
 )
 
 // MockClient is a mock of Client interface

test/fakeapi/controller_control.go

@@ -83,6 +83,9 @@ func (ctl *Controller) AddUserLabel(username string, label *pmapi.Label) error {
 	}
 	label.ID = ctl.labelIDGenerator.next(prefix)
 	label.Name = labelName
+	if label.Path == "" {
+		label.Path = label.Name
+	}
 	ctl.labelsByUsername[username] = append(ctl.labelsByUsername[username], label)
 	ctl.resetUsers()
 	return nil

test/fakeapi/labels.go

@@ -53,6 +53,9 @@ func (api *FakePMAPI) CreateLabel(label *pmapi.Label) (*pmapi.Label, error) {
 		prefix = "folder"
 	}
 	label.ID = api.controller.labelIDGenerator.next(prefix)
+	if label.Path == "" {
+		label.Path = label.Name
+	}
 	api.labels = append(api.labels, label)
 	api.addEventLabel(pmapi.EventCreate, label)
 	return label, nil
@@ -67,6 +70,9 @@ func (api *FakePMAPI) UpdateLabel(label *pmapi.Label) (*pmapi.Label, error) {
 			// Request doesn't have to include all properties and these have to stay the same.
 			label.Type = existingLabel.Type
 			label.Exclusive = existingLabel.Exclusive
+			if label.Path == "" {
+				label.Path = label.Name
+			}
 			api.labels[idx] = label
 			api.addEventLabel(pmapi.EventUpdate, label)
 			return label, nil

test/features/bridge/imap/mailbox/list.feature

@@ -1,11 +1,11 @@
 Feature: IMAP list mailboxes
   Background:
     Given there is connected user "user"
-    And there is "user" with mailbox "Folders/mbox1"
-    And there is "user" with mailbox "Labels/mbox2"
-    And there is IMAP client logged in as "user"
 
   Scenario: List mailboxes
+    Given there is "user" with mailbox "Folders/mbox1"
+    And there is "user" with mailbox "Labels/mbox2"
+    And there is IMAP client logged in as "user"
     When IMAP client lists mailboxes
     Then IMAP response contains "INBOX"
     Then IMAP response contains "Sent"
@@ -14,3 +14,16 @@ Feature: IMAP list mailboxes
     Then IMAP response contains "All Mail"
     Then IMAP response contains "Folders/mbox1"
     Then IMAP response contains "Labels/mbox2"
+
+  @ignore-live
+  Scenario: List mailboxes with subfolders
+    # Escaped slash in the name contains slash in the name.
+    # Not-escaped slash in the name means tree structure.
+    # We keep escaping in an IMAP communication so each mailbox is unique and
+    # both mailboxes are accessible. The slash is visible in the IMAP client.
+    Given there is "user" with mailbox "Folders/a\/b"
+    And there is "user" with mailbox "Folders/a/b"
+    And there is IMAP client logged in as "user"
+    When IMAP client lists mailboxes
+    Then IMAP response contains "Folders/a\/b"
+    Then IMAP response contains "Folders/a/b"

test/features/bridge/imap/message/update_spam.feature

@@ -0,0 +1,21 @@
+Feature: IMAP update messages in Spam folder
+  Background:
+    Given there is connected user "user"
+    # Messages are inserted in opposite way to keep increasing ID.
+    # Sequence numbers are then opposite than listed above.
+    And there are messages in mailbox "Spam" for "user"
+      | from              | to         | subject | body  | read  | starred | deleted |
+      | john.doe@mail.com | user@pm.me | foo     | hello | false | false   | false   |
+      | jane.doe@mail.com | name@pm.me | bar     | world | true  | true    | false   |
+    And there is IMAP client logged in as "user"
+    And there is IMAP client selected in "Spam"
+
+  Scenario: Mark message as read only
+    When IMAP client marks message "2" with "\Seen"
+    Then IMAP response is "OK"
+    And message "1" in "Spam" for "user" is marked as read
+    And message "1" in "Spam" for "user" is marked as unstarred
+    And API mailbox "Spam" for "user" has messages
+      | from              | to         | subject |
+      | john.doe@mail.com | user@pm.me | foo     |
+      | jane.doe@mail.com | name@pm.me | bar     |