diff --git a/Changelog.md b/Changelog.md index 912735bf..31d97c44 100644 --- a/Changelog.md +++ b/Changelog.md @@ -4,51 +4,22 @@ Changelog [format](http://keepachangelog.com/en/1.0.0/) ## Unreleased -### Added -* GODT-360 Detect charset embedded in html/xml. - -### Changed -* GODT-354 Do not label/unlabel messsages from `All Mail` folder. -* GODT-388 Support for both bridge and import/export credentials by package users. -* GODT-387 Store factory to make store optional. -* GODT-386 Renamed bridge to general users and keep bridge only for bridge stuff. -* GODT-308 Better user error message when request is canceled. -* GODT-312 Validate recipient emails in send before asking for their public keys. -* GODT-368 Bump docker-credential-helpers version. -* GODT-280 Migrate to gopenpgp v2: - * `Unlock()` call on pmapi-client unlocks both User keys and Address keys. - * Salt is available via `AuthSalt()` method. -* GODT-394 Don't check SMTP message send time in integration tests. -* GODT-280 Migrate to gopenpgp v2. - * `Unlock()` call on pmapi-client unlocks both User keys and Address keys. - * Salt is available via `AuthSalt()` method. - -### Fixed -* GODT-356 Fix crash when removing account while mail client is fetching messages (regression from GODT-204). -* GODT-390 Don't logout user if AuthRefresh fails because internet was off. -* GODT-358 Bad timeouts with Alternative Routing. -* GODT-363 Drafts are not deleted when already created on webapp. -* GODT-390 Don't logout user if AuthRefresh fails because internet was off. -* GODT-341 Fixed flaky unittest for Store synchronization cooldown. -* Crash when failing to match necessary html element. -* Crash in message.combineParts when copying nil slice. -* Handle double charset better by using local ParseMediaType instead of mime.ParseMediaType. -* Don't remove log dir. -* GODT-422 Fix element not found (avoid listing credentials, prefer getting). -* GODT-404 Don't keep connections to proxy servers alive if user disables DoH. -* Ensure DoH is used at startup to load users for the initial auth. -* Issue causing deadlock when reloading users keys due to double-locking of a mutex. -* Correctly handle failure to unlock single key. -* GODT-479 Fix flaky integration tests. -* GODT-484 Fix infinite loop when decoding invalid 2231 charset - -## [v1.2.7] Donghai-hotfix - beta (2020-05-07) - ### Added * IMAP mailbox info update when new mailbox is created. * GODT-72 Use ISO-8859-1 encoding if charset is not specified and it isn't UTF-8. ### Changed +* GODT-360 Detect charset embedded in html/xml. +* GODT-354 Do not label/unlabel messsages from `All Mail` folder. +* GODT-388 Support for both bridge and import/export credentials by package users. +* GODT-387 Store factory to make store optional. +* GODT-386 Renamed bridge to general users and keep bridge only for bridge stuff. +* GODT-312 Validate recipient emails in send before asking for their public keys. +* GODT-368 Bump docker-credential-helpers version. +* GODT-394 Don't check SMTP message send time in integration tests. +* GODT-280 Migrate to gopenpgp v2. + * `Unlock()` call on pmapi-client unlocks both User keys and Address keys. + * Salt is available via `AuthSalt()` method. * GODT-308 Better user error message when request is canceled. * GODT-162 User Agent does not contain bridge version, only client in format `client name/client version (os)`. * GODT-258 Update go-imap to v1. @@ -72,13 +43,38 @@ Changelog [format](http://keepachangelog.com/en/1.0.0/) * Dead code from `pkg/message`. ### Fixed +* GODT-356 Fix crash when removing account while mail client is fetching messages (regression from GODT-204). +* GODT-358 Bad timeouts with Alternative Routing. +* GODT-363 Drafts are not deleted when already created on webapp. +* GODT-390 Don't logout user if AuthRefresh fails because internet was off. +* GODT-341 Fixed flaky unittest for Store synchronization cooldown. +* Crash when failing to match necessary html element. +* Crash in message.combineParts when copying nil slice. +* Handle double charset better by using local ParseMediaType instead of mime.ParseMediaType. +* Don't remove log dir. +* GODT-422 Fix element not found (avoid listing credentials, prefer getting). +* GODT-404 Don't keep connections to proxy servers alive if user disables DoH. +* Ensure DoH is used at startup to load users for the initial auth. +* Issue causing deadlock when reloading users keys due to double-locking of a mutex. +* Correctly handle failure to unlock single key. +* GODT-479 Fix flaky integration tests. +* GODT-484 Fix infinite loop when decoding invalid 2231 charset * GODT-267 Correctly detect if a message is a draft even if does not have DraftLabel. * GODT-308 Reduce minimum read speed threshold to avoid issues with flaky internet. * GODT-321 Changing address ordering would cause all messages to disappear in combined mode. * GODT-129 Fix custom message PGP by using template. -## [v1.2.7] Donghai-hotfix - beta (2020-05-07) +## [v1.2.8] Donghai-fix-append (beta 2020-06-XXX) + +### Changed +* GODT-396 reduce number of EXISTS calls. +* GODT-143 Allow appending to Sent folder when sender matches account address. + +### Fixed +* GODT-502 Fixed crash when unable to parse a message header. + +## [v1.2.7] Donghai-fix-sync - (beta 2020-05-07 live 2020-04-20) ### Added * IMAP extension MOVE with UIDPLUS support. diff --git a/internal/bridge/release_notes.go b/internal/bridge/release_notes.go index 9c310be7..02a66d94 100644 --- a/internal/bridge/release_notes.go +++ b/internal/bridge/release_notes.go @@ -15,14 +15,12 @@ // You should have received a copy of the GNU General Public License // along with ProtonMail Bridge. If not, see . -// Code generated by ./release-notes.sh at Tue 09 Jun 2020 10:21:50 AM CEST. DO NOT EDIT. +// Code generated by ./release-notes.sh at Thu 21 May 2020 07:59:59 AM CEST. DO NOT EDIT. package bridge -const ReleaseNotes = `• Changed directory for storing logs on Linux +const ReleaseNotes = ` ` -const ReleaseFixedBugs = `• Fixed sync errors due to localization -• Fixed draft deletion -• Fixed re-appearing messages in Outlook +const ReleaseFixedBugs = `• Fixed ignored import to Sent folder ` diff --git a/internal/imap/mailbox_message.go b/internal/imap/mailbox_message.go index 48fdfe02..85e7f0d5 100644 --- a/internal/imap/mailbox_message.go +++ b/internal/imap/mailbox_message.go @@ -131,10 +131,6 @@ func (im *imapMailbox) CreateMessage(flags []string, date time.Time, body imap.L // We didn't find the message in the store, so we are currently sending it. logEntry.WithField("time", date).Info("No matching UID, continuing APPEND to Sent") - - // For now we don't import user's own messages to Sent because GetUIDByHeader is not smart enough. - // This will be fixed in GODT-143. - return nil } // This is an APPEND to the Sent folder, so we will set the sent flag diff --git a/internal/store/mailbox_ids.go b/internal/store/mailbox_ids.go index 4fc170b5..a1950040 100644 --- a/internal/store/mailbox_ids.go +++ b/internal/store/mailbox_ids.go @@ -219,6 +219,11 @@ func (storeMailbox *Mailbox) GetUIDByHeader(header *mail.Header) (foundUID uint3 // in PM message. Message-Id in normal copy/move will be the PM internal ID. messageID := header.Get("Message-Id") + // There is nothing to find, when no Message-Id given. + if messageID == "" { + return uint32(0) + } + // The most often situation is that message is APPENDed after it was sent so the // Message-ID will be reflected by ExternalID in API message meta-data. externalID := strings.Trim(messageID, "<> ") // remove '<>' to improve match diff --git a/internal/store/mailbox_message.go b/internal/store/mailbox_message.go index 15fd45c0..59cd46d5 100644 --- a/internal/store/mailbox_message.go +++ b/internal/store/mailbox_message.go @@ -303,7 +303,6 @@ func (storeMailbox *Mailbox) txCreateOrUpdateMessages(tx *bolt.Tx, msgs []*pmapi seqNum, msg, ) - shouldSendMailboxUpdate = true } continue } diff --git a/internal/store/user_message_test.go b/internal/store/user_message_test.go index 55bd760b..4e28ca29 100644 --- a/internal/store/user_message_test.go +++ b/internal/store/user_message_test.go @@ -80,7 +80,7 @@ func TestCreateOrUpdateMessageMetadata(t *testing.T) { a.Equal(t, []*pmapi.Attachment(nil), msg.Attachments) a.Equal(t, int64(-1), msg.Size) a.Equal(t, "", msg.MIMEType) - a.Equal(t, mail.Header(nil), msg.Header) + a.Equal(t, make(mail.Header), msg.Header) // Change the calculated data. wantSize := int64(42) diff --git a/pkg/pmapi/events_test.go b/pkg/pmapi/events_test.go index 2ee47421..68a6272e 100644 --- a/pkg/pmapi/events_test.go +++ b/pkg/pmapi/events_test.go @@ -20,6 +20,7 @@ package pmapi import ( "fmt" "net/http" + "net/mail" "regexp" "strconv" "strings" @@ -113,6 +114,7 @@ var ( EventItem: EventItem{ID: "hdI7aIgUO1hFplCIcJHB0jShRVsAzS0AB75wGCaiNVeIHXLmaUnt4eJ8l7c7L6uk4g0ZdXhGWG5gfh6HHgAZnw==", Action: EventCreate}, Created: &Message{ ID: "hdI7aIgUO1hFplCIcJHB0jShRVsAzS0AB75wGCaiNVeIHXLmaUnt4eJ8l7c7L6uk4g0ZdXhGWG5gfh6HHgAZnw==", + Header: make(mail.Header), Subject: "Hey there", }, }, @@ -153,6 +155,7 @@ var ( EventItem: EventItem{ID: "msgID1", Action: EventCreate}, Created: &Message{ ID: "id", + Header: make(mail.Header), Subject: "Hey there", }, }, @@ -160,6 +163,7 @@ var ( EventItem: EventItem{ID: "msgID2", Action: EventCreate}, Created: &Message{ ID: "id", + Header: make(mail.Header), Subject: "Hey there again", }, }, diff --git a/pkg/pmapi/messages.go b/pkg/pmapi/messages.go index 82c6c13a..adff6099 100644 --- a/pkg/pmapi/messages.go +++ b/pkg/pmapi/messages.go @@ -24,6 +24,7 @@ import ( "encoding/base64" "encoding/json" "errors" + "fmt" "io" "net/http" "net/mail" @@ -32,6 +33,7 @@ import ( "strings" "github.com/ProtonMail/gopenpgp/v2/crypto" + "github.com/sirupsen/logrus" "golang.org/x/crypto/openpgp/packet" ) @@ -220,9 +222,13 @@ func (m *Message) UnmarshalJSON(b []byte) error { if raw.Header != "" && raw.Header != "(No Header)" { msg, err := mail.ReadMessage(strings.NewReader(raw.Header + "\r\n\r\n")) - if err == nil { - m.Header = msg.Header + if err != nil { + logrus.WithField("rawHeader", raw.Header).Trace("Failed to parse header") + return fmt.Errorf("failed to parse header of message %v: %v", m.ID, err.Error()) } + m.Header = msg.Header + } else { + m.Header = make(mail.Header) } return nil diff --git a/release-notes/bugs.txt b/release-notes/bugs.txt index af3c9508..3387c026 100644 --- a/release-notes/bugs.txt +++ b/release-notes/bugs.txt @@ -1,3 +1 @@ -• Fixed sync errors due to localization -• Fixed draft deletion -• Fixed re-appearing messages in Outlook +• Fixed ignored import to Sent folder diff --git a/release-notes/notes.txt b/release-notes/notes.txt index dbc1a7c5..e69de29b 100644 --- a/release-notes/notes.txt +++ b/release-notes/notes.txt @@ -1 +0,0 @@ -• Changed directory for storing logs on Linux diff --git a/test/README.md b/test/README.md index 0c1e3ef7..08de3544 100644 --- a/test/README.md +++ b/test/README.md @@ -36,6 +36,14 @@ graph LR We want to test Bridge app from outside as much as possible. So we mock server (API), credentials store and call commands to IMAP or SMTP the same way as client would do. +## Running tests + +In order to run Integration tests just go into the test folder `cd test` +and run `make test`. + +You can also test only specific feature (or subset of features) by using `FEATURES` environment +variable: `FEATURES=features/imap/message/create.feature make test`. + ## Example test BDD test in gherkin (cucumber) format (https://cucumber.io/docs/gherkin/reference/). @@ -99,6 +107,10 @@ we can always be sure what each steps does or should do. In the code, we separate those parts in its own files to make sure it's clear how the function should be implemented. +In the `Given` phase is also generally better to setup data (as `there are messages...`) +first, then users (`there is connected user...`) and then connections (`there is IMAP client...`). +This can prevent some hitches in internal implementation of integration tests. + ## API faked by fakeapi or liveapi We need to control what server returns. Instead of using raw JSONs, diff --git a/test/features/imap/message/create.feature b/test/features/imap/message/create.feature index ad242073..1933d25c 100644 --- a/test/features/imap/message/create.feature +++ b/test/features/imap/message/create.feature @@ -18,7 +18,6 @@ Feature: IMAP create messages | from | to | subject | read | | [primary] | john.doe@email.com | foo | true | - @ignore Scenario: Creates message sent from user's primary address Given there is IMAP client selected in "Sent" When IMAP client creates message "foo" from address "primary" of "userMoreAddresses" to "john.doe@email.com" with body "hello world" in "Sent" @@ -29,7 +28,6 @@ Feature: IMAP create messages | [primary] | john.doe@email.com | foo | true | And mailbox "INBOX" for "userMoreAddresses" has no messages - @ignore Scenario: Creates message sent from user's secondary address Given there is IMAP client selected in "Sent" When IMAP client creates message "foo" from address "secondary" of "userMoreAddresses" to "john.doe@email.com" with body "hello world" in "Sent" @@ -57,3 +55,16 @@ Feature: IMAP create messages | from | to | subject | read | | notuser@gmail.com | alsonotuser@gmail.com | foo | true | And mailbox "INBOX" for "userMoreAddresses" has no messages + + # Importing duplicate messages when messageID cannot be found in Sent already. + # + # Previously, we discarded messages for which sender matches account address to + # avoid duplicates, but this led to discarding messages imported through mail client. + Scenario: Imports a similar (duplicate) message to sent + Given there are messages in mailbox "Sent" for "userMoreAddresses" + | from | to | subject | body | + | [primary] | chosen@one.com | Meet the Twins | Hello, Mr. Anderson | + And there is IMAP client selected in "Sent" + When IMAP client creates message "Meet the Twins" from address "primary" of "userMoreAddresses" to "chosen@one.com" with body "Hello, Mr. Anderson" in "Sent" + Then IMAP response is "OK" + And mailbox "Sent" for "userMoreAddresses" has 2 messages \ No newline at end of file