We build too many walls and not enough bridges

doc/bridge.md

@@ -0,0 +1,135 @@
+# Bridge
+
+## Main blocks
+
+This is basic overview of the main bridge blocks.
+
+Note connection between IMAP/SMTP and PMAPI. IMAP and SMTP packages are in the queue to be refactored
+and we would like to try to have functionality in bridge core or bridge utilities (such as messages)
+than direct usage of PMAPI from IMAP or SMTP. Also database (BoltDB) should be moved to bridge core.
+
+```mermaid
+graph LR
+    S[Server]
+    C[Client]
+    U[User]
+
+    subgraph "Bridge app"
+        Core[Bridge core]
+        API[PMAPI]
+        Store
+        DB[BoltDB]
+        Frontend["Qt / CLI"]
+        IMAP
+        SMTP
+
+        IMAP --> Store
+        IMAP --> Core
+        SMTP --> Core
+        SMTP --> API
+        Core --> API
+        Core --> Store
+        Store --> API
+        Store --> DB
+        Frontend --> Core
+
+    end
+
+    C --> IMAP
+    C --> SMTP
+    U --> Frontend
+    API --> S
+```
+
+## Code structure
+
+More detailed graph of main types used in Bridge app and connection between them. Here is already
+communication to PMAPI only from bridge core which is not true, yet. IMAP and SMTP are still calling
+PMAPI directly.
+
+```mermaid
+graph TD
+
+    C["Client (e.g. Thunderbird)"]
+    PM[ProtonMail Server]
+
+    subgraph "Bridge app"
+        subgraph "Bridge core"
+            B[Bridge]
+            U[User]
+
+            B --> U
+        end
+
+        subgraph Store
+            StoreU[Store User]
+            StoreA[Address]
+            StoreM[Mailbox]
+
+            StoreU --> StoreA
+            StoreA --> StoreM
+        end
+
+        subgraph Credentials
+            CredStore[Store]
+            Creds[Credentials]
+
+            CredStore --> Creds
+        end
+
+        subgraph Frontend
+            CLI
+            Qt
+        end
+
+        subgraph IMAP
+            IB[IMAP backend]
+            IA[IMAP address]
+            IM[IMAP mailbox]
+
+            IB --> B
+            IB --> IA
+            IA --> IM
+            IA --> U
+            IA --> StoreA
+            IM --> StoreM
+        end
+
+        subgraph SMTP
+            SB[SMTP backend]
+            SS[SMTP session]
+
+            SB --> B
+            SB --> SS
+            SS --> U
+        end
+    end
+
+    subgraph PMAPI
+        AC[Client]
+    end
+
+    C --> IB
+    C --> SB
+
+    CLI --> B
+    Qt --> B
+
+    U --> CredStore
+    U --> Creds
+
+    U --> StoreU
+
+    StoreU --> AC
+    StoreA --> AC
+    StoreM --> AC
+
+    B --> AC
+    U --> AC
+
+    AC --> PM
+```
+
+## How to debug
+
+Run `make run-debug` which starts [Delve](https://github.com/go-delve/delve).

doc/communication.md

@@ -0,0 +1,114 @@
+# Communication
+
+## First login and sync
+
+When user logs in to the bridge for the first time, immediatelly starts the first sync.
+First sync downloads all headers of all e-mails and creates database to have proper UIDs
+and indexes for IMAP. See [database](database.md) for more information.
+
+By default, whenever it's possible, sync downloads only all e-mails maiblox which already
+have list of labels so we can construct all mailboxes (inbox, sent, trash, custom folders
+and lables) without need to download each e-mail headers many times.
+
+Note that we need to download also bodies to calculate size of the e-mail and set proper
+content type (clients uses content type for guess if e-mail contains attachment)--but only
+body, not attachment. Also it's downloaded only for the first time. After that we store
+those information in our database so next time we only sync headers, labels and so on.
+
+First sync takes some time. List of 150 messages takes about second and then we need to
+download bodies for each message. We still need to do some optimalizations. Anyway, if
+user has reasonable amount of e-mails, there is good chance user will see e-mails in the
+client right after adding account.
+
+When account is added to client, client start the sync. This sync will ask Bridge app
+for all headers (done quickly) and then starts to download all bodies and attachment.
+Unfortunatelly for some e-mail more than once if the same e-mail is in more mailboxes
+(e.g. inbox and all mail)--there is no way to tell over IMAP it's the same message.
+
+After successful login of client to IMAP, Bridge starts event loop. That periodicly ask
+servers (each 30 seconds) for new updates (new message, keys, …).
+
+```mermaid
+sequenceDiagram
+    participant S as Server
+    participant B as Bridge
+    participant C as Client
+
+    Note right of B: Set up PM account<br/>by user
+
+    loop First sync
+        B ->> S: Fetch body and attachements
+        Note right of B: Build local database<br/>(e-mail UIDs)
+    end
+
+    Note right of C: Set up IMAP/SMTP<br/>by user
+
+    C ->> B: IMAP login
+    B ->> S: Authenticate user
+    Note right of B: Create IMAP user
+
+    loop Event loop, every 30 sec
+        B ->> S: Fetch e-mail headers
+        B ->> C: Send IMAP IDLE response
+    end
+
+    C ->> B: IMAP LIST directories
+
+    loop Client sync
+        C ->> B: IMAP SELECT directory
+        C ->> B: IMAP SEARCH e-mails UIDs
+        C ->> B: IMAP FETCH of e-mail UID
+        B ->> S: Fetch body and attachements
+        Note right of B: Decrypt message<br/>and attachement
+        B ->> C: IMAP response
+    end
+```
+
+## IMAP IDLE extension
+
+IMAP IDLE is extension, it has to be supported by both client and server. IMAP server (in our case
+the bridge) supports it so clients can use it. It works by issuing `IDLE` command by the client and
+keeps the connection open. When the server has some update, server (the bridge) will respond to that
+by `EXISTS` (new message), `APPEND` (imported message), `EXPUNGE` (deleted message) or `MOVE` response.
+
+Even when there is connection with IDLE open, server can mark the client as inactive. Therefore,
+it's recommended the client should reissue the connection after each 29 minutes. This is not the
+real push and can fail!
+
+Our event loop is also simple pull and it will trigger IMAP IDLE when we get some new update from 
+the server. Would be good to have push from the server, but we need to wait for the support on API.
+
+RFC: https://tools.ietf.org/html/rfc2177
+
+```mermaid
+sequenceDiagram
+    participant S as Server
+    participant B as Bridge
+    participant C as Client
+
+    C ->> B: IMAP IDLE
+
+    loop Every 30 seconds
+        S ->> B: Checking events
+        B ->> C: IMAP response
+    end
+```
+
+## Sending e-mails
+
+E-mail are sent over standard SMTP protocol. Our bridge takes the message, encrypts and sent it
+further to our server which will then send the message to its final destination. The important
+and tricky part is encryption. See [encryption](encryption.md) or [PMEL document](https://docs.google.com/document/d/1lEBkG0DC5FOWlumInKtu4a9Cc1Eszp48ZhFy9UpPQso/edit)
+for more information.
+
+```mermaid
+sequenceDiagram
+    participant S as Server
+    participant B as Bridge
+    participant C as Client
+
+    C ->> B: SMTP send e-mail
+    Note right of B: Encrypt messages
+    B ->> S: Send encrypted e-mail
+    B ->> C: Respond OK
+```

doc/database.md

@@ -0,0 +1,27 @@
+# Database
+
+Bridge needs to have a small database to pair our IDs with IMAP UIDs and indexes. IMAP protocol
+requires every message to have an unique UID in mailbox. In this context, mailbox is not an account,
+but a folder or label. This means that one message can have more UIDs, one for each mailbox (folder), 
+and that two messages can have the same UID, but each for different mailbox (folder).
+
+IMAP index is just an index. Look at it like to an array: `["UID1", "UID2", "UID3"]`. We can access
+message by UID or index; for example index 2 and UID `UID2`. When this message is deleted, we need
+to re-index all following messages. The array will look now like `["UID1", "UID3"]` and the last
+message can be accessed by index 2 or UID `UID3`.
+
+See RFCs for more information:
+
+* https://tools.ietf.org/html/rfc822
+* https://tools.ietf.org/html/rfc3501
+
+Our database is currently built on BBolt and have those buckets (key-value storage):
+
+* Message metadata bucket:
+
+    * `[metadataBucket][API_ID] -> pmapi.Message{subject, from, to, size, other headers...}` (without body or attachment)
+
+* Mapping buckets
+
+    * `[mailboxesBucket][addressID-mailboxID][api_ids][API_ID] -> UID`
+    * `[mailboxesBucket][addressID-mailboxID][imap_ids][UID] -> API_ID`

doc/encryption.md

@@ -0,0 +1,12 @@
+# Encryption
+
+Encryption is done in PMAPI, bridge utils and bridge itself. The best would be to keep encryption
+in PMAPI and bridge utils (in pacakge such as messages). All packages are using our high-level
+GopenPGP library on top of openpgp.
+
+## `gopenpgp.KeyRing`
+
+We use one `KeyRing` per address. Our usage then contains all keys for specific address. Primary
+key is always on the first position, then there old ones to be able to decrypt last e-mail.
+Openpgp encrypts given message with all available keys, so we need to first get first (primary)
+key for encryption to have message encrypted only once with primary key.

doc/index.md

@@ -0,0 +1,9 @@
+# Documentation
+
+Documentation pages in order to read for a novice:
+
+* [Development cycle](development.md)
+* [Bridge code](bridge.md)
+* [Internal Bridge database](database.md)
+* [Communication between Bridge, Client and Server](communication.md)
+* [Encryption](encryption.md)