Using Etebase/Etesync for encrypted e-mail storage

[bohwaz]

First: thank you for Etesync!

Having contacts, calendars and notes is great, but for a complete PIM, encrypted storage of e-mails would be even better.

E-mails can be encrypted using PGP, but if stored as-is, the metadata is still in plaintext. So using E2E encryption would be better to ensure personal data is kept private.

Etebase is currently the best option to do that, because it is already providing other PIM services, but also because the logic is very similar.

This would allow to create email clients similar to what Protonmail offers, but with a completely open source stack.

What would be different for Etebase is that with email you need a few more things than what currently exists in Etebase. This message is very long, I'm sorry for this. This issue is posted to receive feedback and try to see if E-mail storage inside Etebase is a viable idea (I think it is). It is completely open to discussion on all the proposed ideas and suggestions I am making.

I am in no way saying the Etebase developer or anyone else should implement all of the things I am discussing. I am just sharing my thoughts, and if people agree on something, we might be able to pay some of the developments required for this solution (we are a small NGO, don't expect millions).

I did spend a long time in 2023 working on an idea of an encrypted mail server. Etebase does exactly the same things that I was proposing in my project, so it seems to be a good idea to use Etebase for email as well, instead of reinventing the wheel :)

Let me list the features you would expect from an encrypted email server, and whether they would require changes to Etebase or not.

Feature	Requires changes to Etebase	Requires development of other server-side software
Receive messages	Yes?	Yes
List messages per folder	No	No
List all messages	Maybe?	No
Move messages between folders	Maybe	No
Delete messages	No	No
Store drafts and sent messages in specific folders	No	No
Search in messages (offline too)	Maybe?	No
Send messages	No	Yes (shared auth with Etebase)
Store, list and fetch attachments	No	No
Applying filters and rules on incoming messages (eg. Sieve)	No	No
Exchange of public PGP keys with senders (eg. Web Key Directory)	No?	Yes (shared auth with Etebase)
Discovery of recipients public PGP key (via Web Key Directory)	No	No
Being able to recover access to your account if you forget your password (eg. recovery passphrase)	Yes	No
Automatic removal of messages from the server after a set number of days	Yes?	Maybe?
Being able to access the messages using their regular desktop client (eg. Thunderbird, Claws-mail, etc.)	No	No (but a IMAP bridge client)
Optional: storage of your private PGP key so that you don't have to copy it from device to device	No	No
Optional : share an encrypted message with someone who does not have PGP	No	Yes (shared auth with Etebase)

I might have missed features and details in this write-up, don't hesitate to add more.

Now I will try to explain how each of these features could be handled with Etebase and what is currently missing in Etebase.

Receiving messages

1. The message is received by a SMTP server (MTA), which will block any invalid DKIM/SPF, or obvious spam, it could also modify the message to add headers if the message is likely to be spam.
2. The MTA passes the message to a mail delivery agent (MDA). The MDA is a client of Etebase, but does not have access to a user password.
3. The MDA will fetch the public Etebase key of the user (recipient).
4. The MDA will encrypt (and sign?) the received message, and store it in the user "inbox".

When the user uses its Etesync email client:

1. The client will then list and fetch all messages from this "inbox"
2. If the message is PGP-encrypted or signed, it will decrypt and/or verify the message, and store the result in the message metadata (we don't need to keep the message as encrypted, as it will be stored encrypted in Etebase)
3. The client will apply filters (eg. move message to folder, or delete message, or auto-reply, etc.)
4. The client will extract attachments and store them (and add the attachments IDs to the message metadata)
5. The message is then moved in the desired folder (according to the rules)

As far as I understand, in Etebase, messages could be stored as items of a collection. Each folder would be a collection, eg. etebase.mail.folder, with a metadata giving its name.

Attachments can be stored in a etebase.mail.attachments collection.

A specific collection, eg. etebase.mail.inbox could be created to store incoming messages (as items).

What seems to be currently missing in Etebase is the ability for an external client to deliver items inside a collection without having access to the user password, only by using its public key.

The MTA could be a regular Exim/Postfix server calling a specific server-side script. There is no requirement for Etebase to directly handle pipe data from the MTA (but it would make it easier to integrate Etebase with the MTA).

List messages per folder

This could be done using existing Etebase API routes to fetch items of a collection.

What seems to be missing is the ability to fetch only the metadata of an item (eg. if you just want to fetch the list of messages, but not their contents).

List all messages

This would be trickier to do as if you have dozens of folders, this would imply fetching the list of messages of all folders collections.

Another solution would be to store all the messages in the same collection, and have the folder being only a metadata of each item. But then you would have to list all messages to know which ones are in a folder when you click on a folder.

Removing the idea of folders, and using only a notion of "tags" would make this a good option. But this would mean developing a IMAP/etebase bridge would be hard to do.

It also seems that most email users are using folders, so it would be make it hard to migrate people from regular email to Etebase email storage.

Move messages between folders

Etebase currently lacks the ability to move an item between two collections. If the idea of a separate collection for each folder is abandoned, then it would be easy to move a message, it would just mean a change in its metadata.

Delete messages

No issue here I think.

Store drafts and sent messages in specific folders

This would not be an issue either.

Search in messages (offline too)

This is the major issue of any E2E encrypted software. Protonmail gets around this by not encrypting some metadata (subject, date, sender).

Various cryptography ideas exist, but a solution would need to be easy to implement so that clients don't struggle to implement this.

The easiest idea would be to store a search index in a specific collection, eg. etebase.mail.search, where each item contains a key-value index of words matching IDs of messages items. The clients would fetch this collection to be able to make searches without fetching all messages. When a message is deleted or added to a folder, a new item would be created in the search collection (this would be a "delta", instructing clients to update their index), so that you would get an up-to-date index. From time to time, a client could "compact" all the deltas to a new exhaustive index to reduce the amount of items in this collection. This is just an idea, and probably not the best. But it would not require any change to Etebase itself. But the index would probably perform badly

Send messages

This could be handled directly by a MTA.

If handled by the MTA, the encrypted mail client would have to store the sent message in the "Sent" folder itself.

The MTA would still need to be able to authenticate the sending user. This would require for Etebase to expose something that allows a trusted software to verify that a user is authenticated, without having for the user to send their password. This is discussed in other features below.

Store, list and fetch attachments

This can be done by using a etebase.mail.attachments collection. No issue for Etebase.

Separating the attachments makes it quicker to fetch messages for mail clients. This also allows the user to delete attachments without modifying the messages.

I'm not sure how we could handle finding and updating any message that is using the deleted attachment though. Maybe the attachment would need to also contain a list of messages that are using this attachment, and update the message metadata as well.

But on the other side, this means we no longer keep an exact copy of the received message. I don't think this is an issue as I chose above to already decrypt PGP-encrypted messages instead of storing them encrypted. This also means that a potential IMAP bridge would have to "rebuild" messages from the message itself, and its attachments, to make a complete raw message body.

Applying filters and rules on incoming messages (eg. Sieve)

As seen above, the filtering would be done by the client, this does not concern Etebase. A specific Etebase collection could be created, eg. etebase.mail.filters.

Each item would be a Sieve file.

Using the Sieve language would mean it is easier to migrate to/from regular email servers.

The difficulty would be for the encrypted email client to parse the Sieve file.

Exchange of public PGP keys with senders (eg. Web Key Directory)

2 options here:

1. Etebase handles the storage of public PGP keys for each of its users, and it will have a HTTPS endpoint for the Web Key Directory to expose these public PGP keys, but currently Etebase does not seem to allow the storage of unencrypted data for the users.
2. Etebase purely stays a storage server. A specific front-end would need to be developed, store the PGP public key of the user and expose it via a WKD.

The second option seems harder to do as this front-end would not have access to Etebase authentication to know if a user is correctly authenticated and allowed to update its stored public PGP key.

We would need to find a solution for Etebase to somehow tell this front-end "this user is authenticated and they have this username" (which matches an email address).

Discovery of recipients public PGP key (using Web Key Directory)

This has to be done by the encrypted email client.

It will then store the public key inside the users contacts, or maybe in a specific collection etebase.mail.keys.

Being able to recover access to your account if you forget your password (eg. recovery passphrase)

Etebase does not seem to offer any recovery option currently.

Maybe having the ability for the user to have multiple passwords, and one of them being a recovery passphrase, would solve the issue? That way you would store the passphrase separately, and use it just like a password.

For example the request to /api/v1/authentication/login_challenge/ could be:

{"username": "XXX", "type": "recovery"}

This is quite likely to be another issue for Etebase, as this might have its own security implications.

Automatic removal of messages from the server after a set number of days

There is no way to do this currently in Etebase as items don't have a date AFAIK.

Doing it on the client side only would defeat the purpose of this feature.

Not sure if another server component could store the expiration date of an item and delete this item from Etebase without having the users password.

It might make sense to add an "autodelete" feature to Etebase where you specify the expiry date of an item and Etebase handles the deletion.

Being able to access the messages using their regular desktop client (eg. Thunderbird, Claws-mail, etc.)

Most of the discussed features would allow to create IMAP bridge software (from my limited knowledge of the IMAP protocol). For SMTP there is no need for a bridge I think, but I'm not sure. This software would act as a client on a local computer or a on a server for example if you want to expose a IMAP server to your users.

Optional: storage of your private PGP key so that you don't have to copy it from device to device

This could be done in a collection like etebase.mail.settings that could store other client-side settings and preferences.

Optional : share an encrypted message with someone who does not have PGP

Example: store the message on a server, and expose it via HTTPS, and encrypt it with a passwordr, then send the link to the stored message to the recipient, who will be able to use a web browser to decrypt it.

This is a feature that exists in Protonmail. This would quite likely be feasible by a companion server component to Etebase, but the issue remains that this component currently can not have access to Etebase authentication.

Etebase would probably require a component that exposes to another trusted server, "this session is trusted and matches this username" without having to expose the current session and allow the server to access the contents of the users Etebase collections. Maybe using invitations could work? I'm not sure I need to read more.