IMAP UID is not unique and thus violates RFC 2060

[ajkessel]

Per RFC 2060 § 2.3.1.1, messages should have unique 32-bit values as UID. It appears that the UID exposed by Hydroxide is a sequence number based on the order of messages in the mail store. So the first message is UID 1, second UID 2, etc. If a message is added to the mail store that appears chronologically in the middle of the stored sequence, it causes all the UIDs to be bumped -- so what used to be UID 2 is now UID 3, for example. This means that IMAP agents like getmail are unable to correctly track which messages have been previously seen.

I thought Proton actually exposed a true UID for each message item; couldn't that be used for the IMAP UID rather than just a sequence number which breaks tracking?

[emersion]

The UID is coming from the key of the key-value store, so it's not a sequence number. Maybe you're mixing up IMAP UIDs and IMAP sequence numbers?

[ajkessel]

What I'm seeing from the IMAP client side is what should be a UID is just a sequence number:

server:~/.getmail> telnet localhost 1143
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR CHILDREN UNSELECT MOVE IDLE APPENDLIMIT AUTH=PLAIN] IMAP4rev1 Service Ready
1 login [redacted]
1 SELECT "INBOX"
1 FETCH * (UID)
1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR CHILDREN UNSELECT MOVE IDLE APPENDLIMIT] LOGIN completed
* FLAGS (\Seen \Deleted Bulk newsletter \Flagged \Draft BCN_import)
* OK [PERMANENTFLAGS (\Seen \Deleted Bulk newsletter \Flagged \Draft BCN_import)] Flags permitted.
* 0 RECENT
* OK [UIDNEXT 4] Predicted next UID
* OK [UIDVALIDITY 1] UIDs valid
* 3 EXISTS
1 OK [READ-WRITE] SELECT completed
* 1 FETCH (UID 1)
* 2 FETCH (UID 2)
* 3 FETCH (UID 3)
1 OK FETCH completed

As I understand it, the number after UID in the response to that fetch command should be a unique identifier, not just a sequence number. Am I missing something? The way it is currently implemented, IMAP clients like getmail can't correctly track seen messages because the UID reported in response to a fetch is not statically attached to those emails.

I added some logging to this line and it appears seqNum and uid are always the same value. So the way the emails are being added to the database originally is not putting a Proton uid in as uid but rather just the sequence number. Right?

[ajkessel]

Shouldn't this line be:

uid = unserializeUID(v)

rather than

uid = unserializeUID(k)

?

[ajkessel]

Based on some preliminary testing, that edit fixes the problem I identified where the IMAP client cannot keep track of messages when the sequence changes. Shouldn't you be unserializing the value of the UID, rather than the key?

[ajkessel]

Sorry for the serial updates -- I guess my approach is not right either. But the UID I see telnetting into the official bridge is not the same as the UID that Hydroxide is exposing.