Resent messages are not moved to DeltaChat folder

[link2xt]

Delta Chat has a "resend" function exposed via dc_resend_msgs CFFI API and resend_messages JSON-RPC API. This results in resending the same message with old Message-ID so it is received as a new message by new chat members and as a duplicate by old chat members.

If old chat member has mvbox_move option enabled, original message is moved to DeltaChat folder, but this is not true for duplicates. We should probably move duplicates to DeltaChat folder as well.

It may be annoying if you try to move a message from DeltaChat to some other folder and Delta Chat finds it and moves back into DeltaChat each time, but still better than leaving duplicates in the Inbox. If someone complains, we can rethink it.

[iequidoo]

Why not delete duplicates at all? In case if we mistake that it's just a duplicate? Anyway it's not seen in the client, so probably useless because can't be read in another MUA if it can't decrypt it.

EDIT: Ok, if the user moves a message to another folder, it's not a duplicate. But maybe we can have some heuristic for this?

[link2xt]

Why not delete duplicates at all? In case if we mistake that it's just a duplicate?

This is how it worked before and resulted in issues when user moved the message manually: #3157
The problem is that original message may already be removed, in which case this new message is not a duplicate. It's safer to keep all copies and remember them, then when we want to delete the message from the server just try to delete all copies we know about.

[link2xt]

Actually I have explicitly removed moving of the message twice in #4544 because of some server that had INBOX and INBOX.DeltaChat pointing to the same folder, resulting in infinite moving of the message from INBOX to itself, assigning new UID over and over.

So we need a better solution for this "resend" function. Maybe actually assign a new Message-ID to each resent copy but somehow indicate that the message is a retransmission.

[adbenitez]

This is how it worked before and resulted in issues when user moved the message manually: #3157

ugh? are re-sent message copies not deleted from inbox? does this mean that every time I resend a game in the gamers group (big message due to all updates over time etc) these copies while not redownloaded they are kept polluting the INBOX without way to remove them in DC?

also it often happens that someone doesn't have a game I resend the game but it turns out they had the message already but in pending download state that expired, and trying to download the message fails, that is why I was thinking duplicates are removed, because to actually download the resent message they need to delete the old message in pending-download state and then I have to resend again so they get it as a new message

[link2xt]

Duplicates should still be noticed and added to imap table, so deleted when the message is deleted from IMAP server.

[iequidoo]

So we need a better solution for this "resend" function. Maybe actually assign a new Message-ID to each resent copy but somehow indicate that the message is a retransmission.

I think this might not be good for compatibility with other MUAs, they also can resend and receive message copies.

As for duplicates deletion, we can check that the old message is still there before deletion of the new one. Or maybe use the INTERNALDATE attribute, probably it's not changed on message moving, but as we can't be sure, we can use this heuristic only for known providers before a duplicate deletion.

[link2xt]

I think this might not be good for compatibility with other MUAs, they also can resend and receive message copies.

Thunderbird already displays all the duplicates without caring about them having the same Message-ID. The problem would be with assigning replies to the message, especially webxdc updates to webxdc instance if it is a webxdc instance that is resent.

As for duplicates deletion, we can check that the old message is still there before deletion of the new one.

This is not reliable e.g. in multi-device case if a message is deleted automatically from one (low-memory device) and later you receive a duplicate, for one device this would be an original message and for the other one a duplicate. Also this does not solve the problem of aliased folders, e.g. INBOX and INBOX.DeltaChat pointing to the same place. Instead of infinite move loop as in #4544 you will check that the message exists both in INBOX and INBOX.DeltaChat, then delete duplicate from one of the folders and end up with no copies at all because it was a single message.

Or maybe use the INTERNALDATE attribute, probably it's not changed on message moving, but as we can't be sure, we can use this heuristic only for known providers before a duplicate deletion.

INTERNALDATE should not change when messages is moved/copied, this is why SAVEDATE extension exists.

[link2xt]

So I have checked, currently when we resend a message we also preserve a Date.
How about this solution: we change the Date to the current one every time we resend a message so it appears with a different date for those who don't have it. It does not really matter which date is displayed IMO, maybe even less confusing that it is not some old date. Then we can use this Date to detect if the message is actually a resent message or not. If we have not seen message with such Message-ID and Date we consider it a new message wrt. moving, but then when we decide whether we already have message we only look at the Message-ID and can decide not to download it or deduplicate if it is passed into receive_imf anyway somehow.

Thanks to "smeared timestamp" mechanics we are never going to resend the same message with the same Date twice.

Apparently Microsoft Exchange uses Message-ID + Date combination in deduplication for different reasons, because some clients don't generate unique Message-IDs: https://techcommunity.microsoft.com/t5/exchange-team-blog/how-does-duplicate-detection-work/ba-p/610180 We have not seen such problems so I'd say it does not matter for us and we can still deduplicate by only Message-ID, I only propose to use Date field when deciding whether we have seen this copy of the message or not and whether we want to move it or not.

I also don't think INTERNALDATE is a good idea because if the server is offline for some time it may receive a bunch of messages afterwards and assign them all the same INTERNALDATE. But Date field Delta Chat can ensure to be unique and it is anyway unlikely that someone clicks "resend" twice in a second or immediately after sending a message and excepts this to work, even bots should not do this.

[iequidoo]

Thanks to "smeared timestamp" mechanics we are never going to resend the same message with the same Date twice.

But there's an interesting comment in the code:

/// Returns the current smeared timestamp,                                                                                                                                                                           
///                                                                                                                                                                                                                  
/// The returned timestamp MUST NOT be sent out.                                                                                                                                                                     
pub(crate) fn smeared_time(context: &Context) -> i64 {

Would be interesting to know the reasons. @link2xt @r10s

[link2xt]

This is because you need to use create_smeared_timestamp if you want to use the returned timestamp in the Date field.