Log primary-replica resync failures

[jasontedor]

Today we do not fail a replica shard if the primary-replica resync to that replica fails. Yet, we should at least log the failure messages. This commit causes this to be the case.

Relates #24841, relates #27418

[dakrone]

LGTM

[ywelsch]

I'm not sure it's a good idea to do this. For example, when shutting down a cluster (full cluster restart), this might result in these warnings being logged, which could look alarming to users even though there is no reason to be alarmed. The primary-replica resync being best-effort at the moment anyhow, what advantage can the user gain from seeing these warnings? If it's for debugging purposes, I'm fine logging this as debug here.

[jasontedor]

@ywelsch I disagree; today we log and only log primary-replica resync completed with {} operations and that is misleading. I do not want to throw out having these error messages in the logs when there is a genuine failure (the baby) solely to not have them in the logs when this occurs during shutdown (the bathwater, which will occur rarely).

[ywelsch]

I agree that the current log message is misleading. I'm not sure though that we should log every replication failure on a primary-replica resync, though. I'll reach out to discuss.

[ywelsch]

LGTM if log-level is changed to info

[jasontedor]

I discussed this with @ywelsch and we agreed that info logging is acceptable here.