Add heartbeats and stale detection

[mcoetzee]

What does this change?

• adds a file-less/message based heartbeat mechanism that sends an hourly heartbeat message to each peer

• adds stale detection for the

  • outgoing hyperdrive
  • incoming heartbeat (from the peer)
  • incoming hyperdrive

  Periodic mirroring gets initiated when staleness is detected.

How was this implemented?

Each Hyperdrive has an associated version number that gets increased for each new entry in the drive's history. By:

• sending the outgoing drive version as the heartbeat value to the peer
• recording when and what heartbeat value was last received from the peer
• recording when and what incoming drive version was last mirrored (by the change detection mechanism)
• recording when and what outgoing drive version was last mirrored (by the change detection mechanism)

We can detect a stale incoming heartbeat, by periodically checking if the last received heartbeat is older than the heartbeat interval.

We can detect a stale incoming drive, by periodically checking if the last received heartbeat value is ahead of the last mirrored incoming drive version.

We can detect a stale outgoing drive, by periodically getting the latest outgoing drive version, and checking if it is ahead of the last mirrored outgoing drive version.