Sled Agent should always use the "A" M.2 slot for ledgered data

[sunshowers]

Currently, Sled Agent tries to read from and write to both M.2 slots:

• sometimes scanning both disks and looking at the highest generation across them (Ledgerable trait)
• sometimes considering the boot slot as authoritative (e.g. mupdate override data)

That is incorrect in some pretty important ways:

• if one of the disks disappears temporarily, implementations of the Ledgerable trait might read old data
• if the disks couldn't be synced and the boot disk slot changes, we'll suddenly be making decisions based on outdated information

If we had an odd number of disks we could potentially address this through majority consensus. But we have two disks, and it's not really feasible to keep them in sync at all times. For this kind of data we should just always pick a specific slot -- between the A and the B slots, the natural one to pick is A.

[jclulow]

What happens if the device in the A slot fails and is unusable, but we can still boot from and use the B slot?

[andrewjstone]

What happens if the device in the A slot fails and is unusable, but we can still boot from and use the B slot?

Ledgers will fail to be read and the bootstrap agent will prevent the sled-agent from coming online. There will need to be a support call to replace the M.2. It would be the same as if both M.2's failed with respect to the ledger.

This has been a known issue for a long time with lots of discussion on and offline.

We've talked about just loading sleds with a single M.2 to solve this, but that doesn't allow us to swap disks for boot software and so is infeasible in the current update path. This is a solution that allows us to write the software to alternating M.2s but still treat ledgered configuration as if there is only a single M.2.

The trade off is that we get a known failure model vs possible corruption that is very hard to detect with 2 ledgers.

Now, what we do in the support call is still an issue. Can we recover the ledgers off the failed disk at all? If not, we probably must expunge and re-add the sled. We may also be able to regenerate the ledgers based on what's in Nexus, and trust-quorum can almost certainly re-learn its ledgered information and recompute its share as long as a reconfiguration is not already in progress. My guess is that the complexity and unknowns around higher level software make this all very unsafe though.

[andrewjstone]

It was discussed in today's control plane huddle that an RFD should be written about the issues with having ledgers utilize both M.2s and possible solutions and tradeoffs. That RFD does not currently have a proposed author or timeline given higher priority issues.

[jclulow]

FWIW, if we're considering moving to one M.2 device there isn't much on the OS side that would need to change. I left four identically sized slices at the front of the disk. We could adjust things to boot from M.2 A slice 2 or whatever instead of M.2 B slice 0 with relative ease.