schema update: orchestration of the handoff from old to new Nexus

[smklein]

When the time is right, old Nexus instances need to hand off to the new ones.

• "The time is right" is probably a planner step that looks at inventory and determines that this is the next step in the update.
• When we've made this determination, old Nexus instances quiesce themselves and check on whether all other old Nexus instances are quiesced.
• Once an old Nexus instance determines that all of them are quiesced in this way, it invokes the db-handoff API on the new Nexus instances
  • EDIT: in the comment below, this handoff may actually be implicit, by reading the DB? It may not need to be an explicit handoff API

[smklein]

Discussed this a bit with @davepacheco today, here are some of our notes:

Goals

• In this schema update scenario, we're going to have old and new nexuses running at the same time. During the handoff process, either one should be able to reboot without risking database corruption.
• Before schema migration has started, the new nexuses should not access the database
• Once schema migration has started, the old nexuses should not access the database
• "Access the database" really means "should not access portions of the schema that may be mutated by a schema change" - extremely limited access to e.g. the singleton db_metadata table may be valid, but should be minimzed

Plan

• We add two booleans to db_metadata: one named quisce_started, the other named quiesce_completed
  • quiesce_started means "old nexuses should stop accessing the database, though they may have in-flight connections"
  • quiesce_completed means "old nexuses have stopped accessing the database completely, and will not access it again"

1. As a part of the update process, old nexuses will boot the "new nexus" zone, but it will not begin accessing the database because it'll notice "the schema version does not match what I need" (we're still running the old version).
2. When the old nexuses start the quiescing process, the old nexuses can set the quiesce_started bool to true. They'll then proceed to terminate their connections to the database.
3. The old nexuses can find each other via internal DNS, and communicate with each other to confirm that they have terminated their connection pools to the database. Once they have confirmed that all other nexuses have terminated, they can set the queisce_completed boolean to true. This is the point of no return - once this bool is written, responsibility jumps to the new nexus. (Yes, I'm cheating a bit - they do need a connection to access this db_metadata information - but this is an exception. When I say "terminate their connection pools", I'm referring to "everything else" -- including access to schemas that might mutate.)
4. The "new nexus" can read the quiesce_completed boolean -- when it's true, it can start the automatic schema migration process, and know that "old nexuses" are no longer running. As the last stage of schema migration, it can flip both quiesce_... bools back to false.

[davepacheco]

The "on startup" part of this (i.e., resuming quiesce when we come online and see quiesce_started = true) should happen after saga recovery to make sure that we don't start quiescing and then find out we've got some sagas to finish.

[smklein]

@davepacheco and I chatted about this today - a glaring problem with this proposal is that "it does not work if we are performing an upgrade without a schema change".

If this was to happen, the "old" and "new" nexuses would not be able to tell each other apart.

I'm working on an updated proposal to attempt to address this.