DNS zones should provision storage in a dataset explicitly

[smklein]

How Omicron manages storage

Zones generally store data in one of two ways:

• "Anywhere within their zone filesystem" - this is the case for data such as logs, tmp directories within zones, etc
• "A durable dataset mounted at a well-known location" - this is the case for zones like Crucible, CockroachDB, and Clickhouse, which explicitly manage storage that should be durable across reboots

How DNS manages storage

The internal and external DNS services are configured to use the "anywhere within their zone filesystem" storage.

omicron/smf/internal-dns/config.toml

Line 14 in 7e8273e

storage_path = "/var/oxide/dns"

omicron/smf/external-dns/config.toml

Line 14 in 7e8273e

storage_path = "/var/oxide/dns"

In general, this data acting as a cache from Nexus is fine - after all, Nexus periodically updates these values, bumps a generation number, and should ensure that a (redundant!) number of DNS servers are storing this information.

However in the case of cold boot, it's pretty important that at least one internal DNS server boots, to provide the necessary machinery for:

• The CockroachDB instances to boot up and find each other, and
• Also for Nexus to be able to find CockroachDB

Why would this be a problem

@jclulow has identified that the "service model" for zones means that we should be able to delete/recreate zone filesystems arbitrarily, and recreate them with persistent dataset, to recreate a stable state after cold boot.

Personally, I'm on board with this, as it means there's less "reconstruction" logic the Sled Agent needs to perform after reboot (e.g., parsing of vnics, IP interfaces, etc). We can just set everything as temporary, and recreate what we need, relying only on datasets to be durable.

[smklein]

FYI @davepacheco - I can take a stab at this, but I wanted to give you a heads-up

[davepacheco]

Thanks for looking at this. I'd been wondering about this.

The intent was always that the DNS data was on persistent storage so that it could come up in a functional state without any other dependencies. (This is mentioned in RFD 248.) However, another important assumption is that the persistent storage is tied to the image -- i.e., the server would never find stuff in /var/oxide/dns that was written by a different version of the server. I know we've discussed this in the context of update and I don't recall where we landed. (We obviously could support that, but it's a lot more work, both up front and ongoing, and I think it's more brittle, riskier upgrade process.)

[jclulow]

Could you do something extremely coarse-grained, like have a "v": 1 field in the data files, or some kind of marker file in the top of the dataset, and just delete the entire contents iff it isn't the number you expect? Or you could put the DNS server commit ID in that file, if it truly isn't stable from commit to commit? That way, even if we don't get the version constraints outside right, it's not broken.

[smklein]

There seems like a massive tension between:

• "We need this information to be able to cold boot the rack"
  and
• "We cannot rely on this information to be stable"

If we need to modify the format, it should be relatively cheap to do so -- just provision a "version 2" DNS server, send records there, and have it persist the new records. However, it seems important that at least one DNS server be up, and able to respond to requests, throughout the duration of that process.

[smklein]

Either way - I think this is the only zone which stores information outside an explicit dataset, where if it was lost, could result in a loss of functionality within the rack. AFAIK all other zones filesystems could be arbitrarily deleted / re-created without issue.

[davepacheco]

Could you do something extremely coarse-grained, like have a "v": 1 field in the data files, or some kind of marker file in the top of the dataset, and just delete the entire contents iff it isn't the number you expect? Or you could put the DNS server commit ID in that file, if it truly isn't stable from commit to commit? That way, even if we don't get the version constraints outside right, it's not broken.

We can (and should) have it fail cleanly if it's not the version that it expects. I don't think we can reasonably come online if that happens because we'll be serving the wrong data (which is no data). So this is more of a fail-fast-and-clearly thing than something that will help us stay online if this happens.

There seems like a massive tension between:

* "We need this information to be able to cold boot the rack"
  and

* "We cannot rely on this information to be stable"

I think I get why it feels that way but I don't think that's true. As long as we do upgrades by provisioning a new instance and waiting for Nexus to propagate config to it before removing the old instance, we will always have one working DNS server, even if we loser power in the middle of an upgrade.

Either way - I think this is the only zone which stores information outside an explicit dataset, where if it was lost, could result in a loss of functionality within the rack. AFAIK all other zones filesystems could be arbitrarily deleted / re-created without issue.

I'm not sure what you mean by this. This data definitely needs to be stored persistently for the cold boot case. I'm not sure if that means it has to be in a dataset on a U.2 or not? (I'm not sure where the zone root filesystems come from today?) As I have said elsewhere, I think it would be good for Cockroach to work the same way, but there are tradeoffs, and it may not work out.

[jclulow]

I think I get why it feels that way but I don't think that's true. As long as we do upgrades by provisioning a new instance and waiting for Nexus to propagate config to it before removing the old instance, we will always have one working DNS server, even if we loser power in the middle of an upgrade.

As long as there is not also a hardware failure on the working DNS server, right?

---

Sean, as an aside: though we do need to do a new zoneadm install (thus getting a pristine zone root) each power cycle of the host, we could always use the same input image tar file each time we do that. So this is probably fine, as long as we can record not just that there is a DNS server with a dataset, but the exact image version that was used to provision it. But yes the persistent data then needs to be under a delegated /data or whatever, like with Cockroach and Clickhouse and so on.

I do wonder if this upgrade pattern will not become the impetus to start naming zones for some kind of UUID rather than a name like oxz_cockroach that sort of implies you can only have one on a system at a time.

[davepacheco]

I think I get why it feels that way but I don't think that's true. As long as we do upgrades by provisioning a new instance and waiting for Nexus to propagate config to it before removing the old instance, we will always have one working DNS server, even if we loser power in the middle of an upgrade.

As long as there is not also a hardware failure on the working DNS server, right?

The way I suggested upgrading, the upgrade starts by increasing the number of DNS servers above the steady-state number. So any subsequent hardware failure isn't worse than if that failure happened under steady-state. i.e., if we have deployed one DNS server in steady-state, and then do upgrades by deploying a second one and then removing the first one, it's true that we're vulnerable to a single hardware failure, but that was true before the upgrade. We would mitigate this by deploying more DNS servers. Our DNS servers should be pretty cheap, I think. If we deployed three DNS servers with the intent of surviving two hardware failures, and assuming we want to upgrade them one at a time, then during upgrade we would deploy a fourth before removing any of the first ones. At every point we would still be able to survive two hardware failures (and the system could come back online after a power loss after both hardware failures).

Sorry if I'm misunderstanding the scenario you have in mind?

I do wonder if this upgrade pattern will not become the impetus to start naming zones for some kind of UUID rather than a name like oxz_cockroach that sort of implies you can only have one on a system at a time.

Yeah, that seems useful anyway. (What if I want to deploy a multi-node cluster on a single dev system? Or even if we want to have more than one Nexus on a production sled?)

---

Getting off topic a bit, but to elaborate on the Cockroach bit: unlike our DNS server, Cockroach is designed to be able to read data written by an older version of itself. While fully supported, I would assume that's riskier than not doing that. I also expect it's risky to assume that it can read data written by a newer version, even if the upgrade to that newer version was never finalized. It seems altogether a lot safer to say that we upgrade Cockroach nodes by deploying new ones, decommissioning older ones, and waiting for the cluster to converge, to avoid asking it to read data that the current version did not write. This also preserves low-risk rollback to the existing nodes. The tradeoff I alluded to is that this could take a bit longer, use more flash lifetime, use some bandwidth, etc. I think this isn't a no-brainer either way. I mention all this by way of explanation that I don't think this pattern is exotic. I think we may want it in a few places.