[sled-agent] Integrate config-reconciler #8064

jgallagher · 2025-04-29T17:35:10Z

This PR integrates the new sled-agent-config-reconciler crate with sled-agent. It will not currently pass tests due to the reconciler not being completely implemented, but I'd like to get any feedback on this integration work itself (particularly as it pertains to the API of sled-agent-config-reconciler). See the description of #8063 for more context.

There are a couple serious warts with this PR:

The inventory system has not been updated with all the details we need to report for the reconciler. This is a bigger chunk of work because it involves a database migration and touches various bits of Nexus, so I'll do that in a separate PR.
This integration removes most uses of the StorageManager (because its functionality is being absorbed into sled-agent-config-reconciler); however, the storage manager also has a rich set of test support. This PR leaves a couple sled-agent submodules using that test support (support-bundle/storage and zone-bundle). In the long run I think it'd be better to rework these (if there are no remaining production uses of StorageManager), but for now I think this is... okay? Feedback welcome.

This is somewhat extracted from #8064, but can be landed independently and will make some of the followup sled-agent-config-reconciler PRs a little cleaner. Fixes #7774.

smklein · 2025-04-29T20:09:52Z

dev-tools/omdb/src/bin/omdb/sled_agent.rs

    #[clap(subcommand)]
    Zones(ZoneCommands),

-    /// print information about zpools


Are you expecting that inventory will supplant this info? Or are you planning on replacing this access to the sled agent later?

I was expecting that inventory would supplant this. (I think maybe it already has, in practice? I definitely only look at inventory when I'm curious about zpools; I don't think I've ever used these omdb subcommands.)

nexus/reconfigurator/planning/src/system.rs

sled-agent/src/services.rs

sled-agent/src/artifact_store.rs

sled-agent/src/long_running_tasks.rs

This is somewhat extracted from #8064, but can be landed independently and will make some of the followup sled-agent-config-reconciler PRs a little cleaner. We don't yet ledger `OmicronSledConfig`s to disk, so we're free to fiddle with the details of its fields without worrying about backwards compatibility. Fixes #7774.

…ig reconciler (#8188) The primary change here is replacing these inventory fields (a subset of `OmicronSledConfig`): ```rust pub omicron_zones: OmicronZonesConfig, pub omicron_physical_disks_generation: Generation, ``` with these: ```rust pub ledgered_sled_config: Option<OmicronSledConfig>, pub reconciler_status: ConfigReconcilerInventoryStatus, pub last_reconciliation: Option<ConfigReconcilerInventory>, ``` Once #8064 lands, all three of these will be filled in meaningfully; as of this PR, only `ledgered_sled_config` is populated. (`reconciler_status` is always `NotYetRun` and `last_reconciliation` is always `None`, since there is no reconciler yet.) The rest of the changes are all fallout from changing inventory: * Update `omdb` printing * Update sled-agent to report the new inventory fields * Update consumers of inventory (tests, reconfigurator planner, one Nexus RPW) - these all just look at `ledgered_sled_config` for now, but will need to be updated on #8064 once other fields are populated * Update database schema, model, and queries (the bulk of the diff). This requires dropping all preexisting collections, since there's no way to migrate from just `omicron_zones` to a full `OmicronSledConfig`. The first few schema migrations take care of this. Before merging I'll go through an upgrade on a racklette and confirm things come back up okay after the schema migration blows away all the pre-update inventory collections. (We think this is fine, but it'd be good to confirm.) But I think this is close enough that it's reviewable. Couple other minor changes that came along for the ride: * Closes #6770 (`inv_sled_omicron_zones` is gone now) * Fixes #8084 (added `image_source` columns to the inventory zone config table, so we don't lose `ImageSource::Artifact { hash }` values reported by sled-agent)

jgallagher · 2025-05-28T18:28:08Z

I'm putting racklette testing notes for this branch plus a few followups in comments on the last of those followups (#8220).

andrewjstone

This is a hard PR to review, given its broad scope. It was made somewhat easier by recognizing a few patterns such as replacing calls to the storage manager with rx channels for disk and datasets.

It all appears correct to me, but again, hard to really tell. I'm sure it was tedious to implement as well :)

Regardless, looks good enough to merge and continue with.

andrewjstone · 2025-05-28T17:58:13Z

sled-agent/api/src/lib.rs

-        method = GET,
-        path = "/datasets",
-    }]
-    async fn datasets_get(


Were these only used by the OMDB commands that got removed?

Yes, and I think these are obviated by all the information available via /inventory?

andrewjstone · 2025-05-28T18:05:22Z

sled-agent/src/support_bundle/storage.rs

+impl LocalStorage for ConfigReconcilerHandle {
    async fn dyn_datasets_config_list(&self) -> Result<DatasetsConfig, Error> {
-        self.datasets_config_list().await.map_err(|err| err.into())
+        // TODO-cleanup This is super gross; add a better API (maybe fetch a


Did you want to clean this up in this PR?

No, but thanks for the reminder that I need to do this. I think once this stack of work lands, the only consumers of the current StorageManager are tests (including one that implements this API); I'd like to do this cleanup along with changing those tests to interact with the config reconciler instead.

andrewjstone · 2025-05-28T18:32:09Z

nexus-sled-agent-shared/src/inventory.rs

+    /// Given a sled config, produce a reconciler result that sled-agent could
+    /// have emitted if reconciliation succeeded.
+    ///
+    /// This method should only be used by tests and dev tools; real code should


Should we mark this with #[test] and maybe #[cfg(any(test, feature = "testing"))] ?

I don't think we can, because of the "and dev tools". IIRC reconfigurator-cli and some of the "example system" stuff uses this, neither of which is gated by test / testing.

andrewjstone · 2025-05-28T18:33:44Z

nexus/reconfigurator/planning/src/planner.rs

        // reflects the parent blueprint disk generation. If it does
        // then we mark any expunged disks decommissioned.
+        //
+        // TODO-correctness We inspect `last_reconciliation` here to confirm


On balance, this seems like the right choice to me. We should know the sled agent has acted before decommissioning.

Thanks for the confirmation, I'll update the comment.

… starting zones (#8219) This dramatically reduces the work that `ServiceManager::start_omicron_zone()` does by moving most of it to the config-reconciler: * Moved: shutting down existing zone of the same name * Moved: checking for time sync * Reworked: checking datasets and choosing a root zpool (now checks are performed against the most-recently-reconciled `DatasetConfig`s, and we never choose a root zpool since all zones have a property specifying which they should use) Builds on #8064 + #8218. Fixes #8173.

papertigers

I skimmed some of the nexus bits and mostly focused on the sled-agent/support-bundle side of things and it looks okay to me.

Just noticed one comment with weird wording and a note mostly to future me about some support bundle stuff that looks like it will be an easy change once this lands.

papertigers · 2025-06-04T20:03:37Z

sled-agent/src/support_bundle/logs.rs

-            .all_sled_diagnostics_directories();
-        let tempdir = m2_debug_datasets.first().ok_or(Error::MissingStorage)?;
+        let current_internal_disks = self.internal_disks_rx.current();
+        let mut m2_debug_datasets = current_internal_disks.all_debug_datasets();


Not really related to this PR but just wanted to call out that we will be switching this to a u.2 rather than a m.2 for encryption reasons. #8197

I think once this lands I will pick up that issue and swap things around.

nexus/reconfigurator/planning/src/planner.rs

…g-reconciler-2

…8218) This is more consistent with how the reconciler remembers zones and disks. This is almost all just moving code around. The only nontrivial changes are: 1. In the spot where we ought to delete datasets, we at least "forget" them in-memory and log an error about leaking the ZFS dataset 2. The dataset serialization task remembers fewer details about datasets it ensured (just the names - that's all we need for the error checking it does when dealing with nested datasets) Staged on top of #8064. PR 1 of 3 working towards #8173.

jgallagher · 2025-06-10T12:37:04Z

Just a note before merging: The diff is now quite a bit larger than it was when this PR was reviewed, because d811bda pulls in the stack of 4 PRs that were built on top of this. They were all reviewed independently, and I'd like to land all of this at once.

@askfongjojo

@askfongjojo noticed that our `uptime`s on dogfood are all nonsense: ``` 8 BRM44220011 ok: 14:05:34 up 14066 day(s), 14:06, 1 user, load average: 5.54, 5.60, 5.62 9 BRM44220005 ok: 14:05:34 up 14066 day(s), 14:06, 1 user, load average: 17.51, 17.81, 17.81 10 BRM42220009 ok: 14:05:35 up 14066 day(s), 14:06, 1 user, load average: 15.00, 14.46, 14.01 11 BRM42220006 ok: 14:05:35 up 14066 day(s), 14:06, 0 users, load average: 11.49, 11.21, 11.10 12 BRM42220057 ok: 14:05:35 up 14066 day(s), 14:06, 0 users, load average: 2.88, 2.62, 2.03 ... ``` I _think_ #8064 introduced this. It shuffled around how time sync is checked and added a callback that the config-reconciler is supposed to run when it detects time has synchronized; that callback is responsible for rewriting `uptime` (among other things), but it never actually executes the callback. This PR fixes that. However, we have some racklettes that are running commits that include #8064 that have reasonable uptimes. I'm not sure how that's possible - is there some other way `uptime` can be correct if sled-agent doesn't fix it?

This doorbell API was used by Sled Agent prior to the config reconciler introduced in #8064. Nothing currently appears to use it.

jgallagher requested review from andrewjstone, davepacheco and smklein April 29, 2025 17:35

jgallagher added a commit that referenced this pull request Apr 29, 2025

OmicronSledConfig: Just one Generation

7cd6e49

This is somewhat extracted from #8064, but can be landed independently and will make some of the followup sled-agent-config-reconciler PRs a little cleaner. Fixes #7774.

jgallagher mentioned this pull request Apr 29, 2025

OmicronSledConfig: Just one Generation #8067

Merged

smklein reviewed Apr 29, 2025

View reviewed changes

andrewjstone reviewed Apr 30, 2025

View reviewed changes

sled-agent/src/artifact_store.rs Show resolved Hide resolved

sled-agent/src/long_running_tasks.rs Outdated Show resolved Hide resolved

jgallagher mentioned this pull request Apr 30, 2025

Introduce sled-agent-config-reconciler skeleton #8063

Merged

jgallagher force-pushed the john/sled-agent-config-reconciler-2 branch from abd7542 to 2574c5c Compare April 30, 2025 19:17

Base automatically changed from john/sled-agent-config-reconciler-1 to main May 1, 2025 12:34

jgallagher force-pushed the john/sled-agent-config-reconciler-2 branch from 2574c5c to a057195 Compare May 2, 2025 14:59

This was referenced May 15, 2025

[sled-agent-config-reconciler] Flesh out main reconicilation task #8162

Merged

[sled-agent] Don't implicitly create datasets when launching zones #8173

Closed

[inventory] Add full OmicronSledConfig and fields for upcoming config reconciler #8188

Merged

jgallagher force-pushed the john/sled-agent-config-reconciler-2 branch from a057195 to 0faddda Compare May 21, 2025 20:38

jgallagher added 9 commits May 22, 2025 11:14

[sled-agent] Integrate config-reconciler

501f74f

use new ConfigReconcilerSpawnToken interface

ce36cb7

catch up to changes on main since last rebase

385807f

fill in some TODOs that were waiting on reconciler

2abceb7

fix failing planner tests

92b4916

expectorate

78c60a3

flesh out RSS wait_for_config_reconciliation_on_sled()

6b543b8

cargo fmt

6bce900

clippy

8ff4ae3

jgallagher force-pushed the john/sled-agent-config-reconciler-2 branch from 0faddda to 8ff4ae3 Compare May 22, 2025 15:30

illumos test fix

3ab3e79

jgallagher marked this pull request as ready for review May 22, 2025 21:24

jgallagher requested a review from sunshowers May 22, 2025 21:24

This was referenced May 27, 2025

[sled-agent-config-reconciler] Add separate OmicronDatasets type #8218

Merged

[sled-agent-config-reconciler] Check zone dataset dependencies before starting zones #8219

Merged

andrewjstone approved these changes May 28, 2025

View reviewed changes

jgallagher mentioned this pull request May 30, 2025

Plan zone updates for target release #8024

Merged

2 tasks

comment update

2b57869

jgallagher added 2 commits June 9, 2025 08:20

Merge branch 'main' into john/sled-agent-config-reconciler-2

36cc3df

upgrade planner: look at running zones, not ledgered zones

ecbff26

papertigers reviewed Jun 10, 2025

View reviewed changes

jgallagher added 3 commits June 10, 2025 08:34

fix comment

c86b51b

Merge remote-tracking branch 'origin/main' into john/sled-agent-confi…

625ef9b

…g-reconciler-2

jgallagher enabled auto-merge (squash) June 10, 2025 12:40

jgallagher merged commit d1d3313 into main Jun 10, 2025
18 checks passed

jgallagher deleted the john/sled-agent-config-reconciler-2 branch June 10, 2025 14:06

jgallagher mentioned this pull request Jul 2, 2025

sled-agent-config-reconciler: actually call on_time_sync() #8509

Merged

iliana mentioned this pull request Oct 6, 2025

remove unused sled_firewall_rules_request API #9156

Merged

iliana added a commit that referenced this pull request Oct 7, 2025

remove unused sled_firewall_rules_request API (#9156)

b8efb9a

This doorbell API was used by Sled Agent prior to the config reconciler introduced in #8064. Nothing currently appears to use it.

[sled-agent] Integrate config-reconciler #8064

[sled-agent] Integrate config-reconciler #8064

Uh oh!

Conversation

jgallagher commented Apr 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgallagher commented May 28, 2025

Uh oh!

andrewjstone left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

papertigers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jgallagher commented Jun 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants