Missing zpools after mupdate #7501
Description
A sled on dogfood is missing a zpool after a muptdate to omicron commit da66012
Sled 12 has 10 disks, and did have a pool before the mupdate.
After the mupdate it was noted that there were only 9 crucible zones on sled 12, aka BRM42220057
The sled does show all disks present:
BRM42220057 # diskinfo
TYPE DISK VID PID SIZE RMV SSD
NVME c1t00A075013275374Bd0 NVMe Micron_7300_MTFDHBG1T9TDF 1788.50 GiB no yes
NVME c2t0014EE81000D2F09d0 NVMe WUS4C6432DSP3X3 2980.82 GiB no yes
NVME c3t0014EE81000D30A3d0 NVMe WUS4C6432DSP3X3 2980.82 GiB no yes
NVME c4t0014EE81000D3061d0 NVMe WUS4C6432DSP3X3 2980.82 GiB no yes
NVME c5t0014EE81000D30B1d0 NVMe WUS4C6432DSP3X3 2980.82 GiB no yes
NVME c6t0014EE81000D3052d0 NVMe WUS4C6432DSP3X3 2980.82 GiB no yes
NVME c7t0014EE81000D2E69d0 NVMe WUS4C6432DSP3X3 2980.82 GiB no yes
NVME c8t00A0750132748192d0 NVMe Micron_7300_MTFDHBG1T9TDF 1788.50 GiB no yes
NVME c9t0014EE81000D30AAd0 NVMe WUS4C6432DSP3X3 2980.82 GiB no yes
NVME c10t0014EE81000D30B4d0 NVMe WUS4C6432DSP3X3 2980.82 GiB no yes
NVME c11t0014EE81000D3041d0 NVMe WUS4C6432DSP3X3 2980.82 GiB no yes
NVME c12t0014EE81000D2EF0d0 NVMe WUS4C6432DSP3X3 2980.82 GiB no yes
BRM42220057 # nvmeadm list
nvme0: model: Micron_7300_MTFDHBG1T9TDF, serial: 21413275374B, FW rev: 95420280, NVMe v1.3, Capacity = 1831420 MB
nvme0/1 (c1t00A075013275374Bd0): Size = 1.75 TB, Capacity = 1.75 TB, Used = 1.75 TB
nvme1: model: WUS4C6432DSP3X3, serial: A084A723, FW rev: R2210010, NVMe v1.3, Capacity = 3052360 MB
nvme1/1 (c2t0014EE81000D2F09d0): Size = 2.91 TB, Capacity = 2.91 TB, Used = 2.91 TB
nvme2: model: WUS4C6432DSP3X3, serial: A084A6C9, FW rev: R2210010, NVMe v1.3, Capacity = 3052360 MB
nvme2/1 (c3t0014EE81000D30A3d0): Size = 2.91 TB, Capacity = 2.91 TB, Used = 2.91 TB
nvme3: model: WUS4C6432DSP3X3, serial: A084A617, FW rev: R2210010, NVMe v1.3, Capacity = 3052360 MB
nvme3/1 (c4t0014EE81000D3061d0): Size = 2.91 TB, Capacity = 2.91 TB, Used = 2.91 TB
nvme4: model: WUS4C6432DSP3X3, serial: A084A60F, FW rev: R2210010, NVMe v1.3, Capacity = 3052360 MB
nvme4/1 (c5t0014EE81000D30B1d0): Size = 2.91 TB, Capacity = 2.91 TB, Used = 2.91 TB
nvme5: model: WUS4C6432DSP3X3, serial: A084A820, FW rev: R2210010, NVMe v1.3, Capacity = 3052360 MB
nvme5/1 (c6t0014EE81000D3052d0): Size = 2.91 TB, Capacity = 2.91 TB, Used = 2.91 TB
nvme6: model: WUS4C6432DSP3X3, serial: A084A643, FW rev: R2210010, NVMe v1.3, Capacity = 3052360 MB
nvme6/1 (c7t0014EE81000D2E69d0): Size = 2.91 TB, Capacity = 2.91 TB, Used = 2.91 TB
nvme7: model: Micron_7300_MTFDHBG1T9TDF, serial: 214132748192, FW rev: 95420280, NVMe v1.3, Capacity = 1831420 MB
nvme7/1 (c8t00A0750132748192d0): Size = 1.75 TB, Capacity = 1.75 TB, Used = 1.75 TB
nvme8: model: WUS4C6432DSP3X3, serial: A084A704, FW rev: R2210010, NVMe v1.3, Capacity = 3052360 MB
nvme8/1 (c9t0014EE81000D30AAd0): Size = 2.91 TB, Capacity = 2.91 TB, Used = 2.91 TB
nvme9: model: WUS4C6432DSP3X3, serial: A084A5DA, FW rev: R2210010, NVMe v1.3, Capacity = 3052360 MB
nvme9/1 (c10t0014EE81000D30B4d0): Size = 2.91 TB, Capacity = 2.91 TB, Used = 2.91 TB
nvme10: model: WUS4C6432DSP3X3, serial: A084A7DF, FW rev: R2210010, NVMe v1.3, Capacity = 3052360 MB
nvme10/1 (c11t0014EE81000D3041d0): Size = 2.91 TB, Capacity = 2.91 TB, Used = 2.91 TB
nvme11: model: WUS4C6432DSP3X3, serial: A084A5EE, FW rev: R2210010, NVMe v1.3, Capacity = 3052360 MB
nvme11/1 (c12t0014EE81000D2EF0d0): Size = 2.91 TB, Capacity = 2.91 TB, Used = 2.91 TB
The disk "missing" is c12t0014EE81000D2EF0d0 or nvme11, serial number: A084A5EE
The other disks, and pools on those disks appear to be normal.