sled agent PUT /datasets should validate its constraint that (pool, kind) is unique #7311
Description
We recently realized that because datasets never get deleted today, we can never put two datasets of a particular kind on the same zpool, even if one of those belonged to a zone that's now expunged. I wanted to test this out and see what happens. Unfortunately the details don't fit in a GitHub issue description. The short version is:
- when adding a zone with a durable dataset, the planner can try to put the durable dataset on the same pool where an expunged zone's durable dataset previously was
- this fails at execution-time, with sled agent noticing the id of the dataset is wrong (thankfully), but this is only after it's committed the new dataset config to its ledger
I'm going to file a separate issue for the planner here. This issue covers having sled agent validate this constraint before accepting the request and committing the ledger. I'll comment below with details on how I tested this and how it went wrong.