[omdb] Policy to set which ClickHouse installation oximeter reads from #7522

karencfv · 2025-02-12T04:45:17Z

Overview

Implements a new policy that can set whether oximeter reads from a single node, or the replicated cluster.

Usage

Within the switch zone, omdb can be used to switch between policies

root@oxz_switch:~# omdb nexus oximeter-read-policy get
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::c]:12221
Oximeter Read Policy: 
    version: 1
    creation time: 2025-04-14 02:28:44.374824 UTC
    mode: cluster
root@oxz_switch:~# omdb nexus oximeter-read-policy set single-node -w
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::c]:12221
Successfully inserted new policy at version 2
root@oxz_switch:~# omdb nexus oximeter-read-policy get
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::c]:12221
Oximeter Read Policy: 
    version: 2
    creation time: 2025-04-14 02:49:21.936738 UTC
    mode: single-node
root@oxz_switch:~# omdb nexus blueprints regenerate -w
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::c]:12221
generated new blueprint 00d51111-3c71-4ad9-93aa-886a15b81e7a
root@oxz_switch:~# omdb nexus blueprints target set 00d51111-3c71-4ad9-93aa-886a15b81e7a enabled -w
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::c]:12221
set target blueprint to 00d51111-3c71-4ad9-93aa-886a15b81e7a

Manual testing

After deploying Omicron locally for the first time we can see the oximeter read policy is set to single node and the oximeter-reader SRV record points to the same address clickhouse-native does

root@oxz_switch:~# omdb nexus oximeter-read-policy get
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::b]:12221
Oximeter Read Policy: 
    version: 0
    creation time: 2025-04-11 01:28:06.056609 UTC
    mode: single-node

root@oxz_clickhouse_acaeeb37:~# nslookup -type=SRV _clickhouse-native._tcp.control-plane.oxide.internal
;; Got recursion not available from fd00:1122:3344:1::1, trying next server
;; Got recursion not available from fd00:1122:3344:2::1, trying next server
Server:         fd00:1122:3344:3::1
Address:        fd00:1122:3344:3::1#53

Non-authoritative answer:
_clickhouse-native._tcp.control-plane.oxide.internal    service = 0 0 9000 acaeeb37-b10d-4176-8dc7-b1869fb24520.host.control-plane.oxide.internal.

Authoritative answers can be found from:
acaeeb37-b10d-4176-8dc7-b1869fb24520.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::e

root@oxz_clickhouse_acaeeb37:~# nslookup -type=SRV _oximeter-reader._tcp.control-plane.oxide.internal
;; Got recursion not available from fd00:1122:3344:1::1, trying next server
;; Got recursion not available from fd00:1122:3344:2::1, trying next server
Server:         fd00:1122:3344:3::1
Address:        fd00:1122:3344:3::1#53

Non-authoritative answer:
_oximeter-reader._tcp.control-plane.oxide.internal      service = 0 0 9000 acaeeb37-b10d-4176-8dc7-b1869fb24520.host.control-plane.oxide.internal.

Authoritative answers can be found from:
acaeeb37-b10d-4176-8dc7-b1869fb24520.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::e

Now we deploy a clickhouse cluster, but do not change the oximeter read policy. Internal DNS remains at the same version it was

root@oxz_switch:~# omdb nexus blueprints diff target 0402da73-b8d8-4e80-94ab-204368ab8cf4
<...>
 METADATA:
    internal DNS version:   1 (unchanged)
    external DNS version:   2 (unchanged)

 OXIMETER SETTINGS:
    generation:   0 (unchanged)
    read from::   SingleNode (unchanged)

 CLICKHOUSE CLUSTER CONFIG:
+   generation:::::::::::::::::::::::::::::::::::::   2
+   max used server id:::::::::::::::::::::::::::::   3
+   max used keeper id:::::::::::::::::::::::::::::   5
+   cluster name:::::::::::::::::::::::::::::::::::   oximeter_cluster
+   cluster secret:::::::::::::::::::::::::::::::::   fad24cae-b684-43e5-aba6-74b77d708880
+   highest seen keeper leader committed log index:   0

    clickhouse keepers at generation 2:
    ------------------------------------------------
    zone id                                keeper id
    ------------------------------------------------
+   4aec089e-9096-4d6c-bea1-5d46968da1a4   1        
+   69baaac7-354c-4c19-8ee5-c7d9bd80f190   2        
+   ad47eb43-04ed-45c1-be66-291d50bbdc8e   3        
+   b2e89cdf-64ac-4cef-b7f0-1edc2c53b819   4        
+   b3ecdb92-82f3-4e86-aef6-9f354be2b92d   5        

    clickhouse servers at generation 2:
    ------------------------------------------------
    zone id                                server id
    ------------------------------------------------
+   6c927925-e716-44a8-b15d-75f80db476be   1        
+   a5538a27-cc34-4caf-99bb-0c0fbb303be9   2        
+   ed6493ce-c4fa-4f15-b844-2c82b8650ceb   3

root@oxz_switch:~# omdb nexus blueprints target set 0402da73-b8d8-4e80-94ab-204368ab8cf4 enabled -w
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::b]:12221
set target blueprint to 0402da73-b8d8-4e80-94ab-204368ab8cf4

We verify that the oximeter-reader SRV record still points to the same address as clickhouse-native

root@oxz_clickhouse_acaeeb37:~# nslookup -type=SRV _clickhouse-native._tcp.control-plane.oxide.internal
;; Got recursion not available from fd00:1122:3344:1::1, trying next server
;; Got recursion not available from fd00:1122:3344:2::1, trying next server
Server:         fd00:1122:3344:3::1
Address:        fd00:1122:3344:3::1#53

Non-authoritative answer:
_clickhouse-native._tcp.control-plane.oxide.internal    service = 0 0 9000 acaeeb37-b10d-4176-8dc7-b1869fb24520.host.control-plane.oxide.internal.

Authoritative answers can be found from:
acaeeb37-b10d-4176-8dc7-b1869fb24520.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::e

root@oxz_clickhouse_acaeeb37:~# nslookup -type=SRV _oximeter-reader._tcp.control-plane.oxide.internal
;; Got recursion not available from fd00:1122:3344:1::1, trying next server
;; Got recursion not available from fd00:1122:3344:2::1, trying next server
Server:         fd00:1122:3344:3::1
Address:        fd00:1122:3344:3::1#53

Non-authoritative answer:
_oximeter-reader._tcp.control-plane.oxide.internal      service = 0 0 9000 acaeeb37-b10d-4176-8dc7-b1869fb24520.host.control-plane.oxide.internal.

Authoritative answers can be found from:
acaeeb37-b10d-4176-8dc7-b1869fb24520.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::e

root@oxz_clickhouse_acaeeb37:~# nslookup -type=SRV _clickhouse-cluster-native._tcp.control-plane.oxide.internal
;; Got recursion not available from fd00:1122:3344:1::1, trying next server
;; Got recursion not available from fd00:1122:3344:2::1, trying next server
Server:         fd00:1122:3344:3::1
Address:        fd00:1122:3344:3::1#53

Non-authoritative answer:
_clickhouse-cluster-native._tcp.control-plane.oxide.internal    service = 0 0 9000 6c927925-e716-44a8-b15d-75f80db476be.host.control-plane.oxide.internal.
_clickhouse-cluster-native._tcp.control-plane.oxide.internal    service = 0 0 9000 a5538a27-cc34-4caf-99bb-0c0fbb303be9.host.control-plane.oxide.internal.
_clickhouse-cluster-native._tcp.control-plane.oxide.internal    service = 0 0 9000 ed6493ce-c4fa-4f15-b844-2c82b8650ceb.host.control-plane.oxide.internal.

Authoritative answers can be found from:
6c927925-e716-44a8-b15d-75f80db476be.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::27
a5538a27-cc34-4caf-99bb-0c0fbb303be9.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::28
ed6493ce-c4fa-4f15-b844-2c82b8650ceb.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::26

Now we verify queries are being directed to the single node only.

I ran several queries through OxQL and we can see then in the clickana dashboard for the single node

In the clickana dashboard for the cluster we see no queries (the initial peak is when the cluster was created)

We now change the oximeter-reader policy to cluster, we can see interal DNS has changed too

omdb nexus blueprints diff target d2a932f0-f195-432c-a94b-f8f167de100c
<...>
 METADATA:
*   internal DNS version:   1 -> 2
    external DNS version:   2 (unchanged)

 OXIMETER SETTINGS:
*   generation:   0 -> 1
*   read from::   SingleNode -> Cluster
<...>

root@oxz_switch:~# omdb nexus blueprints target set d2a932f0-f195-432c-a94b-f8f167de100c enabled -w
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::b]:12221
set target blueprint to d2a932f0-f195-432c-a94b-f8f167de100c

We verify that the oximeter-reader SRV record still points to the same address as clickhouse-cluster-native

root@oxz_clickhouse_acaeeb37:~# nslookup -type=SRV _clickhouse-native._tcp.control-plane.oxide.internal
;; Got recursion not available from fd00:1122:3344:1::1, trying next server
;; Got recursion not available from fd00:1122:3344:2::1, trying next server
Server:         fd00:1122:3344:3::1
Address:        fd00:1122:3344:3::1#53

Non-authoritative answer:
_clickhouse-native._tcp.control-plane.oxide.internal    service = 0 0 9000 acaeeb37-b10d-4176-8dc7-b1869fb24520.host.control-plane.oxide.internal.

Authoritative answers can be found from:
acaeeb37-b10d-4176-8dc7-b1869fb24520.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::e

root@oxz_clickhouse_acaeeb37:~# nslookup -type=SRV _oximeter-reader._tcp.control-plane.oxide.internal
;; Got recursion not available from fd00:1122:3344:1::1, trying next server
;; Got recursion not available from fd00:1122:3344:2::1, trying next server
Server:         fd00:1122:3344:3::1
Address:        fd00:1122:3344:3::1#53

Non-authoritative answer:
_oximeter-reader._tcp.control-plane.oxide.internal      service = 0 0 9000 6c927925-e716-44a8-b15d-75f80db476be.host.control-plane.oxide.internal.
_oximeter-reader._tcp.control-plane.oxide.internal      service = 0 0 9000 a5538a27-cc34-4caf-99bb-0c0fbb303be9.host.control-plane.oxide.internal.
_oximeter-reader._tcp.control-plane.oxide.internal      service = 0 0 9000 ed6493ce-c4fa-4f15-b844-2c82b8650ceb.host.control-plane.oxide.internal.

Authoritative answers can be found from:
6c927925-e716-44a8-b15d-75f80db476be.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::27
a5538a27-cc34-4caf-99bb-0c0fbb303be9.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::28
ed6493ce-c4fa-4f15-b844-2c82b8650ceb.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::26

root@oxz_clickhouse_acaeeb37:~# nslookup -type=SRV _clickhouse-cluster-native._tcp.control-plane.oxide.internal
;; Got recursion not available from fd00:1122:3344:1::1, trying next server
;; Got recursion not available from fd00:1122:3344:2::1, trying next server
Server:         fd00:1122:3344:3::1
Address:        fd00:1122:3344:3::1#53

Non-authoritative answer:
_clickhouse-cluster-native._tcp.control-plane.oxide.internal    service = 0 0 9000 6c927925-e716-44a8-b15d-75f80db476be.host.control-plane.oxide.internal.
_clickhouse-cluster-native._tcp.control-plane.oxide.internal    service = 0 0 9000 a5538a27-cc34-4caf-99bb-0c0fbb303be9.host.control-plane.oxide.internal.
_clickhouse-cluster-native._tcp.control-plane.oxide.internal    service = 0 0 9000 ed6493ce-c4fa-4f15-b844-2c82b8650ceb.host.control-plane.oxide.internal.

Authoritative answers can be found from:
6c927925-e716-44a8-b15d-75f80db476be.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::27
a5538a27-cc34-4caf-99bb-0c0fbb303be9.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::28
ed6493ce-c4fa-4f15-b844-2c82b8650ceb.host.control-plane.oxide.internal  has AAAA address fd00:1122:3344:101::26

Now we verify queries are being directed to the cluster only.

I ran several queries through OxQL and we can see then in the clickana dashboard for the cluster

In the clickana dashboard for the single node we see no new queries (the peak in the middle are the previous queries)

Closes: #7420, #5999, #7338

plotnick

Thanks for tackling this, it looks like a great start!

I have two general naming suggestions. First, I think we should use read instead of reads everywhere. "Reads" sounds like a verb to me, and read/write seems like a more familiar pair of terms than reads/writes.

Second, I think (but please correct me if I'm mistaken) that your version field is basically a generation number, so I think it should be called generation (or something like that) and use the provided Generation API types. We seem to be converging on semantic versions everywhere, so I think the term version should probably be reserved for those.

nexus/db-queries/src/db/datastore/oximeter_reads_policy.rs

plotnick · 2025-02-12T21:20:03Z

nexus/db-queries/src/db/datastore/oximeter_reads_policy.rs

+            .authorize(authz::Action::Modify, &authz::BLUEPRINT_CONFIG)
+            .await?;
+
+        let num_inserted = if policy.version == 1 {


Again completely up to you, but I initially had this same kind of no-rows/some-rows split in #7518, and found that it simplified a lot of things to insert a default row (single_node in this case, probably) as part of the schema migration so that there was always guaranteed to be at least one row.

karencfv

Thanks for the review @plotnick!

I have two general naming suggestions. First, I think we should use read instead of reads everywhere. "Reads" sounds like a verb to me, and read/write seems like a more familiar pair of terms than reads/writes.

Yeah, that makes sense. Now that I look at it I don't really know why I used "reads" in the first place 😄

Second, I think (but please correct me if I'm mistaken) that your version field is basically a generation number, so I think it should be called generation (or something like that) and use the provided Generation API types. We seem to be converging on semantic versions everywhere, so I think the term version should probably be reserved for those.

Hm, I used "version" because I saw it being used in other places https://github.com/oxidecomputer/omicron/blob/main/nexus/db-model/src/deployment.rs#L113-L117 https://github.com/oxidecomputer/omicron/blob/main/nexus/db-model/src/deployment.rs#L60-L70 https://github.com/oxidecomputer/omicron/blob/main/nexus/db-model/src/clickhouse_policy.rs#L31-L37 🤔

nexus/db-model/src/schema_versions.rs

andrewjstone

Karen, this looks awesome! Great job!

I only had some minor comments.

sled-agent/src/rack_setup/service.rs

schema/crdb/oximeter-read-policy/up3.sql

schema/crdb/dbinit.sql

schema/crdb/oximeter-read-policy/up3.sql

nexus/reconfigurator/planning/src/system.rs

andrewjstone · 2025-04-16T17:52:11Z

nexus/types/src/deployment/planning_input.rs

+
+#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema)]
+pub struct OximeterReadPolicy {
+    pub version: u32,


Seems like we should be using a Generation here as we do elsewhere.

Looks like it needs to be u32 because we later convert it to Sql32

nexus/db-queries/src/db/datastore/deployment.rs

andrewjstone · 2025-04-16T18:18:24Z

nexus/db-queries/src/db/datastore/oximeter_read_policy.rs

+        assert!(policy.version > 0);
+        let prev_version = policy.version - 1;
+
+        sql_query(


thanks, but I basically copy/pasted it from yours lol https://github.com/oxidecomputer/omicron/blob/main/nexus/db-queries/src/db/datastore/clickhouse_policy.rs#L126-L134

andrewjstone · 2025-04-16T18:23:42Z

nexus/internal-api/src/lib.rs

+    /// Get the current oximeter read policy
+    #[endpoint {
+            method = GET,
+            path = "/oximeter-read/policy"


I don't think that oximeter-read really means anything without the word policy. I'd make it all one path like oximeter-read-policy. That way it's only a noun.

yeah, that makes sense. I changed it to "/oximeter/read-policy" bc it seemed nicer though

nexus/internal-api/src/lib.rs

karencfv · 2025-04-17T03:01:29Z

Thanks for taking the time to review this PR @andrewjstone!

[oximeter] Policy to direct reads

cf14421

karencfv changed the title ~~[oximeter] Policy to direct DB reads~~ [oximeter] Policy to redirect DB reads Feb 12, 2025

plotnick reviewed Feb 12, 2025

View reviewed changes

nexus/db-queries/src/db/datastore/oximeter_reads_policy.rs Outdated Show resolved Hide resolved

plotnick reviewed Feb 12, 2025

View reviewed changes

karencfv commented Feb 12, 2025

View reviewed changes

read not reads

027e828

karencfv commented Feb 13, 2025

View reviewed changes

nexus/db-model/src/schema_versions.rs Outdated Show resolved Hide resolved

karencfv added 19 commits February 13, 2025 17:59

add tests and sql

a80df0f

fmt

e271d95

simplify policy table insertion

4469496

Add the files

661dab2

fix test

5c72d0c

merge main

5c0d768

Fix after merge

908e7da

Merge branch 'main' into oximeter-policy

a16fff6

fmt

0a6879c

omdb support

f7517a4

fmt

b9c0fa2

tiny fix

712d298

merge main

49b40ce

Adapt to new crdb schema crate

501c800

clippy

dbe2594

Implement oximeter policy in blueprint

ebe3a63

fix diff display

d702f33

clean up

d3112ec

expectorate tests

4e14357

karencfv mentioned this pull request Apr 9, 2025

Reduce ClickHouse log level #7946

Open

karencfv added 2 commits April 10, 2025 10:17

fix tests

5dee4fb

update openapi spec

035f8b0

karencfv added 8 commits April 10, 2025 15:23

openapi fix

423b7d1

update internal DNS

cadff5f

clean up

6ec661c

fix test

57035fe

remove option

5d50bed

fix tests

f2d3471

merge main

4e17e9e

fmt

ff85f66

karencfv changed the title ~~[oximeter] Policy to redirect DB reads~~ [omdb] Policy to set which ClickHouse installation oximeter reads from Apr 14, 2025

karencfv requested review from andrewjstone, bnaecker and jgallagher April 14, 2025 02:53

karencfv marked this pull request as ready for review April 14, 2025 02:53

andrewjstone approved these changes Apr 16, 2025

View reviewed changes

karencfv added 5 commits April 17, 2025 10:48

merge main

b884114

fmt

d5a6142

Start generations at 1 instead of 0

458163c

update API spec

3068abd

expectorate

83590c7

karencfv enabled auto-merge (squash) April 17, 2025 03:46

karencfv merged commit c281459 into oxidecomputer:main Apr 17, 2025
17 checks passed

karencfv deleted the oximeter-policy branch April 17, 2025 03:47

This was referenced Apr 17, 2025

Reconfigurator: Tracking issue for multi-node clickhouse deployment #5999

Closed

[clickhouse] Make it possible for oximeter to read/write to a clickhouse cluster and/or single node #7338

Closed

[omdb] Policy to set which ClickHouse installation oximeter reads from #7522

[omdb] Policy to set which ClickHouse installation oximeter reads from #7522

Uh oh!

Conversation

karencfv commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Usage

Manual testing

Uh oh!

plotnick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

plotnick Feb 12, 2025

Choose a reason for hiding this comment

Uh oh!

karencfv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andrewjstone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrewjstone Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

karencfv Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andrewjstone Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

karencfv Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

andrewjstone Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

karencfv Apr 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

karencfv commented Apr 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

karencfv commented Feb 12, 2025 •

edited

Loading