[reconfigurator] clickhouse_server SMF service and oximeter replica…

…ted mode (#6343)

## Overview

This commit introduces a few changes: 
- a new `clickhouse_server` smf service which runs the old "replicated"
mode from the `clickhouse` service
- a new `replicated` field for the oximeter configuration file which is
consumed by the `oximeter` binary that runs the replicated SQL against a
database. It now connects to the listen address from
`ServiceName::ClickhouseServer` or `ServiceName::Clickhouse` depending
which zone has been deployed.
- a new `--clickhouse-topology` build target flag which builds artifacts
based on either a `single-node` or `replicated-cluster` setup. The
difference between the two is whether the `oximeter` SMF service is
executing the `oximeter` CLI with the `--replicated` flag or not.
__CAVEAT:__ It's still necessary to manually change the RSS [node count
constants](https://github.com/oxidecomputer/omicron/blob/ffc8807caf04ca3f81b543c520ddbe26b3284264/sled-agent/src/rack_setup/plan/service.rs#L57-L77)
to the specified amount for each clickhouse topology mode. This
requirement will be short lived as we are moving to use reconfigurator.

## Usage

To run single node ClickHouse nothing changes, artifacts can be built
the same way as before.

To run replicated ClickHouse set the [node count
constants](https://github.com/oxidecomputer/omicron/blob/ffc8807caf04ca3f81b543c520ddbe26b3284264/sled-agent/src/rack_setup/plan/service.rs#L57-L77)
to the specified amount, and set the build target in the following
manner:

```console
$ cargo run --locked --release --bin omicron-package -- -t <NAME> target create -i standard -m non-gimlet -s softnpu -r single-sled -c replicated-cluster
    Finished `release` profile [optimized] target(s) in 1.03s
     Running `target/release/omicron-package -t <NAME> target create -i standard -m non-gimlet -s softnpu -r single-sled -c replicated-cluster`
Logging to: /home/coatlicue/src/omicron/out/LOG
Created new build target 'centzon' and set it as active
$ cargo run --locked --release --bin omicron-package -- -t <NAME> package
<...>
$ pfexec ./target/release/omicron-package -t <NAME> install
```

## Purpose

As laid out in [RFD 468](https://rfd.shared.oxide.computer/rfd/0468), to
roll out replicated ClickHouse we will need the ability to roll out
either replicated or single node ClickHouse for an undetermined amount
of time. This commit is a step in that direction. We need to have
separate services for running replicated or single-node ClickHouse
servers.

## Testing

Deploying omicron on a helios box with both modes.

Single node:

```console
$ cargo run --locked --release --bin omicron-package -- -t centzon target create -i standard -m non-gimlet -s softnpu -r single-sled
    Finished `release` profile [optimized] target(s) in 0.94s
     Running `target/release/omicron-package -t centzon target create -i standard -m non-gimlet -s softnpu -r single-sled`
Logging to: /home/coatlicue/src/omicron/out/LOG
Created new build target 'centzon' and set it as active
$ cargo run --locked --release --bin omicron-package -- -t centzon package
<...>
$ pfexec ./target/release/omicron-package -t centzon install
Logging to: /home/coatlicue/src/omicron/out/LOG
$ zoneadm list | grep clickhouse
oxz_clickhouse_7ce86c8b-2c9e-4d02-a857-269cb0a99c2e
root@oxz_clickhouse_7ce86c8b:~# /opt/oxide/clickhouse/clickhouse client --host fd00:1122:3344:101::e
ClickHouse client version 23.8.7.1.
Connecting to fd00:1122:3344:101::e:9000 as user default.
Connected to ClickHouse server version 23.8.7 revision 54465.

oxz_clickhouse_7ce86c8b-2c9e-4d02-a857-269cb0a99c2e.local :) SHOW TABLES FROM oximeter

SHOW TABLES FROM oximeter

Query id: 5e91fafb-4d70-4a27-a188-75fb83bb7e5e

┌─name───────────────────────┐
│ fields_bool                │
│ fields_i16                 │
│ fields_i32                 │
│ fields_i64                 │
│ fields_i8                  │
│ fields_ipaddr              │
│ fields_string              │
│ fields_u16                 │
│ fields_u32                 │
│ fields_u64                 │
│ fields_u8                  │
│ fields_uuid                │
│ measurements_bool          │
│ measurements_bytes         │
│ measurements_cumulativef32 │
│ measurements_cumulativef64 │
│ measurements_cumulativei64 │
│ measurements_cumulativeu64 │
│ measurements_f32           │
│ measurements_f64           │
│ measurements_histogramf32  │
│ measurements_histogramf64  │
│ measurements_histogrami16  │
│ measurements_histogrami32  │
│ measurements_histogrami64  │
│ measurements_histogrami8   │
│ measurements_histogramu16  │
│ measurements_histogramu32  │
│ measurements_histogramu64  │
│ measurements_histogramu8   │
│ measurements_i16           │
│ measurements_i32           │
│ measurements_i64           │
│ measurements_i8            │
│ measurements_string        │
│ measurements_u16           │
│ measurements_u32           │
│ measurements_u64           │
│ measurements_u8            │
│ timeseries_schema          │
│ version                    │
└────────────────────────────┘

41 rows in set. Elapsed: 0.014 sec.

oxz_clickhouse_7ce86c8b-2c9e-4d02-a857-269cb0a99c2e.local :) SELECT * FROM oximeter.fields_i64

SELECT *
FROM oximeter.fields_i64

Query id: 4bbcec72-101f-4cf4-9966-680381f5b62c

┌─timeseries_name────────────────────────┬───────timeseries_key─┬─field_name──┬─field_value─┐
│ http_service:request_latency_histogram │  8326032694586838023 │ status_code │         200 │
<...>

$ pfexec zlogin oxz_oximeter_b235200f-f0ad-4218-9184-d995df5acaf0
[Connected to zone 'oxz_oximeter_b235200f-f0ad-4218-9184-d995df5acaf0' pts/3]
The illumos Project     helios-2.0.22784        July 2024
root@oxz_oximeter_b235200f:~# cat /var/svc/manifest/site/oximeter/config.toml 
# Example configuration file for running an oximeter collector server

[db]
batch_size = 1000
batch_interval = 5 # In seconds
replicated = false

[log]
level = "debug"
mode = "file"
path = "/dev/stdout"
if_exists = "append"
```

Replicated cluster:

```console
$ cargo run --locked --release --bin omicron-package -- -t centzon target create -i standard -m non-gimlet -s softnpu -r single-sled -c replicated-cluster
    Finished `release` profile [optimized] target(s) in 1.03s
     Running `target/release/omicron-package -t centzon target create -i standard -m non-gimlet -s softnpu -r single-sled -c replicated-cluster`
Logging to: /home/coatlicue/src/omicron/out/LOG
Created new build target 'centzon' and set it as active
$ cargo run --locked --release --bin omicron-package -- -t centzon package
<...>
$ pfexec ./target/release/omicron-package -t centzon install
Logging to: /home/coatlicue/src/omicron/out/LOG
$ zoneadm list | grep clickhouse
oxz_clickhouse_keeper_73e7fda2-20af-4a90-9a61-c89ceed47c1a
oxz_clickhouse_server_74876663-5337-4d9b-85cb-99d1e88bdf8a
oxz_clickhouse_keeper_8eaac4f9-d9e0-4d56-b269-eab7da0c73a3
oxz_clickhouse_keeper_01f3a6af-5249-4dff-b9a4-f1076e467c9a
oxz_clickhouse_server_bc6010bf-507c-4b5a-ad4c-3a7af889a6c0
$ pfexec zlogin oxz_clickhouse_server_74876663-5337-4d9b-85cb-99d1e88bdf8a
[Connected to zone 'oxz_clickhouse_server_74876663-5337-4d9b-85cb-99d1e88bdf8a' pts/3]
The illumos Project     helios-2.0.22784        July 2024
root@oxz_clickhouse_server_74876663:~# /opt/oxide/clickhouse_server/clickhouse client --host fd00:1122:3344:101::e
ClickHouse client version 23.8.7.1.
Connecting to fd00:1122:3344:101::e:9000 as user default.
Connected to ClickHouse server version 23.8.7 revision 54465.

oximeter_cluster node 1 :) SHOW TABLES FROM oximeter

SHOW TABLES FROM oximeter

Query id: a5603063-1cbc-41a5-bfbd-33c986764e92

┌─name─────────────────────────────┐
│ fields_bool                      │
│ fields_bool_local                │
│ fields_i16                       │
│ fields_i16_local                 │
│ fields_i32                       │
│ fields_i32_local                 │
│ fields_i64                       │
│ fields_i64_local                 │
│ fields_i8                        │
│ fields_i8_local                  │
│ fields_ipaddr                    │
│ fields_ipaddr_local              │
│ fields_string                    │
│ fields_string_local              │
│ fields_u16                       │
│ fields_u16_local                 │
│ fields_u32                       │
│ fields_u32_local                 │
│ fields_u64                       │
│ fields_u64_local                 │
│ fields_u8                        │
│ fields_u8_local                  │
│ fields_uuid                      │
│ fields_uuid_local                │
│ measurements_bool                │
│ measurements_bool_local          │
│ measurements_bytes               │
│ measurements_bytes_local         │
│ measurements_cumulativef32       │
│ measurements_cumulativef32_local │
│ measurements_cumulativef64       │
│ measurements_cumulativef64_local │
│ measurements_cumulativei64       │
│ measurements_cumulativei64_local │
│ measurements_cumulativeu64       │
│ measurements_cumulativeu64_local │
│ measurements_f32                 │
│ measurements_f32_local           │
│ measurements_f64                 │
│ measurements_f64_local           │
│ measurements_histogramf32        │
│ measurements_histogramf32_local  │
│ measurements_histogramf64        │
│ measurements_histogramf64_local  │
│ measurements_histogrami16        │
│ measurements_histogrami16_local  │
│ measurements_histogrami32        │
│ measurements_histogrami32_local  │
│ measurements_histogrami64        │
│ measurements_histogrami64_local  │
│ measurements_histogrami8         │
│ measurements_histogrami8_local   │
│ measurements_histogramu16        │
│ measurements_histogramu16_local  │
│ measurements_histogramu32        │
│ measurements_histogramu32_local  │
│ measurements_histogramu64        │
│ measurements_histogramu64_local  │
│ measurements_histogramu8         │
│ measurements_histogramu8_local   │
│ measurements_i16                 │
│ measurements_i16_local           │
│ measurements_i32                 │
│ measurements_i32_local           │
│ measurements_i64                 │
│ measurements_i64_local           │
│ measurements_i8                  │
│ measurements_i8_local            │
│ measurements_string              │
│ measurements_string_local        │
│ measurements_u16                 │
│ measurements_u16_local           │
│ measurements_u32                 │
│ measurements_u32_local           │
│ measurements_u64                 │
│ measurements_u64_local           │
│ measurements_u8                  │
│ measurements_u8_local            │
│ timeseries_schema                │
│ timeseries_schema_local          │
│ version                          │
└──────────────────────────────────┘

81 rows in set. Elapsed: 0.010 sec. 

oximeter_cluster node 1 :) SELECT * FROM oximeter.fields_i64

SELECT *
FROM oximeter.fields_i64

Query id: 14f07468-0e33-4de1-8893-df3e11eb7660

┌─timeseries_name────────────────────────┬───────timeseries_key─┬─field_name──┬─field_value─┐
│ http_service:request_latency_histogram │   436117616059041516 │ status_code │         200 │
<...>

$ pfexec zlogin oxz_oximeter_bcba1c06-1ca5-49cf-b277-8c2387975274
[Connected to zone 'oxz_oximeter_bcba1c06-1ca5-49cf-b277-8c2387975274' pts/3]
The illumos Project     helios-2.0.22784        July 2024
root@oxz_oximeter_bcba1c06:~# cat /var/svc/manifest/site/oximeter/config.toml
# Example configuration file for running an oximeter collector server

[db]
batch_size = 1000
batch_interval = 5 # In seconds
replicated = true

[log]
level = "debug"
mode = "file"
path = "/dev/stdout"
if_exists = "append"

```

Related: #5999

internal-dns-cli/src/bin/dnswait.rs

@@ -36,15 +36,17 @@ struct Opt {
 #[value(rename_all = "kebab-case")]
 enum ServiceName {
     Cockroach,
-    Clickhouse,
     ClickhouseKeeper,
+    ClickhouseServer,
 }
 
 impl From<ServiceName> for internal_dns::ServiceName {
     fn from(value: ServiceName) -> Self {
         match value {
             ServiceName::Cockroach => internal_dns::ServiceName::Cockroach,
-            ServiceName::Clickhouse => internal_dns::ServiceName::Clickhouse,
+            ServiceName::ClickhouseServer => {
+                internal_dns::ServiceName::ClickhouseServer
+            }
             ServiceName::ClickhouseKeeper => {
                 internal_dns::ServiceName::ClickhouseKeeper
             }

internal-dns/src/config.rs

@@ -510,6 +510,10 @@ mod test {
             ServiceName::ClickhouseKeeper.dns_name(),
             "_clickhouse-keeper._tcp",
         );
+        assert_eq!(
+            ServiceName::ClickhouseServer.dns_name(),
+            "_clickhouse-server._tcp",
+        );
         assert_eq!(ServiceName::Cockroach.dns_name(), "_cockroach._tcp",);
         assert_eq!(ServiceName::InternalDns.dns_name(), "_nameservice._tcp",);
         assert_eq!(ServiceName::Nexus.dns_name(), "_nexus._tcp",);

nexus/test-utils/src/lib.rs

@@ -1428,6 +1428,7 @@ pub async fn start_oximeter(
         address: Some(SocketAddr::new(Ipv6Addr::LOCALHOST.into(), db_port)),
         batch_size: 10,
         batch_interval: 1,
+        replicated: false,
     };
     let config = oximeter_collector::Config {
         nexus_address: Some(nexus_address),

oximeter/collector/src/agent.rs

@@ -17,7 +17,6 @@ use futures::TryStreamExt;
 use internal_dns::resolver::Resolver;
 use internal_dns::ServiceName;
 use nexus_client::types::IdSortMode;
-use omicron_common::address::CLICKHOUSE_PORT;
 use omicron_common::backoff;
 use omicron_common::backoff::BackoffError;
 use oximeter::types::ProducerResults;
@@ -380,6 +379,7 @@ impl OximeterAgent {
         db_config: DbConfig,
         resolver: &Resolver,
         log: &Logger,
+        replicated: bool,
     ) -> Result<Self, Error> {
         let (result_sender, result_receiver) = mpsc::channel(8);
         let log = log.new(o!(
@@ -393,10 +393,15 @@ impl OximeterAgent {
         // database.
         let db_address = if let Some(address) = db_config.address {
             address
+        } else if replicated {
+            SocketAddr::V6(
+                resolver
+                    .lookup_socket_v6(ServiceName::ClickhouseServer)
+                    .await?,
+            )
         } else {
-            SocketAddr::new(
-                resolver.lookup_ip(ServiceName::Clickhouse).await?,
-                CLICKHOUSE_PORT,
+            SocketAddr::V6(
+                resolver.lookup_socket_v6(ServiceName::Clickhouse).await?,
             )
         };
 
@@ -422,7 +427,6 @@ impl OximeterAgent {
                 ..
             }) => {
                 debug!(log, "oximeter database does not exist, creating");
-                let replicated = client.is_oximeter_cluster().await?;
                 client
                     .initialize_db_with_version(
                         replicated,

oximeter/collector/src/lib.rs

@@ -78,12 +78,18 @@ pub struct DbConfig {
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub address: Option<SocketAddr>,
 
-    /// Batch size of samples at which to insert
+    /// Batch size of samples at which to insert.
     pub batch_size: usize,
 
     /// Interval on which to insert data into the database, regardless of the number of collected
     /// samples. Value is in seconds.
     pub batch_interval: u64,
+
+    // TODO (https://github.com/oxidecomputer/omicron/issues/4148): This field
+    // should be removed if single node functionality is removed.
+    /// Whether ClickHouse is running as a replicated cluster or
+    /// single-node server.
+    pub replicated: bool,
 }
 
 impl DbConfig {
@@ -95,12 +101,16 @@ impl DbConfig {
     /// ClickHouse.
     pub const DEFAULT_BATCH_INTERVAL: u64 = 5;
 
+    /// Default ClickHouse topology.
+    pub const DEFAULT_REPLICATED: bool = false;
+
     // Construct config with an address, using the defaults for other fields
     fn with_address(address: SocketAddr) -> Self {
         Self {
             address: Some(address),
             batch_size: Self::DEFAULT_BATCH_SIZE,
             batch_interval: Self::DEFAULT_BATCH_INTERVAL,
+            replicated: Self::DEFAULT_REPLICATED,
         }
     }
 }
@@ -207,6 +217,7 @@ impl Oximeter {
                     config.db,
                     &resolver,
                     &log,
+                    config.db.replicated,
                 )
                 .await?,
             ))

package-manifest.toml

@@ -140,13 +140,15 @@ source.type = "local"
 source.rust.binary_names = ["oximeter", "clickhouse-schema-updater"]
 source.rust.release = true
 source.paths = [
-  { from = "smf/oximeter", to = "/var/svc/manifest/site/oximeter" },
   { from = "oximeter/db/schema", to = "/opt/oxide/oximeter/schema" },
+  { from = "smf/oximeter/{{clickhouse-topology}}/config.toml", to = "/var/svc/manifest/site/oximeter/config.toml" },
+  { from = "smf/oximeter/manifest.xml", to = "/var/svc/manifest/site/oximeter/manifest.xml" },
 ]
 output.type = "zone"
 output.intermediate_only = true
 
 [package.clickhouse]
+# This service runs a single-node ClickHouse server.
 service_name = "clickhouse"
 only_for_targets.image = "standard"
 source.type = "composite"
@@ -169,13 +171,45 @@ source.paths = [
   { from = "out/clickhouse", to = "/opt/oxide/clickhouse" },
   { from = "smf/clickhouse/manifest.xml", to = "/var/svc/manifest/site/clickhouse/manifest.xml" },
   { from = "smf/clickhouse/method_script.sh", to = "/opt/oxide/lib/svc/manifest/clickhouse.sh" },
-  { from = "smf/clickhouse/config_replica.xml", to = "/opt/oxide/clickhouse/config.d/config_replica.xml" },
+]
+output.type = "zone"
+output.intermediate_only = true
+setup_hint = "Run `cargo xtask download clickhouse` to download the necessary binaries"
+
+[package.clickhouse_server]
+# This service runs a server for a replicated ClickHouse cluster.
+# It is complimentary to the clickhouse_keeper service.
+# One cannot be run without the other.
+service_name = "clickhouse_server"
+only_for_targets.image = "standard"
+source.type = "composite"
+source.packages = [
+  "clickhouse_server_svc.tar.gz",
+  "internal-dns-cli.tar.gz",
+  "omicron-clickhouse-admin.tar.gz",
+  "zone-setup.tar.gz",
+  "zone-network-install.tar.gz"
+]
+output.type = "zone"
+
+[package.clickhouse_server_svc]
+service_name = "clickhouse_server_svc"
+only_for_targets.image = "standard"
+source.type = "local"
+source.paths = [
+  { from = "out/clickhouse", to = "/opt/oxide/clickhouse_server" },
+  { from = "smf/clickhouse_server/manifest.xml", to = "/var/svc/manifest/site/clickhouse_server/manifest.xml" },
+  { from = "smf/clickhouse_server/method_script.sh", to = "/opt/oxide/lib/svc/manifest/clickhouse_server.sh" },
+  { from = "smf/clickhouse_server/config_replica.xml", to = "/opt/oxide/clickhouse_server/config.d/config_replica.xml" },
 ]
 output.type = "zone"
 output.intermediate_only = true
 setup_hint = "Run `cargo xtask download clickhouse` to download the necessary binaries"
 
 [package.clickhouse_keeper]
+# This service runs a keeper for a replicated ClickHouse cluster.
+# It is complimentary to the clickhouse_server service.
+# One cannot be run without the other.
 service_name = "clickhouse_keeper"
 only_for_targets.image = "standard"
 source.type = "composite"

package/src/bin/omicron-package.rs

@@ -265,12 +265,19 @@ async fn do_target(
         format!("failed to create directory {}", target_dir)
     })?;
     match subcommand {
-        TargetCommand::Create { image, machine, switch, rack_topology } => {
+        TargetCommand::Create {
+            image,
+            machine,
+            switch,
+            rack_topology,
+            clickhouse_topology,
+        } => {
             let target = KnownTarget::new(
                 image.clone(),
                 machine.clone(),
                 switch.clone(),
                 rack_topology.clone(),
+                clickhouse_topology.clone(),
             )?;
 
             let path = get_single_target(&target_dir, name).await?;

package/src/lib.rs

@@ -68,6 +68,21 @@ pub enum TargetCommand {
         /// fail in a single-sled environment. `single-sled` relaxes this
         /// requirement.
         rack_topology: crate::target::RackTopology,
+
+        #[clap(
+            short,
+            long,
+            default_value = Some("single-node"),
+            required = false
+        )]
+        // TODO (https://github.com/oxidecomputer/omicron/issues/4148): Remove
+        // once single-node functionality is removed.
+        /// Specify whether clickhouse will be deployed as a replicated cluster
+        /// or single-node configuration.
+        ///
+        /// Replicated cluster configuration is an experimental feature to be
+        /// used only for testing.
+        clickhouse_topology: crate::target::ClickhouseTopology,
     },
     /// List all existing targets
     List,

package/src/target.rs

@@ -62,13 +62,26 @@ pub enum RackTopology {
     SingleSled,
 }
 
+/// Topology of the ClickHouse installation within the rack.
+#[derive(Clone, Debug, strum::EnumString, strum::Display, ValueEnum)]
+#[strum(serialize_all = "kebab-case")]
+#[clap(rename_all = "kebab-case")]
+pub enum ClickhouseTopology {
+    /// Use configurations suitable for a replicated ClickHouse cluster deployment.
+    ReplicatedCluster,
+
+    /// Use configurations suitable for a single-node ClickHouse deployment.
+    SingleNode,
+}
+
 /// A strongly-typed variant of [Target].
 #[derive(Clone, Debug)]
 pub struct KnownTarget {
     image: Image,
     machine: Option<Machine>,
     switch: Option<Switch>,
     rack_topology: RackTopology,
+    clickhouse_topology: ClickhouseTopology,
 }
 
 impl KnownTarget {
@@ -77,6 +90,7 @@ impl KnownTarget {
         machine: Option<Machine>,
         switch: Option<Switch>,
         rack_topology: RackTopology,
+        clickhouse_topology: ClickhouseTopology,
     ) -> Result<Self> {
         if matches!(image, Image::Trampoline) {
             if machine.is_some() {
@@ -93,7 +107,7 @@ impl KnownTarget {
             bail!("'switch=asic' is only valid with 'machine=gimlet'");
         }
 
-        Ok(Self { image, machine, switch, rack_topology })
+        Ok(Self { image, machine, switch, rack_topology, clickhouse_topology })
     }
 }
 
@@ -104,6 +118,7 @@ impl Default for KnownTarget {
             machine: Some(Machine::NonGimlet),
             switch: Some(Switch::Stub),
             rack_topology: RackTopology::MultiSled,
+            clickhouse_topology: ClickhouseTopology::SingleNode,
         }
     }
 }
@@ -119,6 +134,10 @@ impl From<KnownTarget> for Target {
             map.insert("switch".to_string(), switch.to_string());
         }
         map.insert("rack-topology".to_string(), kt.rack_topology.to_string());
+        map.insert(
+            "clickhouse-topology".to_string(),
+            kt.clickhouse_topology.to_string(),
+        );
         Target(map)
     }
 }
@@ -140,6 +159,7 @@ impl std::str::FromStr for KnownTarget {
         let mut machine = None;
         let mut switch = None;
         let mut rack_topology = None;
+        let mut clickhouse_topology = None;
 
         for (k, v) in target.0.into_iter() {
             match k.as_str() {
@@ -155,6 +175,9 @@ impl std::str::FromStr for KnownTarget {
                 "rack-topology" => {
                     rack_topology = Some(v.parse()?);
                 }
+                "clickhouse-topology" => {
+                    clickhouse_topology = Some(v.parse()?);
+                }
                 _ => {
                     bail!(
                         "Unknown target key {k}\nValid keys include: [{}]",
@@ -173,6 +196,7 @@ impl std::str::FromStr for KnownTarget {
             machine,
             switch,
             rack_topology.unwrap_or(RackTopology::MultiSled),
+            clickhouse_topology.unwrap_or(ClickhouseTopology::SingleNode),
         )
     }
 }

sled-agent/src/rack_setup/plan/service.rs

@@ -58,14 +58,23 @@ const OXIMETER_COUNT: usize = 1;
 // when Nexus provisions Clickhouse.
 // TODO(https://github.com/oxidecomputer/omicron/issues/4000): Use
 // omicron_common::policy::CLICKHOUSE_SERVER_REDUNDANCY once we enable
-// replicated ClickHouse
+// replicated ClickHouse.
+// Set to 0 when testing replicated ClickHouse.
 const CLICKHOUSE_COUNT: usize = 1;
 // TODO(https://github.com/oxidecomputer/omicron/issues/732): Remove
 // when Nexus provisions Clickhouse keeper.
 // TODO(https://github.com/oxidecomputer/omicron/issues/4000): Use
 // omicron_common::policy::CLICKHOUSE_KEEPER_REDUNDANCY once we enable
 // replicated ClickHouse
+// Set to 3 when testing replicated ClickHouse.
 const CLICKHOUSE_KEEPER_COUNT: usize = 0;
+// TODO(https://github.com/oxidecomputer/omicron/issues/732): Remove
+// when Nexus provisions Clickhouse server.
+// TODO(https://github.com/oxidecomputer/omicron/issues/4000): Use
+// omicron_common::policy::CLICKHOUSE_SERVER_REDUNDANCY once we enable
+// replicated ClickHouse.
+// Set to 2 when testing replicated ClickHouse
+const CLICKHOUSE_SERVER_COUNT: usize = 0;
 // TODO(https://github.com/oxidecomputer/omicron/issues/732): Remove.
 // when Nexus provisions Crucible.
 const MINIMUM_U2_COUNT: usize = 3;
@@ -628,6 +637,47 @@ impl Plan {
             });
         }
 
+        // Provision Clickhouse server zones, continuing to stripe across sleds.
+        // TODO(https://github.com/oxidecomputer/omicron/issues/732): Remove
+        // Temporary linter rule until replicated Clickhouse is enabled
+        #[allow(clippy::reversed_empty_ranges)]
+        for _ in 0..CLICKHOUSE_SERVER_COUNT {
+            let sled = {
+                let which_sled =
+                    sled_allocator.next().ok_or(PlanError::NotEnoughSleds)?;
+                &mut sled_info[which_sled]
+            };
+            let id = OmicronZoneUuid::new_v4();
+            let ip = sled.addr_alloc.next().expect("Not enough addrs");
+            // TODO: This may need to be a different port if/when to have single node
+            // and replicated running side by side as per stage 1 of RFD 468.
+            let port = omicron_common::address::CLICKHOUSE_PORT;
+            let address = SocketAddrV6::new(ip, port, 0, 0);
+            dns_builder
+                .host_zone_with_one_backend(
+                    id,
+                    ip,
+                    ServiceName::ClickhouseServer,
+                    port,
+                )
+                .unwrap();
+            let dataset_name =
+                sled.alloc_dataset_from_u2s(DatasetType::ClickhouseServer)?;
+            let filesystem_pool = Some(dataset_name.pool().clone());
+            sled.request.zones.push(OmicronZoneConfig {
+                // TODO-cleanup use TypedUuid everywhere
+                id: id.into_untyped_uuid(),
+                underlay_address: ip,
+                zone_type: OmicronZoneType::ClickhouseServer {
+                    address,
+                    dataset: OmicronZoneDataset {
+                        pool_name: dataset_name.pool().clone(),
+                    },
+                },
+                filesystem_pool,
+            });
+        }
+
         // Provision Clickhouse Keeper zones, continuing to stripe across sleds.
         // TODO(https://github.com/oxidecomputer/omicron/issues/732): Remove
         // Temporary linter rule until replicated Clickhouse is enabled