[reconfigurator] clickhouse-admin endpoints to generate ClickHouse config files #6476

karencfv · 2024-08-29T08:35:33Z

Overview

This commit introduces two new endpoints to the clickhouse-admin API. One to generate a ClickHouse server XML configuration file and the other to generate a ClickHouse keeper XML configuration file.

Purpose

"Reconfigurator" will need to call these endpoints to generate the necessary files when bringing up new clickhouse_server and clickhouse_keeper zones. They will also be necessary to update existing zones with the keeper and server information from new zones.

API structure

Moving forward the API endpoints will follow the following structure:

/keeper/ : For endpoints that will perform actions on keeper nodes solely.
/server/ : For endpoints that will perform actions on server nodes solely.

Usage

To generate a server XML config file:

$ curl -X put http://example/server/config \ 
-H "Content-Type: application/json" \
-d '{"generation": {GENERATION_NUM}, settings: {"id": {ID}, "keepers": [{"ipv6|ipv4|domain": "{ADDRESS}"}], "remote_servers": [{"ipv6|ipv4|domain": "{ADDRESS}"}], "config_dir": "{CONFIG_PATH}", "datastore_path": "{DATA_PATH}", "listen_addr": "{LISTEN_ADDRESS}" }}'

To generate a keeper XML config file:

$ curl -X put http://example/keeper/config \ 
-H "Content-Type: application/json" \
-d '{"generation": {GENERATION_NUM}, settings: {"node_id": {ID}, "raft_servers": [{"id": {NODE_ID}, "host": {"ipv6|ipv4|domain": "{ADDRESS}"}}], "config_dir": "{CONFIG_PATH}", "datastore_path": "{DATA_PATH}", "listen_addr": "{LISTEN_ADDRESS}"}}'

Caveats

For the time being the generation number isn't used for anything. In a follow up PR we will use it to keep track of the configuration version to make sure we're not generating an incorrect XML configuration file.

Testing

Dropshot server:

$ cargo run --bin=clickhouse-admin -- run -c ./smf/clickhouse-admin/config.toml -a [::1]:8888
   Compiling omicron-clickhouse-admin v0.1.0 (/Users/karcar/src/omicron/clickhouse-admin)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.52s
     Running `target/debug/clickhouse-admin run -c ./smf/clickhouse-admin/config.toml -a '[::1]:8888'`
note: configured to log to "/dev/stdout"
{"msg":"listening","v":0,"name":"clickhouse-admin","level":30,"time":"2024-09-02T07:32:00.106783Z","hostname":"ixchel","pid":11318,"local_addr":"[::1]:8888","component":"dropshot","file":"/Users/karcar/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205"}
{"msg":"accepted connection","v":0,"name":"clickhouse-admin","level":30,"time":"2024-09-02T07:32:17.626437Z","hostname":"ixchel","pid":11318,"local_addr":"[::1]:8888","component":"dropshot","file":"/Users/karcar/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775","remote_addr":"[::1]:51896"}
Path { inner: GenerationNum { generation: Generation(123) } }
{"msg":"request completed","v":0,"name":"clickhouse-admin","level":30,"time":"2024-09-02T07:32:17.628323Z","hostname":"ixchel","pid":11318,"uri":"/node/server/generate-config/123","method":"post","req_id":"0b623f6b-441c-4418-9189-99504fbea5bf","remote_addr":"[::1]:51896","local_addr":"[::1]:8888","component":"dropshot","file":"/Users/karcar/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914","latency_us":1044,"response_code":"201"}
{"msg":"accepted connection","v":0,"name":"clickhouse-admin","level":30,"time":"2024-09-02T07:34:07.829199Z","hostname":"ixchel","pid":11318,"local_addr":"[::1]:8888","component":"dropshot","file":"/Users/karcar/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775","remote_addr":"[::1]:52117"}
Path { inner: GenerationNum { generation: Generation(123) } }
{"msg":"request completed","v":0,"name":"clickhouse-admin","level":30,"time":"2024-09-02T07:34:07.830355Z","hostname":"ixchel","pid":11318,"uri":"/node/keeper/generate-config/123","method":"post","req_id":"1c729e0c-04bf-4e75-af0a-b23eef40f20f","remote_addr":"[::1]:52117","local_addr":"[::1]:8888","component":"dropshot","file":"/Users/karcar/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914","latency_us":985,"response_code":"201"}

Generate server XML file:

$ curl -X put http://[::1]:8888/server/config -H "Content-Type: application/json" -d '{"generation": 3, "settings": {"id": 45, "keepers": [{"ipv6": "ff::01"}, {"ipv4": "127.0.0.1"}, {"domain_name": "hi.there"}], "remote_servers": [{"ipv6": "ff::08"}, {"ipv4": "127.0.0.2"}], "config_dir": "./", "datastore_path": "./", "listen_addr": "::1" }}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   858  100   602  100   256   210k  91789 --:--:-- --:--:-- --:--:--  418k
{
  "logger": {
    "level": "trace",
    "log": "./log/clickhouse.log",
    "errorlog": "./log/clickhouse.err.log",
    "size": 100,
    "count": 1
  },
  "macros": {
    "shard": 1,
    "replica": 45,
    "cluster": "oximeter_cluster"
  },
  "listen_host": "::1",
  "http_port": 8123,
  "tcp_port": 9000,
  "interserver_http_port": 9009,
  "remote_servers": {
    "cluster": "oximeter_cluster",
    "secret": "some-unique-value",
    "replicas": [
      {
        "host": {
          "ipv6": "ff::8"
        },
        "port": 9000
      },
      {
        "host": {
          "ipv4": "127.0.0.2"
        },
        "port": 9000
      }
    ]
  },
  "keepers": {
    "nodes": [
      {
        "host": {
          "ipv6": "ff::1"
        },
        "port": 9181
      },
      {
        "host": {
          "ipv4": "127.0.0.1"
        },
        "port": 9181
      },
      {
        "host": {
          "domain_name": "hi.there"
        },
        "port": 9181
      }
    ]
  },
  "data_path": "./data"
}

Generate keeper XML file:

$ curl -X put http://[::1]:8888/keeper/config -H "Content-Type: application/json" -d '{"generation": 3, "settings": {"id": 1, "raft_servers": [{"id": 1, "host": {"ipv6": "ff::01"}}, {"id": 2, "host":{"ipv4": "127.0.0.1"}}, {"id": 3, "host":{"domain_name": "hi.there"}}], "config_dir": "./", "datastore_path": "./", "listen_addr": "::1" }}' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   820  100   568  100   252   301k   133k --:--:-- --:--:-- --:--:--  800k
{
  "logger": {
    "level": "trace",
    "log": "./log/clickhouse-keeper.log",
    "errorlog": "./log/clickhouse-keeper.err.log",
    "size": 100,
    "count": 1
  },
  "listen_host": "::1",
  "tcp_port": 9181,
  "server_id": 1,
  "log_storage_path": "./coordination/log",
  "snapshot_storage_path": "./coordination/snapshots",
  "coordination_settings": {
    "operation_timeout_ms": 10000,
    "session_timeout_ms": 30000,
    "raft_logs_level": "trace"
  },
  "raft_config": {
    "servers": [
      {
        "id": 1,
        "hostname": {
          "ipv6": "ff::1"
        },
        "port": 9234
      },
      {
        "id": 2,
        "hostname": {
          "ipv4": "127.0.0.1"
        },
        "port": 9234
      },
      {
        "id": 3,
        "hostname": {
          "domain_name": "hi.there"
        },
        "port": 9234
      }
    ]
  }
}

Related: #5999 , #3824

karencfv · 2024-09-04T04:54:32Z

Heya @andrewjstone 👋 Could I get a review when you get a chance?

andrewjstone

Generally looks good @karencfv . Great progress!

andrewjstone · 2024-09-04T16:27:00Z

clickhouse-admin/api/src/lib.rs

+    /// directory.
+    #[endpoint {
+        method = POST,
+        path = "/node/keeper/generate-config/{generation}",


A few notes on the API structure here:

We should be using REST style APIs with nouns, not verbs, similar to the rest of our APIs. I suggest /keeper/config.

These APIs are also idempotent and should be PUTs.

The generation number should be part of the payload, and not a query parameter. We typically do this by wrapping the inner type in a container type with generation number.

The /node prefix seems somewhat superfluous. If we need a sharable endpoint for something like stats we can add it and name it precisely without a prefix.

andrewjstone · 2024-09-04T16:56:32Z

clickhouse-admin/src/http_entrypoints.rs

+    ) -> Result<HttpResponseCreated<KeeperConfig>, HttpError> {
+        let ctx = rqctx.context();
+        let keeper_settings = body.into_inner();
+        let output = ctx.clickward().generate_keeper_config(keeper_settings)?;


What does the returned HttpError look like on conversion here?

$ curl -X put http://[::1]:8888/keeper/config -H "Content-Type: application/json" -d '{"generation": 3, "settings": {"node_id": 1, "raft_servers": [{"id": 1, "host": {"ipv6": "ff::01"}}, {"id": 2, "host":{"ipv4": "127.0.0.1"}}, {"id": 3, "host":{"domain_name": "hi.there"}}], "config_dir": "./bob/bob", "datastore_path": "./", "listen_addr": "::1" }}' | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 439 100 175 100 264 67281 99k --:--:-- --:--:-- --:--:-- 214k { "request_id": "f79d5187-6ed8-4b55-ab1f-fb1b5aa010c6", "error_code": "Internal", "message": "clickward XML generation failure: No such file or directory (os error 2)" }

Hmm. That looks pretty good. I'm assuming Internal means 500, which is probably correct here. I'm not an expert in http response codes 😄.

haha yep! https://github.com/oxidecomputer/omicron/blob/main/clickhouse-admin/src/clickward.rs#L26 https://docs.rs/http/latest/http/status/struct.StatusCode.html#associatedconstant.INTERNAL_SERVER_ERROR

andrewjstone · 2024-09-04T16:59:26Z

clickhouse-admin/types/src/lib.rs

-    pub id: KeeperId,
-    pub raft_servers: Vec<RaftServerConfig>,
+    /// Unique ID of the keeper
+    pub node_id: KeeperId,


keeper_id ?

How about just id? It feels a bit redundant otherwise?

Works for me.

clickhouse-admin/api/src/lib.rs

clickhouse-admin/src/http_entrypoints.rs

karencfv

Thanks for taking a look @andrewjstone ! I think I've addressed all your comments. Please let me know if you think there's anything missing :)

clickhouse-admin/api/src/lib.rs

clickhouse-admin/src/http_entrypoints.rs

karencfv · 2024-09-04T23:42:35Z

clickhouse-admin/src/http_entrypoints.rs

+    ) -> Result<HttpResponseCreated<KeeperConfig>, HttpError> {
+        let ctx = rqctx.context();
+        let keeper_settings = body.into_inner();
+        let output = ctx.clickward().generate_keeper_config(keeper_settings)?;


$ curl -X put http://[::1]:8888/keeper/config -H "Content-Type: application/json" -d '{"generation": 3, "settings": {"node_id": 1, "raft_servers": [{"id": 1, "host": {"ipv6": "ff::01"}}, {"id": 2, "host":{"ipv4": "127.0.0.1"}}, {"id": 3, "host":{"domain_name": "hi.there"}}], "config_dir": "./bob/bob", "datastore_path": "./", "listen_addr": "::1" }}' | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 439 100 175 100 264 67281 99k --:--:-- --:--:-- --:--:-- 214k { "request_id": "f79d5187-6ed8-4b55-ab1f-fb1b5aa010c6", "error_code": "Internal", "message": "clickward XML generation failure: No such file or directory (os error 2)" }

karencfv · 2024-09-04T23:44:56Z

clickhouse-admin/types/src/lib.rs

-    pub id: KeeperId,
-    pub raft_servers: Vec<RaftServerConfig>,
+    /// Unique ID of the keeper
+    pub node_id: KeeperId,


How about just id? It feels a bit redundant otherwise?

andrewjstone

Looks great! Thanks for the fixes @karencfv.

karencfv added 12 commits August 29, 2024 20:31

skeleton

5ca633c

fmt

5888f23

stronger types

14fe73e

moar stronger types

eb25fd4

return information

2c31a80

remove test endpoint

04c1a94

Simplify types

c128ef9

generate keeper config

27ba91f

take generation number

0412185

Simplify server types

112bdc3

Simplify keeper api

3f48362

Add some documentation

2f7f8bd

karencfv changed the title ~~[reconfigurator] WIP clickhouse-admin endpoint to generate ClickHouse config files~~ [reconfigurator] clickhouse-admin endpoints to generate ClickHouse config files Sep 2, 2024

karencfv marked this pull request as ready for review September 2, 2024 07:36

karencfv requested a review from andrewjstone September 2, 2024 07:36

andrewjstone reviewed Sep 4, 2024

View reviewed changes

karencfv added 2 commits September 5, 2024 11:29

Address comments

aa1375f

clean up

48f8cf9

karencfv commented Sep 4, 2024

View reviewed changes

andrewjstone approved these changes Sep 5, 2024

View reviewed changes

karencfv enabled auto-merge (squash) September 5, 2024 00:35

karencfv merged commit 61cac94 into oxidecomputer:main Sep 5, 2024
18 checks passed

karencfv deleted the ch-gen-api branch September 5, 2024 03:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[reconfigurator] clickhouse-admin endpoints to generate ClickHouse config files #6476

[reconfigurator] clickhouse-admin endpoints to generate ClickHouse config files #6476

karencfv commented Aug 29, 2024 •

edited

Loading

karencfv commented Sep 4, 2024

andrewjstone left a comment

andrewjstone Sep 4, 2024

andrewjstone Sep 4, 2024

karencfv Sep 4, 2024

andrewjstone Sep 5, 2024

karencfv Sep 5, 2024

andrewjstone Sep 4, 2024

karencfv Sep 4, 2024

andrewjstone Sep 5, 2024

karencfv left a comment

karencfv Sep 4, 2024

karencfv Sep 4, 2024

andrewjstone left a comment

[reconfigurator] clickhouse-admin endpoints to generate ClickHouse config files #6476

[reconfigurator] clickhouse-admin endpoints to generate ClickHouse config files #6476

Conversation

karencfv commented Aug 29, 2024 • edited Loading

Overview

Purpose

API structure

Usage

Caveats

Testing

karencfv commented Sep 4, 2024

andrewjstone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karencfv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewjstone left a comment

Choose a reason for hiding this comment

karencfv commented Aug 29, 2024 •

edited

Loading