Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pluto service takes 5 minutes to commit settings #267

Open
Tsonov opened this issue Nov 15, 2024 · 5 comments
Open

Pluto service takes 5 minutes to commit settings #267

Tsonov opened this issue Nov 15, 2024 · 5 comments

Comments

@Tsonov
Copy link

Tsonov commented Nov 15, 2024

Making this bug report because we observe slow init time for bottlerocket EKS nodes. Nodes get stuck in ~5 minutes before starting kubelet and joining the cluster. So far we pinpointed the source to slow pluto.service commit stage (which seems to come from this repo, correct?).

The clusters are using the latest EKS-optimized Bottlerocket image. It reproduces consistently on every new node but not for every cluster.

The question is how to investigate and fix the cause of this? We are not sure if this is a package issue or configuration issue in the clusters. The clusters have IDMS enabled. Not sure what else is required for this process.

Package I'm using:
pluto.service

What I expected to happen:
Startup to take 1-2 minutes and not 5+ minutes.

What actually happened:
Looking at systemd logs, pluto.service took 5 minutes to complete. We extracted logs from it and we observe the Committing settings step taking 5 minutes.

Logs from pluto:

bash-5.0# journalctl -u pluto.service
Nov 14 07:42:00 localhost systemd[1]: Starting Generate additional settings for Kubernetes...
Nov 14 07:42:00 localhost settings-committer[1832]: 07:42:00 [INFO] Checking pending settings.
Nov 14 07:42:00 localhost settings-committer[1832]: 07:42:00 [INFO] Committing settings.
Nov 14 07:47:01 localhost systemd[1]: Finished Generate additional settings for Kubernetes.

How to reproduce the problem:
Unclear, we only see this issue in some customer clusters but not on a fresh cluster.

** Extra information **
bash-5.0# apiclient get os
{
"os": {
"arch": "x86_64",
"build_id": "360b7a38",
"pretty_name": "Bottlerocket OS 1.26.2 (aws-k8s-1.30)",
"variant_id": "aws-k8s-1.30",
"version_id": "1.26.2"
}
}
@bcressey
Copy link
Contributor

I expect what's happening is that pluto is timing out trying to make one of its API calls, either ec2::get_private_dns_name or eks::get_cluster_network_config. This can happen for private VPCs with no Internet access, and no PrivateLink endpoint added for those services.

You can provision the node with some of these settings already defined in order to avoid the dependency:

[settings.kubernetes]
cluster-dns-ip = "some-ipv4-or-ipv6-address"
hostname-override-source = "instance-id"

Note that using the instance ID as the hostname means that you will want to enable resource-based naming so that the hostname that kubelet registers with is actually present in the VPC's DNS zone.

Alternatively, if the VPC has a proxy server set up, you can set the https proxy setting and pluto will use it when making outbound requests.

@Tsonov
Copy link
Author

Tsonov commented Nov 18, 2024

Thank you for the tips, investigating configuration with customer to see if those things match.

Is there a way to enable more verbose logging for pluto, perhaps something that shows failed retries, etc?

@koooosh
Copy link
Contributor

koooosh commented Nov 26, 2024

pluto does not currently have more verbose logging. We actually have a long-standing issue to improve this and our general logging verbosity: bottlerocket-os/bottlerocket#2200

@Tsonov
Copy link
Author

Tsonov commented Dec 4, 2024

Update on this from our side, I built an AMI with increased logging level for settings-committer and sundog. Unfortunately it didn't help much so still looking for pointers.

Sample log from pluto.log with settings-applier set to TRACE level - notice the 5 minute gap between the last log and when it finishes. Can this be because there are no logs from pluto but only from settings-committer ?

Dec 04 11:12:25 localhost systemd[1]: Starting Generate additional settings for Kubernetes...
Subject: A start job for unit pluto.service has begun execution
Defined-By: systemd
Support: https://github.com/bottlerocket-os/bottlerocket/discussions

A start job for unit pluto.service has begun execution.

The job identifier is 101.
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [INFO] Checking pending settings.
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (1) settings_committer: GET-ing /tx?tx=bottlerocket-launch to determine if there are pending settings
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (1) hyper::client::pool: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/client/pool.rs:639] checkout waiting for idle connection: ("unix", 2f72756e2f6170692e736f636b:0)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (1) hyper::client::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/client/conn.rs:1007] client handshake Http1
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (1) hyper::client::client: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/client/client.rs:509] handshake complete, spawning background dispatcher task
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (9) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:731] flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Busy }
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (1) hyper::client::pool: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/client/pool.rs:681] checkout dropped for ("unix", 2f72756e2f6170692e736f636b:0)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (9) tracing::span: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:110] encode_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (9) tracing::span::active: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:110] -> encode_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (9) hyper::proto::h1::role: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:1113] Client::encode method=GET, body=None
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (9) tracing::span::active: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:110] <- encode_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (9) tracing::span: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:110] -- encode_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (9) hyper::proto::h1::io: flushed 111 bytes
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (9) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:731] flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:193] Conn::read_head
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::io: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/io.rs:267] received 802 bytes
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) tracing::span: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:97] parse_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) tracing::span::active: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:97] -> parse_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::role: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:949] Response.parse bytes=802
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::role: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:958] Response.parse Complete(109)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) tracing::span::active: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:97] <- parse_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) tracing::span: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:97] -- parse_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (2) hyper::proto::h1::io: parsed 3 headers
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (2) hyper::proto::h1::conn: incoming body is content-length (693 bytes)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::decode: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/decode.rs:127] decode; state=Length(693)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (2) hyper::proto::h1::conn: incoming body completed
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:471] maybe_notify; read_from_io blocked
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:731] flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:731] flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) hyper::client::pool: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/client/pool.rs:333] put; add idle connection for ("unix", 2f72756e2f6170692e736f636b:0)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (8) hyper::client::pool: pooling idle connection for ("unix", 2f72756e2f6170692e736f636b:0)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (1) settings_committer: Pending settings for tx bottlerocket-launch: {"boot": Object {"kernel": Object {"initcall_blacklist": Array [String("vmd_drv_init"), String("megasas_init"), String("mpt3sas_init"), String("pqi_init")], "module_blacklist": Array [String("i8042")]}}, "host-containers": Object {"admin": Object {"user-data": String("eyJzc2giOnsiYXV0aG9yaXplZC1rZXlzIjpbXX19")}, "control": Object {"source": String("328549459982.dkr.ecr.us-east-2.amazonaws.com/bottlerocket-control:v0.7.18")}}, "aws": Object {"config": String("W2RlZmF1bHRdCnVzZV9maXBzX2VuZHBvaW50PWZhbHNl")}, "metrics": Object {"send-metrics": Bool(true)}, "updates": Object {"metadata-base-url": String("https://updates.bottlerocket.aws/2020-07-07/aws-k8s-1.30/x86_64/"), "seed": Number(2013), "targets-base-url": String("https://updates.bottlerocket.aws/targets/")}, "kernel": Object {"sysctl": Object {"vm/nr_hugepages": String("0")}}, "network": Object {"hostname": String("ip-10-0-120-107.us-east-2.compute.internal")}}
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [INFO] Committing settings.
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (1) settings_committer: POST-ing to /tx/commit?tx=bottlerocket-launch to move pending settings to live
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) hyper::proto::h1::dispatch: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/dispatch.rs:601] client tx closed
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:955] State::close_read()
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (1) hyper::client::pool: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/client/pool.rs:639] checkout waiting for idle connection: ("unix", 2f72756e2f6170692e736f636b:0)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:961] State::close_write()
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:731] flushed({role=client}): State { reading: Closed, writing: Closed, keep_alive: Disabled }
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:738] shut down IO complete
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (1) hyper::client::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/client/conn.rs:1007] client handshake Http1
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (1) hyper::client::client: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/client/client.rs:509] handshake complete, spawning background dispatcher task
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:731] flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Busy }
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (1) hyper::client::pool: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/client/pool.rs:681] checkout dropped for ("unix", 2f72756e2f6170692e736f636b:0)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) tracing::span: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:110] encode_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) tracing::span::active: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:110] -> encode_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) hyper::proto::h1::role: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:1113] Client::encode method=POST, body=None
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) tracing::span::active: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:110] <- encode_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) tracing::span: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:110] -- encode_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (8) hyper::proto::h1::io: flushed 119 bytes
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (8) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:731] flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:193] Conn::read_head
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::io: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/io.rs:267] received 497 bytes
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) tracing::span: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:97] parse_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) tracing::span::active: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:97] -> parse_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::role: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:949] Response.parse bytes=497
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::role: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:958] Response.parse Complete(109)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) tracing::span::active: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:97] <- parse_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) tracing::span: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/role.rs:97] -- parse_headers;
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (2) hyper::proto::h1::io: parsed 3 headers
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (2) hyper::proto::h1::conn: incoming body is content-length (388 bytes)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::decode: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/decode.rs:127] decode; state=Length(388)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (2) hyper::proto::h1::conn: incoming body completed
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:471] maybe_notify; read_from_io blocked
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:731] flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (2) hyper::proto::h1::conn: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/proto/h1/conn.rs:731] flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [TRACE] (7) hyper::client::pool: [/home/builder/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.30/src/client/pool.rs:333] put; add idle connection for ("unix", 2f72756e2f6170692e736f636b:0)
Dec 04 11:12:25 localhost settings-committer[1838]: 11:12:25 [DEBUG] (7) hyper::client::pool: pooling idle connection for ("unix", 2f72756e2f6170692e736f636b:0)
Dec 04 11:17:26 localhost systemd[1]: Finished Generate additional settings for Kubernetes.
Subject: A start job for unit pluto.service has finished successfully
Defined-By: systemd
Support: https://github.com/bottlerocket-os/bottlerocket/discussions

A start job for unit pluto.service has finished successfully.

The job identifier is 101.

settings-applier service starts afterwards at 11:17:26.

Is the only step forward to add manual logging statements in pluto to see what is happening? I see it doesn't seem to have any logs at the moment (but I haven't used rust so could be wrong)?

I have output of logdog + pluto + sundog + settings-applier available (and can get others if needed) - what could be useful?

EKS setup:
The nodes are deployed in VPC with outbound internet access.
Also, if we create a managed node group with same bottlerocket image and same instance profile, nodes start fine.
What else could be interesting in the setup to share?

@Tsonov
Copy link
Author

Tsonov commented Dec 4, 2024

Does it make sense to increase log level on other components (apiclient or apiserver), would that help?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@bcressey @Tsonov @koooosh