Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ClickHouseKeeper Pods in CrashLoopBackOff State Prevent New Pods from Deploying Even After Applying a New ClickHouseKeeperInstallation #1519

Open
arun-kbharathan-e3768 opened this issue Oct 1, 2024 · 8 comments
Open

ClickHouseKeeper Pods in CrashLoopBackOff State Prevent New Pods from Deploying Even After Applying a New ClickHouseKeeperInstallation #1519

arun-kbharathan-e3768 opened this issue Oct 1, 2024 · 8 comments

Comments

@arun-kbharathan-e3768
Copy link

Hi,

While working with ClickHouseKeeperInstallation (0.23.7), I observed that when the pods are in a CrashLoopBackOff state, new pods are not being deployed with the updated resources specified in the new ClickHouseKeeperInstallation.

An example scenario to reproduce: Here, I am intentionally providing a pod template with very low memory and CPU resources, causing the pods to enter a CrashLoopBackOff state.

apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"
metadata:
  name: chk-3-nodes
spec:
  configuration:
    clusters:
      - name: "simple-3"
        layout:
          replicasCount: 3
    settings:
      logger/level: "trace"
      logger/console: "true"
      listen_host: "0.0.0.0"
      keeper_server/storage_path: /var/lib/clickhouse-keeper
      keeper_server/tcp_port: "2181"
      keeper_server/four_letter_word_white_list: "*"
      keeper_server/coordination_settings/raft_logs_level: "information"
      keeper_server/raft_configuration/server/port: "9444"
      prometheus/endpoint: "/metrics"
      prometheus/port: "7000"
      prometheus/metrics: "true"
      prometheus/events: "true"
      prometheus/asynchronous_metrics: "true"
      prometheus/status_info: "false"
  templates:
    podTemplates:
      - name: default
        spec:
          containers:
            - name: clickhouse-keeper
              imagePullPolicy: IfNotPresent
              image: "clickhouse/clickhouse-keeper:head-alpine"
              resources:
                requests:
                  memory: "5Mi"
                  cpu: "50m"
                limits:
                  memory: 10Mi
                  cpu: 100m

Within a few minutes, all the pods enter a CrashLoopBackOff state.
CHK_Crashloopback

Now, apply a ClickHouseKeeperInstallation with proper resources specified in the pod template.

apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"
metadata:
  name: chk-3-nodes
spec:
  configuration:
    clusters:
      - name: "simple-3"
        layout:
          replicasCount: 3
    settings:
      logger/level: "trace"
      logger/console: "true"
      listen_host: "0.0.0.0"
      keeper_server/storage_path: /var/lib/clickhouse-keeper
      keeper_server/tcp_port: "2181"
      keeper_server/four_letter_word_white_list: "*"
      keeper_server/coordination_settings/raft_logs_level: "information"
      keeper_server/raft_configuration/server/port: "9444"
      prometheus/endpoint: "/metrics"
      prometheus/port: "7000"
      prometheus/metrics: "true"
      prometheus/events: "true"
      prometheus/asynchronous_metrics: "true"
      prometheus/status_info: "false"
  templates:
    podTemplates:
      - name: default
        spec:
          containers:
            - name: clickhouse-keeper
              imagePullPolicy: IfNotPresent
              image: "clickhouse/clickhouse-keeper:head-alpine"
              resources:
                requests:
                  memory: "256M"
                  cpu: "1"
                limits:
                  memory: "4Gi"
                  cpu: "2"

Even after applying the new ClickHouseKeeperInstallation, the pods are not coming up with the updated resources specified in the new ClickHouseKeeperInstallation.

The expected behavior is that, even if the old pods are in a CrashLoopBackOff state, the new pods should be deployed with the updated resources mentioned in the new ClickHouseKeeperInstallation.
@Slach
Copy link
Collaborator

Slach commented Oct 1, 2024

  1. use clickhouse/clickhouse-keeper:latest-alpine image
  2. share kubectl logs chk-3-nodes-0 --since=24h?
  3. unfortunately, kind: ClickHouseKeeperInstallationunstable in 0.23.x and will deeply refactor in 0.24.0

@arun-kbharathan-e3768
Copy link
Author

  1. Used clickhouse/clickhouse-keeper:latest-alpine. This time, it was difficult to reproduce the issue where all 3 pods go into a CrashLoopBackOff state. In most cases, the pod chk-3-nodes-0 consistently remained in a running state.

  2. Initially, a new configuration with proper resources was applied to ClickHouse Keeper.

    • At 2024.10.01 19:47:41.295087, a configuration with very limited resources was applied to ClickHouse Keeper, causing pods chk-3-nodes-1 and chk-3-nodes-2 to enter a CrashLoopBackOff state.
    • At 2024.10.01 19:48:45.440479, a configuration with proper resources was applied to ClickHouse Keeper. However, the new configuration was not reflected, and the pods continued in the same state, with chk-3-nodes-1 and chk-3-nodes-2 in CrashLoopBackOff and chk-3-nodes-0 running.
stern chk-3-nodes-0 --since=24h
+ chk-3-nodes-0 › clickhouse-keeper
+ chk-3-nodes-0 › server-id-injector
chk-3-nodes-0 clickhouse-keeper Processing configuration file '/etc/clickhouse-keeper/keeper_config.xml'.
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.129836 [ 1 ] {} <Information> SentryWriter: Sending crash reports is disabled
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.187120 [ 1 ] {} <Information> Application: Starting ClickHouse Keeper 24.9.1.3278 (revision: 54490, git hash: 6d058d82a8ec7a7dda413d61ab5c6a2ddf6efb25, build id: BDAD68859CAB4FAE4765F83D3F86267423743DCB), PID 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.187201 [ 1 ] {} <Information> Application: starting up
chk-3-nodes-0 server-id-injector + export KEEPER_ID=0
chk-3-nodes-0 server-id-injector + KEEPER_ID=0
chk-3-nodes-0 server-id-injector + sed s/KEEPER_ID/0/g /tmp/clickhouse-keeper/keeper_config.xml
chk-3-nodes-0 server-id-injector + cat /etc/clickhouse-keeper/keeper_config.xml
chk-3-nodes-0 server-id-injector <clickhouse>
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.187221 [ 1 ] {} <Information> Application: OS Name = Linux, OS Version = 6.8.0-45-generic, OS Architecture = x86_64
chk-3-nodes-0 server-id-injector     <keeper_server>
chk-3-nodes-0 server-id-injector         <coordination_settings>
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.187422 [ 1 ] {} <Information> Jemalloc: Value for background_thread set to true (from true)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.189229 [ 1 ] {} <Information> Application: keeper_server.max_memory_usage_soft_limit is set to 270.00 MiB
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.189290 [ 1 ] {} <Warning> Application: Effective user of the process (clickhouse) does not match the owner of the data (root).
chk-3-nodes-0 server-id-injector             <min_session_timeout_ms>10000</min_session_timeout_ms>
chk-3-nodes-0 server-id-injector             <operation_timeout_ms>10000</operation_timeout_ms>
chk-3-nodes-0 server-id-injector             <raft_logs_level>information</raft_logs_level>
chk-3-nodes-0 server-id-injector             <session_timeout_ms>100000</session_timeout_ms>
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.191279 [ 1 ] {} <Information> CgroupsReader: Will create cgroup reader from '/sys/fs/cgroup/' (cgroups version: v2)
chk-3-nodes-0 server-id-injector         </coordination_settings>
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.191312 [ 1 ] {} <Information> MemoryWorker: Starting background memory thread with period of 50ms, using Cgroups as source
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.191404 [ 1 ] {} <Debug> Application: Initializing DateLUT.
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.191409 [ 1 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'.
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.192843 [ 1 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/thermal
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.213831 [ 1 ] {} <Trace> AsynchronousMetrics: Scanning /sys/block
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.214017 [ 1 ] {} <Trace> AsynchronousMetrics: Scanning /sys/devices/system/edac
chk-3-nodes-0 server-id-injector         <four_letter_word_white_list>*</four_letter_word_white_list>
chk-3-nodes-0 server-id-injector         <hostname_checks_enabled>true</hostname_checks_enabled>
chk-3-nodes-0 server-id-injector         <log_storage_path>/var/lib/clickhouse-keeper/coordination/logs</log_storage_path>
chk-3-nodes-0 server-id-injector         <raft_configuration>
chk-3-nodes-0 server-id-injector             <server>
chk-3-nodes-0 server-id-injector                 <id>0</id>
chk-3-nodes-0 server-id-injector                 <hostname>chk-3-nodes-0.chk-3-nodes-headless.test.svc.cluster.local</hostname>
chk-3-nodes-0 server-id-injector                 <port>9444</port>
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.214122 [ 1 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/hwmon
chk-3-nodes-0 server-id-injector             </server>
chk-3-nodes-0 server-id-injector             <server>
chk-3-nodes-0 server-id-injector                 <id>1</id>
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.245557 [ 35 ] {} <Information> MemoryTracker: Correcting the value of global memory tracker from 24.19 KiB to 25.32 MiB
chk-3-nodes-0 server-id-injector                 <hostname>chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local</hostname>
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.256725 [ 1 ] {} <Information> Context: Cannot connect to ZooKeeper (or Keeper) before internal Keeper start, will wait for Keeper synchronously
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.256773 [ 1 ] {} <Debug> KeeperDispatcher: Initializing storage dispatcher
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.256875 [ 1 ] {} <Information> KeeperContext: Keeper feature flag FILTERED_LIST: enabled
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.256881 [ 1 ] {} <Information> KeeperContext: Keeper feature flag MULTI_READ: enabled
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.256884 [ 1 ] {} <Information> KeeperContext: Keeper feature flag CHECK_NOT_EXISTS: disabled
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.256886 [ 1 ] {} <Information> KeeperContext: Keeper feature flag CREATE_IF_NOT_EXISTS: disabled
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.256890 [ 1 ] {} <Information> KeeperContext: Keeper feature flag REMOVE_RECURSIVE: disabled
chk-3-nodes-0 server-id-injector                 <port>9444</port>
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.257402 [ 1 ] {} <Trace> KeeperSnapshotManager: Reading from disk LocalSnapshotDisk
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.257864 [ 1 ] {} <Trace> KeeperSnapshotManager: No snapshots were found on LocalSnapshotDisk
chk-3-nodes-0 server-id-injector             </server>
chk-3-nodes-0 server-id-injector             <server>
chk-3-nodes-0 server-id-injector                 <id>2</id>
chk-3-nodes-0 server-id-injector                 <hostname>chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local</hostname>
chk-3-nodes-0 server-id-injector                 <port>9444</port>
chk-3-nodes-0 server-id-injector             </server>
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.261277 [ 1 ] {} <Warning> DNSResolver: Cannot resolve host (chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local), error 0: Host not found.
chk-3-nodes-0 server-id-injector         </raft_configuration>
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.261882 [ 1 ] {} <Error> bool DB::(anonymous namespace)::isLocalhost(const std::string &): Code: 198. DB::NetException: Not found address of host: chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local. (DNS_ERROR), Stack trace (when copying this message, always include the lines below):
chk-3-nodes-0 server-id-injector         <server_id>0</server_id>
chk-3-nodes-0 clickhouse-keeper 
chk-3-nodes-0 clickhouse-keeper 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ae049db
chk-3-nodes-0 server-id-injector         <snapshot_storage_path>/var/lib/clickhouse-keeper/coordination/snapshots</snapshot_storage_path>
chk-3-nodes-0 clickhouse-keeper 1. DB::NetException::NetException<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000ade0e03
chk-3-nodes-0 server-id-injector         <storage_path>/var/lib/clickhouse-keeper</storage_path>
chk-3-nodes-0 server-id-injector         <tcp_port>2181</tcp_port>
chk-3-nodes-0 server-id-injector     </keeper_server>
chk-3-nodes-0 server-id-injector     <listen_host>0.0.0.0</listen_host>
chk-3-nodes-0 server-id-injector     <logger>
chk-3-nodes-0 server-id-injector         <console>true</console>
chk-3-nodes-0 server-id-injector         <level>trace</level>
chk-3-nodes-0 server-id-injector     </logger>
chk-3-nodes-0 clickhouse-keeper 2. DB::(anonymous namespace)::hostByName(String const&) @ 0x000000000ade2a4e
chk-3-nodes-0 server-id-injector     <max_connections>4096</max_connections>
chk-3-nodes-0 clickhouse-keeper 3. DB::DNSResolver::getResolvedIPAdressessWithFiltering(String const&) @ 0x000000000ade0847
chk-3-nodes-0 clickhouse-keeper 4. DB::KeeperStateManager::parseServersConfiguration(Poco::Util::AbstractConfiguration const&, bool, bool) const @ 0x000000000d54365e
chk-3-nodes-0 clickhouse-keeper 5. DB::KeeperStateManager::KeeperStateManager(int, String const&, String const&, Poco::Util::AbstractConfiguration const&, std::shared_ptr<DB::KeeperContext>) @ 0x000000000d546699
chk-3-nodes-0 clickhouse-keeper 6. DB::KeeperServer::KeeperServer(std::shared_ptr<DB::KeeperConfigurationAndSettings> const&, Poco::Util::AbstractConfiguration const&, ConcurrentBoundedQueue<DB::KeeperStorageBase::ResponseForSession>&, ConcurrentBoundedQueue<DB::CreateSnapshotTask>&, std::shared_ptr<DB::KeeperContext>, DB::KeeperSnapshotManagerS3&, std::function<void (unsigned long, DB::KeeperStorageBase::RequestForSession const&)>) @ 0x000000000d4bdcc6
chk-3-nodes-0 server-id-injector     <openSSL>
chk-3-nodes-0 server-id-injector         <server>
chk-3-nodes-0 server-id-injector             <cacheSessions>true</cacheSessions>
chk-3-nodes-0 server-id-injector             <certificateFile>/etc/clickhouse-keeper/server.crt</certificateFile>
chk-3-nodes-0 server-id-injector             <dhParamsFile>/etc/clickhouse-keeper/dhparam.pem</dhParamsFile>
chk-3-nodes-0 server-id-injector             <disableProtocols>sslv2,sslv3</disableProtocols>
chk-3-nodes-0 server-id-injector             <loadDefaultCAFile>true</loadDefaultCAFile>
chk-3-nodes-0 clickhouse-keeper 7. DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::shared_ptr<DB::Macros const> const&) @ 0x000000000d4a68d2
chk-3-nodes-0 server-id-injector             <preferServerCiphers>true</preferServerCiphers>
chk-3-nodes-0 server-id-injector             <privateKeyFile>/etc/clickhouse-keeper/server.key</privateKeyFile>
chk-3-nodes-0 server-id-injector             <verificationMode>none</verificationMode>
chk-3-nodes-0 server-id-injector         </server>
chk-3-nodes-0 clickhouse-keeper 8. DB::Context::initializeKeeperDispatcher(bool) const @ 0x000000000bd13af8
chk-3-nodes-0 clickhouse-keeper 9. DB::Keeper::main(std::vector<String, std::allocator<String>> const&) @ 0x00000000055bb2bf
chk-3-nodes-0 clickhouse-keeper 10. Poco::Util::Application::run() @ 0x0000000010fef586
chk-3-nodes-0 clickhouse-keeper 11. DB::Keeper::run() @ 0x00000000055b8570
chk-3-nodes-0 clickhouse-keeper 12. Poco::Util::ServerApplication::run(int, char**) @ 0x0000000010ff7887
chk-3-nodes-0 clickhouse-keeper 13. mainEntryClickHouseKeeper(int, char**) @ 0x00000000055b71e2
chk-3-nodes-0 clickhouse-keeper 14. main @ 0x00000000055b6150
chk-3-nodes-0 server-id-injector     </openSSL>
chk-3-nodes-0 server-id-injector     <prometheus>
chk-3-nodes-0 server-id-injector         <asynchronous_metrics>true</asynchronous_metrics>
chk-3-nodes-0 server-id-injector         <endpoint>/metrics</endpoint>
chk-3-nodes-0 clickhouse-keeper 15. ? @ 0x0000717261b4b083
chk-3-nodes-0 clickhouse-keeper 16. _start @ 0x0000000004e4d02e
chk-3-nodes-0 clickhouse-keeper  (version 24.9.1.3278 (official build))
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.264991 [ 1 ] {} <Warning> DNSResolver: Cannot resolve host (chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local), error 0: Host not found.
chk-3-nodes-0 server-id-injector         <events>true</events>
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.265150 [ 1 ] {} <Error> bool DB::(anonymous namespace)::isLocalhost(const std::string &): Code: 198. DB::NetException: Not found address of host: chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local. (DNS_ERROR), Stack trace (when copying this message, always include the lines below):
chk-3-nodes-0 server-id-injector         <metrics>true</metrics>
chk-3-nodes-0 server-id-injector         <port>7000</port>
chk-3-nodes-0 clickhouse-keeper 
chk-3-nodes-0 server-id-injector         <status_info>false</status_info>
chk-3-nodes-0 server-id-injector     </prometheus>
chk-3-nodes-0 server-id-injector </clickhouse>
chk-3-nodes-0 clickhouse-keeper 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ae049db
chk-3-nodes-0 clickhouse-keeper 1. DB::NetException::NetException<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000ade0e03
chk-3-nodes-0 clickhouse-keeper 2. DB::(anonymous namespace)::hostByName(String const&) @ 0x000000000ade2a4e
chk-3-nodes-0 clickhouse-keeper 3. DB::DNSResolver::getResolvedIPAdressessWithFiltering(String const&) @ 0x000000000ade0847
chk-3-nodes-0 clickhouse-keeper 4. DB::KeeperStateManager::parseServersConfiguration(Poco::Util::AbstractConfiguration const&, bool, bool) const @ 0x000000000d54365e
chk-3-nodes-0 clickhouse-keeper 5. DB::KeeperStateManager::KeeperStateManager(int, String const&, String const&, Poco::Util::AbstractConfiguration const&, std::shared_ptr<DB::KeeperContext>) @ 0x000000000d546699
chk-3-nodes-0 clickhouse-keeper 6. DB::KeeperServer::KeeperServer(std::shared_ptr<DB::KeeperConfigurationAndSettings> const&, Poco::Util::AbstractConfiguration const&, ConcurrentBoundedQueue<DB::KeeperStorageBase::ResponseForSession>&, ConcurrentBoundedQueue<DB::CreateSnapshotTask>&, std::shared_ptr<DB::KeeperContext>, DB::KeeperSnapshotManagerS3&, std::function<void (unsigned long, DB::KeeperStorageBase::RequestForSession const&)>) @ 0x000000000d4bdcc6
chk-3-nodes-0 clickhouse-keeper 7. DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::shared_ptr<DB::Macros const> const&) @ 0x000000000d4a68d2
chk-3-nodes-0 clickhouse-keeper 8. DB::Context::initializeKeeperDispatcher(bool) const @ 0x000000000bd13af8
chk-3-nodes-0 clickhouse-keeper 9. DB::Keeper::main(std::vector<String, std::allocator<String>> const&) @ 0x00000000055bb2bf
chk-3-nodes-0 clickhouse-keeper 10. Poco::Util::Application::run() @ 0x0000000010fef586
chk-3-nodes-0 clickhouse-keeper 11. DB::Keeper::run() @ 0x00000000055b8570
chk-3-nodes-0 clickhouse-keeper 12. Poco::Util::ServerApplication::run(int, char**) @ 0x0000000010ff7887
chk-3-nodes-0 clickhouse-keeper 13. mainEntryClickHouseKeeper(int, char**) @ 0x00000000055b71e2
chk-3-nodes-0 clickhouse-keeper 14. main @ 0x00000000055b6150
chk-3-nodes-0 clickhouse-keeper 15. ? @ 0x0000717261b4b083
chk-3-nodes-0 clickhouse-keeper 16. _start @ 0x0000000004e4d02e
chk-3-nodes-0 clickhouse-keeper  (version 24.9.1.3278 (official build))
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.265285 [ 1 ] {} <Trace> KeeperLogStore: Reading from disk LocalLogDisk
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.265359 [ 1 ] {} <Warning> KeeperLogStore: No logs exists in /var/lib/clickhouse-keeper/coordination/logs. It's Ok if it's the first run of clickhouse-keeper.
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.265600 [ 1 ] {} <Information> KeeperLogStore: force_sync enabled
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.265610 [ 1 ] {} <Debug> KeeperDispatcher: Waiting server to initialize
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.265624 [ 1 ] {} <Debug> KeeperStateMachine: Totally have 0 snapshots
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.265628 [ 1 ] {} <Debug> KeeperStateMachine: Trying to load state machine from snapshot up to log index 0
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.265632 [ 1 ] {} <Debug> KeeperStateMachine: No existing snapshots, last committed log index 0
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.265661 [ 1 ] {} <Warning> KeeperLogStore: Removing all changelogs
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.265673 [ 1 ] {} <Trace> Changelog: Starting new changelog changelog_1_100000.bin
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.265743 [ 1 ] {} <Trace> KeeperServer: Last local log idx 0
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.265758 [ 1 ] {} <Information> KeeperServer: No config in log store and snapshot, probably it's initial run. Will use config from .xml on disk
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.267202 [ 1 ] {} <Warning> DNSResolver: Cannot resolve host (chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local), error 0: Host not found.
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.267650 [ 1 ] {} <Error> bool DB::(anonymous namespace)::isLocalhost(const std::string &): Code: 198. DB::NetException: Not found address of host: chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local. (DNS_ERROR), Stack trace (when copying this message, always include the lines below):
chk-3-nodes-0 clickhouse-keeper 
chk-3-nodes-0 clickhouse-keeper 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ae049db
chk-3-nodes-0 clickhouse-keeper 1. DB::NetException::NetException<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000ade0e03
- chk-3-nodes-0 › server-id-injector
chk-3-nodes-0 clickhouse-keeper 2. DB::(anonymous namespace)::hostByName(String const&) @ 0x000000000ade2a4e
chk-3-nodes-0 clickhouse-keeper 3. DB::DNSResolver::getResolvedIPAdressessWithFiltering(String const&) @ 0x000000000ade0847
chk-3-nodes-0 clickhouse-keeper 4. DB::KeeperStateManager::parseServersConfiguration(Poco::Util::AbstractConfiguration const&, bool, bool) const @ 0x000000000d54365e
chk-3-nodes-0 clickhouse-keeper 5. DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) @ 0x000000000d4c5e4e
chk-3-nodes-0 clickhouse-keeper 6. DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::shared_ptr<DB::Macros const> const&) @ 0x000000000d4a6c40
chk-3-nodes-0 clickhouse-keeper 7. DB::Context::initializeKeeperDispatcher(bool) const @ 0x000000000bd13af8
chk-3-nodes-0 clickhouse-keeper 8. DB::Keeper::main(std::vector<String, std::allocator<String>> const&) @ 0x00000000055bb2bf
chk-3-nodes-0 clickhouse-keeper 9. Poco::Util::Application::run() @ 0x0000000010fef586
chk-3-nodes-0 clickhouse-keeper 10. DB::Keeper::run() @ 0x00000000055b8570
chk-3-nodes-0 clickhouse-keeper 11. Poco::Util::ServerApplication::run(int, char**) @ 0x0000000010ff7887
chk-3-nodes-0 clickhouse-keeper 12. mainEntryClickHouseKeeper(int, char**) @ 0x00000000055b71e2
chk-3-nodes-0 clickhouse-keeper 13. main @ 0x00000000055b6150
chk-3-nodes-0 clickhouse-keeper 14. ? @ 0x0000717261b4b083
chk-3-nodes-0 clickhouse-keeper 15. _start @ 0x0000000004e4d02e
chk-3-nodes-0 clickhouse-keeper  (version 24.9.1.3278 (official build))
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.269052 [ 1 ] {} <Warning> DNSResolver: Cannot resolve host (chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local), error 0: Host not found.
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.269131 [ 1 ] {} <Error> bool DB::(anonymous namespace)::isLocalhost(const std::string &): Code: 198. DB::NetException: Not found address of host: chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local. (DNS_ERROR), Stack trace (when copying this message, always include the lines below):
chk-3-nodes-0 clickhouse-keeper 
chk-3-nodes-0 clickhouse-keeper 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000ae049db
chk-3-nodes-0 clickhouse-keeper 1. DB::NetException::NetException<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000ade0e03
chk-3-nodes-0 clickhouse-keeper 2. DB::(anonymous namespace)::hostByName(String const&) @ 0x000000000ade2a4e
chk-3-nodes-0 clickhouse-keeper 3. DB::DNSResolver::getResolvedIPAdressessWithFiltering(String const&) @ 0x000000000ade0847
chk-3-nodes-0 clickhouse-keeper 4. DB::KeeperStateManager::parseServersConfiguration(Poco::Util::AbstractConfiguration const&, bool, bool) const @ 0x000000000d54365e
chk-3-nodes-0 clickhouse-keeper 5. DB::KeeperServer::startup(Poco::Util::AbstractConfiguration const&, bool) @ 0x000000000d4c5e4e
chk-3-nodes-0 clickhouse-keeper 6. DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::shared_ptr<DB::Macros const> const&) @ 0x000000000d4a6c40
chk-3-nodes-0 clickhouse-keeper 7. DB::Context::initializeKeeperDispatcher(bool) const @ 0x000000000bd13af8
chk-3-nodes-0 clickhouse-keeper 8. DB::Keeper::main(std::vector<String, std::allocator<String>> const&) @ 0x00000000055bb2bf
chk-3-nodes-0 clickhouse-keeper 9. Poco::Util::Application::run() @ 0x0000000010fef586
chk-3-nodes-0 clickhouse-keeper 10. DB::Keeper::run() @ 0x00000000055b8570
chk-3-nodes-0 clickhouse-keeper 11. Poco::Util::ServerApplication::run(int, char**) @ 0x0000000010ff7887
chk-3-nodes-0 clickhouse-keeper 12. mainEntryClickHouseKeeper(int, char**) @ 0x00000000055b71e2
chk-3-nodes-0 clickhouse-keeper 13. main @ 0x00000000055b6150
chk-3-nodes-0 clickhouse-keeper 14. ? @ 0x0000717261b4b083
chk-3-nodes-0 clickhouse-keeper 15. _start @ 0x0000000004e4d02e
chk-3-nodes-0 clickhouse-keeper  (version 24.9.1.3278 (official build))
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.271193 [ 1 ] {} <Information> RaftInstance: Raft ASIO listener initiated on :::9444, unsecured
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.271259 [ 1 ] {} <Information> RaftInstance: parameters: election timeout range 1000 - 2000, heartbeat 500, leadership expiry 10000, max batch 100, backoff 50, snapshot distance 100000, enable randomized snapshot creation NO, log sync stop gap 99999, reserved logs 100000, client timeout 10000, auto forwarding on, API call type async, custom commit quorum size 0, custom election quorum size 0, snapshot receiver included, leadership transfer wait time 0, grace period of lagging state machine 0, snapshot IO: blocking, parallel log appending: on
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.271265 [ 1 ] {} <Information> RaftInstance: new election timeout range: 1000 - 2000
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.271281 [ 1 ] {} <Information> RaftInstance:    === INIT RAFT SERVER ===
chk-3-nodes-0 clickhouse-keeper commit index 0
chk-3-nodes-0 clickhouse-keeper term 0
chk-3-nodes-0 clickhouse-keeper election timer allowed
chk-3-nodes-0 clickhouse-keeper log store start 1, end 0
chk-3-nodes-0 clickhouse-keeper config log idx 0, prev log idx 0
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.271489 [ 1 ] {} <Information> RaftInstance: peer 0: DC ID 0, chk-3-nodes-0.chk-3-nodes-headless.test.svc.cluster.local:9444, voting member, 1
chk-3-nodes-0 clickhouse-keeper peer 1: DC ID 0, chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local:9444, voting member, 1
chk-3-nodes-0 clickhouse-keeper peer 2: DC ID 0, chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local:9444, voting member, 1
chk-3-nodes-0 clickhouse-keeper my id: 0, voting_member
chk-3-nodes-0 clickhouse-keeper num peers: 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.271500 [ 1 ] {} <Information> RaftInstance: global manager does not exist. will use local thread for commit and append
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.271680 [ 1 ] {} <Information> RaftInstance: wait for HB, for 50 + [1000, 2000] ms
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.271842 [ 63 ] {} <Information> RaftInstance: bg append_entries thread initiated
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:35.322021 [ 1 ] {} <Debug> KeeperDispatcher: Server initialized, waiting for quorum
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:36.463950 [ 51 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:36.464026 [ 51 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:36.464055 [ 51 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: follower, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:36.464083 [ 51 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:36.466862 [ 55 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:36.468111 [ 64 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:38.457476 [ 50 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:38.457582 [ 50 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:38.457605 [ 50 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:38.457632 [ 50 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:38.457769 [ 50 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:38.457848 [ 50 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:38.457861 [ 50 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:38.460639 [ 59 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:38.462075 [ 53 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:40.295106 [ 52 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:40.295175 [ 52 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:40.295196 [ 52 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:40.295221 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:40.295379 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:40.295456 [ 52 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:40.295474 [ 52 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:40.297957 [ 60 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:40.299595 [ 61 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:41.675807 [ 58 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:41.675878 [ 58 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:41.675900 [ 58 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:41.675924 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:41.676058 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:41.676131 [ 58 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 3
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:41.676147 [ 58 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:41.678704 [ 56 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:41.680353 [ 62 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:42.916381 [ 54 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:42.916471 [ 54 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:42.916498 [ 54 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:42.916530 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:42.916739 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:42.916820 [ 54 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 4
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:42.916834 [ 54 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:42.919371 [ 65 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:42.921135 [ 49 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:44.207026 [ 57 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:44.207088 [ 57 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:44.207106 [ 57 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:44.207133 [ 57 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:44.207258 [ 57 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:44.207320 [ 57 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 5
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:44.207339 [ 57 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:44.209658 [ 55 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:44.210992 [ 64 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:45.799468 [ 51 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:45.799588 [ 51 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:45.799609 [ 51 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:45.799625 [ 51 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:45.799833 [ 51 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:45.799913 [ 51 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 6
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:45.799933 [ 51 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:45.801936 [ 59 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:45.803468 [ 53 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:46.888106 [ 50 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:46.888145 [ 50 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:46.888158 [ 50 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:46.888170 [ 50 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:46.888321 [ 50 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:46.888404 [ 50 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 7
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:46.888419 [ 50 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:46.890364 [ 60 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:46.891950 [ 61 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)

chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:48.082653 [ 52 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:48.082700 [ 52 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:48.082715 [ 52 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:48.082727 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:48.082905 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:48.082981 [ 52 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 8
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:48.082992 [ 52 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:48.085781 [ 56 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:48.087163 [ 62 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:49.721190 [ 58 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:49.721227 [ 58 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:49.721240 [ 58 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:49.721251 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:49.721391 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:49.721459 [ 58 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 9
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:49.721469 [ 58 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:49.724142 [ 65 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:49.725324 [ 49 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:51.011619 [ 54 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:51.011663 [ 54 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:51.011678 [ 54 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:51.011691 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:51.011866 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:51.011939 [ 54 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 10
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:51.011951 [ 54 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:51.014749 [ 55 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:51.016181 [ 64 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.890046 [ 57 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.890081 [ 57 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.890094 [ 57 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 0, state term 0, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.890105 [ 57 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.890235 [ 57 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.890319 [ 57 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 0, live 0, dead 1), at least 2 nodes should respond. failure count 11
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.890330 [ 57 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 0, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.891693 [ 51 ] {} <Information> RaftInstance: 0x71723a8ab718 connected to chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local:9444 (as a client)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.891974 [ 53 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 2 (O), term 0, resp term 0, my role candidate, dead 2, live 0, abandoned 0, num voting members 3, quorum 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.891993 [ 53 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.891998 [ 53 ] {} <Information> RaftInstance: [PRE-VOTE DONE] initiate actual vote
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.892453 [ 60 ] {} <Information> RaftInstance: 0x71723a8aba98 connected to chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local:9444 (as a client)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.894018 [ 53 ] {} <Information> RaftInstance: [VOTE INIT] my id 0, my role candidate, term 1, log idx 0, log term 0, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.894279 [ 56 ] {} <Information> RaftInstance: [PRE-VOTE RESP] peer 1 (O), term 0, resp term 0, my role candidate, dead 3, live 0, abandoned 0, num voting members 3, quorum 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.894303 [ 56 ] {} <Information> RaftInstance: [PRE-VOTE DONE] SUCCESS, term 0
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.894310 [ 56 ] {} <Information> RaftInstance: [PRE-VOTE DONE] actual vote is already initiated, do nothing
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.894352 [ 65 ] {} <Information> RaftInstance: [VOTE RESP] peer 1 (X), resp term 1, my role candidate, granted 1, responded 2, num voting members 3, quorum 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.897864 [ 58 ] {} <Information> RaftInstance: [VOTE RESP] peer 2 (O), resp term 1, my role candidate, granted 2, responded 3, num voting members 3, quorum 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.897886 [ 58 ] {} <Information> RaftInstance: Server is elected as leader for term 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.897898 [ 58 ] {} <Information> RaftInstance: number of pending commit elements: 0
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.897938 [ 58 ] {} <Information> RaftInstance: state machine commit index 0, precommit index 0, last log index 0
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.899933 [ 58 ] {} <Information> RaftInstance: [BECOME LEADER] appended new config at 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.900190 [ 58 ] {} <Information> RaftInstance:   === LEADER (term 1) ===
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.900876 [ 66 ] {} <Information> RaftInstance: config at index 1 is committed, prev config log idx 0
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.900956 [ 66 ] {} <Information> RaftInstance: new config log idx 1, prev log idx 0, cur config log idx 0, prev log idx 0
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.901005 [ 66 ] {} <Information> RaftInstance: new configuration: log idx 1, prev log idx 0
chk-3-nodes-0 clickhouse-keeper peer 0, DC ID 0, chk-3-nodes-0.chk-3-nodes-headless.test.svc.cluster.local:9444, voting member, 1
chk-3-nodes-0 clickhouse-keeper peer 1, DC ID 0, chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local:9444, voting member, 1
chk-3-nodes-0 clickhouse-keeper peer 2, DC ID 0, chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local:9444, voting member, 1
chk-3-nodes-0 clickhouse-keeper my id: 0, leader: 0, term: 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.901163 [ 1 ] {} <Debug> KeeperDispatcher: Quorum initialized
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.901530 [ 1 ] {} <Debug> KeeperDispatcher: Dispatcher initialized
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.902433 [ 1 ] {} <Information> Application: Listening for Keeper (tcp): 0.0.0.0:2181
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:52.902666 [ 1 ] {} <Information> Application: Listening for Prometheus: http://0.0.0.0:7000
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:53.034453 [ 1 ] {} <Debug> ConfigReloader: Loading config '/etc/clickhouse-keeper/keeper_config.xml'
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:53.034615 [ 1 ] {} <Debug> ConfigProcessor: Processing configuration file '/etc/clickhouse-keeper/keeper_config.xml'.
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:53.035623 [ 1 ] {} <Debug> ConfigProcessor: Saved preprocessed configuration to '/var/lib/clickhouse-keeper/preprocessed_configs/keeper_config.xml'.
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:53.035682 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-keeper/keeper_config.xml', performing update on configuration
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:53.035904 [ 1 ] {} <Information> Application: keeper_server.max_memory_usage_soft_limit is set to 270.00 MiB
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:53.037486 [ 1 ] {} <Trace> KeeperDispatcher: Configuration update triggered, but nothing changed for Raft
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:53.049205 [ 1 ] {} <Error> CertificateReloader: Poco::Exception. Code: 1000, e.code() = 0, SSL context exception: Error loading private key from file /etc/clickhouse-keeper/server.key: error:80000002:system library::No such file or directory (version 24.9.1.3278 (official build))
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:53.049233 [ 1 ] {} <Debug> ConfigReloader: Loaded config '/etc/clickhouse-keeper/keeper_config.xml', performed update on configuration
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:53.049257 [ 1 ] {} <Trace> ConfigReloader: Config reload interval set to 2000ms
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:53.050734 [ 1 ] {} <Information> CgroupsMemoryUsageObserver: Started cgroup current memory usage observer thread
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:53.050764 [ 1 ] {} <Information> Application: Ready for connections.
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:46:53.051374 [ 74 ] {} <Information> CgroupsMemoryUsageObserver: Memory amount initially available to the process is 300.00 MiB
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:35.302167 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:59294
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:35.302451 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:38.293987 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:41612
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:38.294196 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:41.295087 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:41638
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:41.295264 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:44.303975 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:41654
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:44.304156 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:47.297397 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:48124
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:47.297565 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:47.431040 [ 54 ] {} <Warning> RaftInstance: peer (2) response error: failed to read response to peer 2, chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local:9444, 
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:47.934304 [ 60 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:48.483592 [ 59 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:49.084146 [ 56 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:49.734336 [ 60 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:50.299902 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:48148
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:50.300031 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:50.435338 [ 62 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:51.184942 [ 65 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:51.935519 [ 52 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:52.434038 [ 64 ] {} <Warning> RaftInstance: peer (1) response error: failed to read response to peer 1, chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local:9444, 
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:52.684863 [ 49 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:52.936948 [ 65 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:53.290347 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:48150
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:53.290519 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:53.435907 [ 60 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:53.486578 [ 58 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:54.086902 [ 51 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:54.185314 [ 52 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:54.737298 [ 62 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:54.936440 [ 56 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:55.437316 [ 59 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:55.685582 [ 54 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:56.187920 [ 57 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:56.293093 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:47480
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:56.293267 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:56.436847 [ 55 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:56.938170 [ 53 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:57.187097 [ 61 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:57.688389 [ 50 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:57.937215 [ 64 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:58.438536 [ 49 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:58.687355 [ 65 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:59.188877 [ 60 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:59.293969 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:47496
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:59.294189 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:59.437693 [ 58 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:47:59.938601 [ 51 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:00.188457 [ 62 ] {} <Warning> RaftInstance: peer (2) response error: failed to resolve host chk-3-nodes-2.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:00.690289 [ 52 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:00.939981 [ 56 ] {} <Warning> RaftInstance: too verbose RPC error on peer (2), will suppress it from now
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:01.439899 [ 59 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:02.187004 [ 65 ] {} <Error> RaftInstance: 2 nodes (out of 3, 3 including learners) are not responding longer than 10000 ms, at least 2 nodes (including leader) should be alive to proceed commit
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:02.187048 [ 65 ] {} <Error> RaftInstance: will yield the leadership of this node
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:02.187055 [ 65 ] {} <Information> RaftInstance: got immediate re-elect request, resign now
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:02.187062 [ 65 ] {} <Information> RaftInstance: [BECOME FOLLOWER] term 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:02.288988 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:47510
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:02.289178 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:03.867239 [ 60 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:03.867293 [ 60 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:03.867308 [ 60 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: follower, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:03.867322 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:03.867394 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:03.867425 [ 60 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:03.871126 [ 51 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:04.869656 [ 58 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:04.869706 [ 58 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:04.869720 [ 58 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:04.869734 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:04.869811 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:04.869939 [ 58 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:04.869953 [ 58 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:04.874070 [ 61 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:05.299207 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:47516
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:05.299395 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:06.422124 [ 53 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:06.422171 [ 53 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:06.422185 [ 53 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:06.422199 [ 53 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:06.422378 [ 53 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:06.422465 [ 53 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:06.422483 [ 53 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:06.425793 [ 56 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:07.983765 [ 52 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:07.983816 [ 52 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:07.983829 [ 52 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:07.983843 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:07.983911 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:07.984064 [ 52 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 3
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:07.984077 [ 52 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:07.987535 [ 49 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:08.299830 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:33938
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:08.300000 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:09.382318 [ 64 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:09.382362 [ 64 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:09.382376 [ 64 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:09.382394 [ 64 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:09.382461 [ 64 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:09.382488 [ 64 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 4
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:09.382500 [ 64 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:09.386154 [ 57 ] {} <Warning> RaftInstance: peer (1) response error: failed to resolve host chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local due to error 1, Host not found (authoritative)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:11.306451 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:33946
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:11.306693 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:11.356688 [ 54 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:11.356734 [ 54 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:11.356747 [ 54 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:11.356767 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:11.356838 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:11.356980 [ 54 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 5
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:11.356995 [ 54 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:11.360751 [ 51 ] {} <Warning> RaftInstance: too verbose RPC error on peer (1), will suppress it from now
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:13.096104 [ 60 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:13.096152 [ 60 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:13.096168 [ 60 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:13.096184 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:13.096269 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:13.096309 [ 60 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 6
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:13.096322 [ 60 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:14.296651 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:33962
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:14.296840 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:14.448439 [ 58 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:14.448484 [ 58 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:14.448497 [ 58 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:14.448509 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:14.448693 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:14.448763 [ 58 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 7
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:14.448774 [ 58 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:15.835968 [ 53 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:15.836028 [ 53 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:15.836039 [ 53 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:15.836052 [ 53 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:15.836205 [ 53 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:15.836287 [ 53 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 8
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:15.836298 [ 53 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:17.301212 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:51118
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:17.301404 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:17.434501 [ 52 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:17.434589 [ 52 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:17.434603 [ 52 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:17.434617 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:17.434803 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:17.434908 [ 52 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 9
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:17.434925 [ 52 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:18.875123 [ 64 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:18.875172 [ 64 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:18.875186 [ 64 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:18.875201 [ 64 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:18.875271 [ 64 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:18.875413 [ 64 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 10
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:18.875427 [ 64 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:19.958622 [ 54 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:19.958668 [ 54 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:19.958681 [ 54 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:19.958695 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:19.958872 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:19.958962 [ 54 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 11
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:19.958975 [ 54 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:20.300147 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:51126
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:20.300345 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:21.016135 [ 60 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:21.016183 [ 60 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:21.016198 [ 60 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:21.016214 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:21.016315 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:21.016798 [ 60 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 12
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:21.016840 [ 60 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:22.899968 [ 58 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:22.900016 [ 58 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:22.900030 [ 58 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:22.900123 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:22.900303 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:22.900394 [ 58 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 13
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:22.900406 [ 58 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:23.302310 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:51128
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:23.302484 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:24.814563 [ 53 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:24.814614 [ 53 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:24.814630 [ 53 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:24.814663 [ 53 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:24.815059 [ 53 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:24.815217 [ 53 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 14
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:24.815238 [ 53 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:26.306138 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:42976
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:26.306333 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:26.353466 [ 52 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:26.353515 [ 52 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:26.353530 [ 52 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:26.353585 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:26.353958 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:26.354070 [ 52 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 15
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:26.354082 [ 52 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:27.486296 [ 64 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:27.486343 [ 64 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:27.486357 [ 64 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:27.486370 [ 64 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:27.486619 [ 64 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:27.486736 [ 64 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 16
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:27.486776 [ 64 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:29.209905 [ 54 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:29.209954 [ 54 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:29.209968 [ 54 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:29.209983 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:29.210178 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:29.210274 [ 54 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 17
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:29.210289 [ 54 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:29.298895 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:42984
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:29.299105 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:30.876529 [ 60 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:30.876596 [ 60 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:30.876608 [ 60 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:30.876622 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:30.876769 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:30.876838 [ 60 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 18
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:30.876853 [ 60 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:32.298010 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:42994
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:32.298182 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:32.577977 [ 58 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:32.578026 [ 58 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:32.578039 [ 58 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:32.578053 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:32.578238 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:32.578316 [ 58 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 19
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:32.578330 [ 58 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:33.889435 [ 53 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:33.889475 [ 53 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:33.889489 [ 53 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:33.889504 [ 53 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:33.889674 [ 53 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:33.889760 [ 53 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 20
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:33.889776 [ 53 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:35.103931 [ 52 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:35.103980 [ 52 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:35.103993 [ 52 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:35.104005 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:35.104162 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:35.104245 [ 52 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 21
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:35.104258 [ 52 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:35.302163 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:42996
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:35.302347 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:36.363449 [ 64 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:36.363495 [ 64 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:36.363508 [ 64 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:36.363523 [ 64 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:36.363719 [ 64 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:36.363800 [ 64 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 22
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:36.363811 [ 64 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:38.176001 [ 54 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:38.176045 [ 54 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:38.176057 [ 54 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:38.176069 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:38.176217 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:38.176291 [ 54 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 23
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:38.176300 [ 54 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:38.294181 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:38270
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:38.294357 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:39.370506 [ 60 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:39.370582 [ 60 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:39.370602 [ 60 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:39.370617 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:39.370792 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:39.370865 [ 60 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 24
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:39.370877 [ 60 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:40.531057 [ 58 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:40.531104 [ 58 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:40.531117 [ 58 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:40.531130 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:40.531301 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:40.531376 [ 58 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 25
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:40.531389 [ 58 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:41.296205 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:38282
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:41.296404 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:42.448482 [ 53 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:42.448517 [ 53 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:42.448529 [ 53 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:42.448554 [ 53 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:42.448723 [ 53 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:42.448794 [ 53 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 26
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:42.448813 [ 53 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:44.301093 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:38296
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:44.301276 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:44.398027 [ 52 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:44.398075 [ 52 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:44.398088 [ 52 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:44.398103 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:44.398267 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:44.398358 [ 52 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 27
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:44.398372 [ 52 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:45.440479 [ 64 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:45.440516 [ 64 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:45.440528 [ 64 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:45.440576 [ 64 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:45.440721 [ 64 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:45.440797 [ 64 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 28
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:45.440814 [ 64 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:46.712990 [ 54 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:46.713035 [ 54 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:46.713049 [ 54 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:46.713063 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:46.713213 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:46.713298 [ 54 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 29
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:46.713311 [ 54 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:47.302420 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:58664
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:47.302646 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:48.688440 [ 60 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:48.688488 [ 60 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:48.688502 [ 60 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:48.688588 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:48.688759 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:48.688829 [ 60 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 30
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:48.688843 [ 60 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:50.207014 [ 58 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:50.207051 [ 58 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:50.207064 [ 58 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:50.207115 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:50.207224 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:50.207307 [ 58 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 31
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:50.207319 [ 58 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:50.295624 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:58674
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:50.295806 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:52.137563 [ 53 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:52.137612 [ 53 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:52.137625 [ 53 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:52.137638 [ 53 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:52.137826 [ 53 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:52.137893 [ 53 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 32
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:52.137908 [ 53 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:53.295956 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:58676
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:53.296117 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:53.537072 [ 52 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:53.537122 [ 52 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:53.537136 [ 52 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:53.537152 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:53.537362 [ 52 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:53.537477 [ 52 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 33
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:53.537496 [ 52 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:55.461787 [ 64 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:55.461832 [ 64 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:55.461846 [ 64 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:55.461860 [ 64 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:55.462028 [ 64 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:55.462110 [ 64 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 34
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:55.462121 [ 64 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:56.300453 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:49300
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:56.300661 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:57.193347 [ 54 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:57.193410 [ 54 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:57.193425 [ 54 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:57.193439 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:57.193679 [ 54 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:57.193767 [ 54 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 35
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:57.193780 [ 54 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:58.201980 [ 60 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:58.202027 [ 60 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:58.202041 [ 60 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:58.202055 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:58.202245 [ 60 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:58.202320 [ 60 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 36
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:58.202331 [ 60 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:59.300903 [ 36 ] {} <Trace> KeeperTCPHandlerFactory: Keeper request. Address: 127.0.0.1:49310
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:59.301087 [ 36 ] {} <Debug> KeeperTCPHandler: Receive four letter command ruok
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:59.421504 [ 58 ] {} <Warning> RaftInstance: Election timeout, initiate leader election
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:59.421567 [ 58 ] {} <Information> RaftInstance: [PRIORITY] decay, target 1 -> 1, mine 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:59.421582 [ 58 ] {} <Information> RaftInstance: [ELECTION TIMEOUT] current role: candidate, log last term 1, state term 1, target p 1, my p 1, hb dead, pre-vote NOT done
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:59.421598 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 2
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:59.421779 [ 58 ] {} <Information> RaftInstance: reset RPC client for peer 1
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:59.421858 [ 58 ] {} <Warning> RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 1, live 0, dead 1), at least 2 nodes should respond. failure count 37
chk-3-nodes-0 clickhouse-keeper 2024.10.01 19:48:59.421875 [ 58 ] {} <Information> RaftInstance: [PRE-VOTE INIT] my id 0, my role candidate, term 1, log idx 1, log term 1, priority (target 1 / mine 1)

Thank You.

@Slach
Copy link
Collaborator

Slach commented Oct 2, 2024

ok looks like something wrong with endpoints in headless service

Not found address of host: chk-3-nodes-1.chk-3-nodes-headless.test.svc.cluster.local. (DNS_ERROR)

Dns name should be resolved when pod have ready status

could you share
kubectl get svc -n test | grep chk?

@Slach Slach mentioned this issue Oct 2, 2024
Open
@arun-kbharathan-e3768
Copy link
Author

altinity_svc

@Slach
Copy link
Collaborator

Slach commented Oct 3, 2024

image

@arun-kbharathan-e3768
Copy link
Author

Sorry,
i am adding chk-3-nodes-1 pod describes as text here, even previous message also.

Name:             chk-3-nodes-1
Namespace:        test
Priority:         0
Service Account:  default
Node:             localk8s-control-plane/172.19.0.2
Start Time:       Wed, 02 Oct 2024 01:17:53 +0530
Labels:           app=chk-3-nodes
                  apps.kubernetes.io/pod-index=1
                  controller-revision-hash=chk-3-nodes-6547b64b95
                  statefulset.kubernetes.io/pod-name=chk-3-nodes-1
                  uid=8b302e39-f7a4-4ea0-8131-36c535283fb9
Annotations:      kubectl.kubernetes.io/restartedAt: 2024-10-01T19:47:46.799435259Z
Status:           Running
IP:               10.244.0.7
IPs:
  IP:           10.244.0.7
Controlled By:  StatefulSet/chk-3-nodes
Init Containers:
  server-id-injector:
    Container ID:  containerd://c5b5280080be4d57187eff7fbc1b11dbb80c8683e467f7f4892ad4c6d8b93333
    Image:         bash
    Image ID:      docker.io/library/bash@sha256:ce062497c248eb1cf4d32927f8c1780cce158d3ed0658c586a5be7308d583cbb
    Port:          <none>
    Host Port:     <none>
    Command:
      bash
      -xc
      export KEEPER_ID=${HOSTNAME##*-}; sed "s/KEEPER_ID/${KEEPER_ID}/g" /tmp/clickhouse-keeper/keeper_config.xml > /etc/clickhouse-keeper/keeper_config.xml; cat /etc/clickhouse-keeper/keeper_config.xml
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 03 Oct 2024 10:29:35 +0530
      Finished:     Thu, 03 Oct 2024 10:29:35 +0530
    Ready:          True
    Restart Count:  3
    Environment:    <none>
    Mounts:
      /etc/clickhouse-keeper from etc-clickhouse-keeper (rw)
      /tmp/clickhouse-keeper from keeper-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gtjwz (ro)
Containers:
  clickhouse-keeper:
    Container ID:   containerd://e821006f4b7dfb5615aa78584f64b15b5ec7b6d48dc76539906723df86fb5d94
    Image:          clickhouse/clickhouse-keeper:latest-alpine
    Image ID:       docker.io/clickhouse/clickhouse-keeper@sha256:067bf2ba6577b93d692d3a8ca19f3653f7326af7cb214f59060bbdf22c5477ff
    Ports:          2181/TCP, 9444/TCP, 7000/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 03 Oct 2024 12:20:14 +0530
      Finished:     Thu, 03 Oct 2024 12:20:15 +0530
    Ready:          False
    Restart Count:  49
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:        100m
      memory:     30Mi
    Liveness:     exec [bash -xc date && OK=$(exec 3<>/dev/tcp/127.0.0.1/2181; printf 'ruok' >&3; IFS=; tee <&3; exec 3<&-;);if [[ "${OK}" == "imok" ]]; then exit 0; else exit 1; fi] delay=60s timeout=1s period=3s #success=1 #failure=10
    Environment:  <none>
    Mounts:
      /etc/clickhouse-keeper from etc-clickhouse-keeper (rw)
      /var/lib/clickhouse-keeper from working-dir (rw)
      /var/lib/clickhouse-keeper/coordination/logs from log-storage-path (rw)
      /var/lib/clickhouse-keeper/coordination/snapshots from snapshot-storage-path (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gtjwz (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  log-storage-path:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  snapshot-storage-path:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  working-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  etc-clickhouse-keeper:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  keeper-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      chk-3-nodes
    Optional:  false
  kube-api-access-gtjwz:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Warning  BackOff  3m42s (x486 over 112m)  kubelet  Back-off restarting failed container clickhouse-keeper in pod chk-3-nodes-1_test(588a5c1b-e1ed-4817-8828-d7451802fc4b)

Get SVC

❯ kubectl get svc -n test | grep chk
chk-3-nodes                   ClusterIP   10.96.9.161    <none>        2181/TCP,7000/TCP   35h
chk-3-nodes-headless          ClusterIP   None           <none>        9444/TCP            35h

Describe headless svc

❯ kubectl describe svc chk-3-nodes-headless -n test
Name:                     chk-3-nodes-headless
Namespace:                test
Labels:                   <none>
Annotations:              <none>
Selector:                 app=chk-3-nodes,uid=8b302e39-f7a4-4ea0-8131-36c535283fb9
Type:                     ClusterIP
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       None
IPs:                      None
Port:                     raft  9444/TCP
TargetPort:               9444/TCP
Endpoints:                10.244.0.8:9444,10.244.0.7:9444,10.244.0.6:9444
Session Affinity:         None
Internal Traffic Policy:  Cluster
Events:                   <none>

@Slach
Copy link
Collaborator

Slach commented Oct 3, 2024

kubectl logs -n test chk-3-nodes-1 --since=24h

@arun-kbharathan-e3768
Copy link
Author

kubectl logs -n test chk-3-nodes-1 --since=24h

❯ kubectl logs -n test chk-3-nodes-1 --since=24h
Defaulted container "clickhouse-keeper" out of: clickhouse-keeper, server-id-injector (init)
Processing configuration file '/etc/clickhouse-keeper/keeper_config.xml'.
2024.10.03 07:10:42.269736 [ 1 ] {} <Information> SentryWriter: Sending crash reports is disabled

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Slach @arun-kbharathan-e3768