ci: etcd using the /livez for LivenessProbe and /readyz for StartupProbe.

[liangyuanpeng]

What type of PR is this?

/kind cleanup
etcd using the /livez for LivenessProbe and /readyz for StartupProbe at CI first,
part of #5419
What this PR does / why we need it:

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

[karmada-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign xishanyongye-chang for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[liangyuanpeng]

/cc @zhzhuang-zju

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 30.85%. Comparing base (a184b40) to head (d4ecff3).

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5438      +/-   ##
==========================================
- Coverage   30.85%   30.85%   -0.01%     
==========================================
  Files         640      640              
  Lines       44384    44384              
==========================================
- Hits        13695    13694       -1     
- Misses      29709    29710       +1     
  Partials      980      980              

Flag	Coverage Δ
unittests	30.85% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[liangyuanpeng]

this is from kubeadm

[zhzhuang-zju]

/assign

[zhzhuang-zju]

The port should be 2379, as our listen-client-urls is set to https://0.0.0.0:2379.

[zhzhuang-zju]

    Liveness:     http-get http://127.0.0.1:2381/livez delay=10s timeout=15s period=10s #success=1 #failure=8
    Readiness:    http-get http://127.0.0.1:2381/readyz delay=0s timeout=15s period=1s #success=1 #failure=3
    Startup:      http-get http://127.0.0.1:2381/readyz delay=10s timeout=15s period=10s #success=1 #failure=24

[liangyuanpeng]

kubeadm have the arg of --listen-metrics-urls=http://127.0.0.1:2381, so this is 2381, and karmada have not set it.

[zhzhuang-zju]

Is once per second too frequent?

[liangyuanpeng]

This is the expected effect of etcd readyz API design, see https://docs.google.com/document/d/1PaUAp76j1X92h3jZF47m32oVlR8Y-p-arB5XOB7Nb6U/edit

[zhzhuang-zju]

The initialDelaySeconds can be set a bit higher.

[zhzhuang-zju]

@liangyuanpeng The livenessProbe should properly reflect the fact that etcd needs a restart. Do you know how /livez accomplishes this? One benefit of using etcdctl get /registry --prefix --keys-only for the liveness check is that if it succeeds, then etcd's external functionality is normal.
In addition, when I used /livez locally, I noticed that kubelet logs the following:

Aug 28 02:48:37 karmada-host-control-plane kubelet[682]: E0828 02:48:37.991001     682 prober.go:239] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=13477 actualBytes=10240

This can cause etcd to restart continuously.

[liangyuanpeng]

@zhzhuang-zju Thank you for your time! seems like this have some weird problem, let me make it to draft first.

Do you know how /livez accomplishes this

https://docs.google.com/document/d/1PaUAp76j1X92h3jZF47m32oVlR8Y-p-arB5XOB7Nb6U/edit#heading=h.78g8a1kutqd8
And also update it to issue.