Week 29 routine

[kiwixbot]

Infra

• Check nodes free space

df -h / && df -h /data

• Nodes and worker system upgrades

apt update && apt upgrade

• Ensure all borg repositories are being updated
• Check Pod errors

k get pods -A -o wide|grep Error

• Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• Upgrade k8s if applicable

Stats

• Ensure download.kiwix.org stats are being recorded
• Check whether matomo should be upgraded

Grafana

• Alert list is normal
• Zimfarm dashboard is normal
• Jobs durations dashboard is normal
• There is no abnormal behaviors on cluster resources consumption

Projects

• UptimeRobot has no alert
• zimit backlog is reasonable
• Analyze zimit failed tasks
• PRs awaiting your review

Security

• Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Storage

• Nothing special (details lost in wrong window closing, too lazy to do it twice the same week)

Misc

• k8s needs to be updated to 1.29 ; let's wait for @rgaudin return to do this update, should anything go wrong I prefer to not be the only one on board ^^

• Jobs duration are sometimes abnormal, sometimes exceptionally fast. Ticket opened at Scaleway last week.

zimit

• pipe is now empty
• (4) NameResolutionError or similar
• (1) page response missing
• (1) ERR_CONNECTION_REFUSED
• (0) ERR_ABORTED
• (0) ERR_CONNECTION_CLOSED
• (4) INTERNAL ERROR (puppeteer) => opened Puppeteer INTERNAL ERROR  openzim/zimit#343
• (2) Navigation timeout of 90000 ms exceeded
• (0) ValueError: Title is too long.
• (0) Invalid seed specified
• (18) Unable to find WARC record for main page openzim/warc2zim#203 Unable to find WARC record for main page
• (5) LookupError: unknown encoding: unicode openzim/warc2zim#331 LookupError: unknown encoding
• (1) Some websites are raising HTTP2 errors on sisyphus worker openzim/zimit#329 Is HTTP2 an issue
• (1) ValueError: Incorrect HttpUrl scheme in value openzim/warc2zim#332 ValueError: Incorrect HttpUrl scheme in value
• () Can we do something when server return bad content-type openzim/warc2zim#346 Can we do something when server return bad content-type