Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update of Supervisor failed version 2024.11.3 #5405

Open
jverlaan opened this issue Nov 11, 2024 · 83 comments
Open

Update of Supervisor failed version 2024.11.3 #5405

jverlaan opened this issue Nov 11, 2024 · 83 comments
Labels

Comments

@jverlaan
Copy link

jverlaan commented Nov 11, 2024

ℹ️ ℹ️ ℹ️ We understand what the issue is and are working on a solution. Please be patient. For now, please ignore any warnings about having an "unhealthy" supervisor. ℹ️ ℹ️ ℹ️

For more background, please read this comment.

Describe the issue you are experiencing

Home Assistant Supervisor
Installed version 2024.11.2
Latest version 2024.11.3

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/commands.py:245
integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 9:49:21 AM (2 occurrences)
Last logged: 9:49:51 AM

[140507219199600] Unexpected exception
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 262, in async_install
await self.coordinator.supervisor_client.supervisor.update()
File "/usr/local/lib/python3.12/site-packages/aiohasupervisor/supervisor.py", line 43, in update
await self._client.post(
File "/usr/local/lib/python3.12/site-packages/aiohasupervisor/client.py", line 173, in post
return await self._request(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohasupervisor/client.py", line 122, in _request
raise exc_type(result.message, result.job_id)
aiohasupervisor.exceptions.SupervisorBadRequestError: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 500 Server Error for http+docker://localhost/v1.47/images/create?tag=2024.11.3&fromImage=ghcr.io%2Fhome-assistant%2Famd64-hassio-supervisor&platform=linux%2Famd64: Internal Server Error ("toomanyrequests: retry-after: 18µs, allowed: 44000/minute")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service
response = await hass.services.async_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2802, in async_call
response_data = await coro
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2845, in _execute_service
return await target(service_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1007, in entity_service_call
single_response = await _handle_entity_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1079, in _handle_entity_call
result = await task
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/update/init.py", line 157, in async_install
await entity.async_install_with_progress(version, backup)
File "/usr/src/homeassistant/homeassistant/components/update/init.py", line 502, in async_install_with_progress
await self.async_install(version, backup)
File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 264, in async_install
raise HomeAssistantError(
homeassistant.exceptions.HomeAssistantError: Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 500 Server Error for http+docker://localhost/v1.47/images/create?tag=2024.11.3&fromImage=ghcr.io%2Fhome-assistant%2Famd64-hassio-supervisor&platform=linux%2Famd64: Internal Server Error ("toomanyrequests: retry-after: 18µs, allowed: 44000/minute")

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Steps to reproduce the issue

  1. Updates says there is a Supervisor Update
  2. Start installing the update
  3. After a few seconds the error messages apairs
    ...

Anything in the Supervisor logs that might be useful for us?

2024-11-11 06:37:37.504 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
2024-11-11 06:37:37.509 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
2024-11-11 06:37:37.510 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
2024-11-11 06:40:50.358 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2024-11-11 07:10:50.911 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2024-11-11 07:17:55.523 INFO (MainThread) [supervisor.host.info] Updating local host information
2024-11-11 07:17:56.644 INFO (MainThread) [supervisor.host.services] Updating service information
2024-11-11 07:17:56.668 INFO (MainThread) [supervisor.host.network] Updating local network information
2024-11-11 07:17:57.312 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information
2024-11-11 07:17:57.331 INFO (MainThread) [supervisor.host.manager] Host information reload completed
2024-11-11 07:37:01.454 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
2024-11-11 07:37:37.514 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state running
2024-11-11 07:37:37.516 INFO (MainThread) [supervisor.resolution.checks.base] Run check for docker_config/system
2024-11-11 07:37:37.520 INFO (MainThread) [supervisor.resolution.checks.base] Run check for no_current_backup/system
2024-11-11 07:37:37.529 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_ipv6_error/dns_server
2024-11-11 07:37:37.531 INFO (MainThread) [supervisor.resolution.checks.base] Run check for ipv4_connection_problem/system
2024-11-11 07:37:37.533 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_failed/dns_server
2024-11-11 07:37:37.542 INFO (MainThread) [supervisor.resolution.checks.base] Run check for trust/supervisor
2024-11-11 07:37:37.580 INFO (MainThread) [supervisor.resolution.checks.base] Run check for pwned/addon
2024-11-11 07:37:37.582 INFO (MainThread) [supervisor.resolution.checks.base] Run check for multiple_data_disks/system
2024-11-11 07:37:37.583 INFO (MainThread) [supervisor.resolution.checks.base] Run check for disabled_data_disk/system
2024-11-11 07:37:37.584 INFO (MainThread) [supervisor.resolution.checks.base] Run check for security/core
2024-11-11 07:37:37.588 INFO (MainThread) [supervisor.resolution.checks.base] Run check for free_space/system
2024-11-11 07:37:37.589 INFO (MainThread) [supervisor.resolution.check] System checks complete
2024-11-11 07:37:37.590 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running
2024-11-11 07:37:38.022 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
2024-11-11 07:37:38.023 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
2024-11-11 07:37:38.023 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
2024-11-11 07:38:27.814 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/thomasmauerer/hassio-addons repository
2024-11-11 07:38:27.829 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/music-assistant/home-assistant-addon repository
2024-11-11 07:38:27.837 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
2024-11-11 07:38:27.848 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/hassio-addons/repository repository
2024-11-11 07:38:27.862 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/home-assistant-addon repository
2024-11-11 07:38:30.903 INFO (MainThread) [supervisor.store] Loading add-ons from store: 81 all - 0 new - 0 remove
2024-11-11 07:38:30.903 INFO (MainThread) [supervisor.store] Loading add-ons from store: 81 all - 0 new - 0 remove
2024-11-11 07:40:51.377 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2024-11-11 08:10:51.959 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2024-11-11 08:37:38.026 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state running
2024-11-11 08:37:38.026 INFO (MainThread) [supervisor.resolution.checks.base] Run check for docker_config/system
2024-11-11 08:37:38.027 INFO (MainThread) [supervisor.resolution.checks.base] Run check for no_current_backup/system
2024-11-11 08:37:38.027 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_ipv6_error/dns_server
2024-11-11 08:37:38.097 INFO (MainThread) [supervisor.resolution.checks.base] Run check for ipv4_connection_problem/system
2024-11-11 08:37:38.100 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_failed/dns_server
2024-11-11 08:37:38.104 INFO (MainThread) [supervisor.resolution.checks.base] Run check for trust/supervisor
2024-11-11 08:37:38.118 INFO (MainThread) [supervisor.resolution.checks.base] Run check for pwned/addon
2024-11-11 08:37:38.120 INFO (MainThread) [supervisor.resolution.checks.base] Run check for multiple_data_disks/system
2024-11-11 08:37:38.120 INFO (MainThread) [supervisor.resolution.checks.base] Run check for disabled_data_disk/system
2024-11-11 08:37:38.121 INFO (MainThread) [supervisor.resolution.checks.base] Run check for security/core
2024-11-11 08:37:38.122 INFO (MainThread) [supervisor.resolution.checks.base] Run check for free_space/system
2024-11-11 08:37:38.122 INFO (MainThread) [supervisor.resolution.check] System checks complete
2024-11-11 08:37:38.122 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running
2024-11-11 08:37:38.422 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
2024-11-11 08:37:38.424 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
2024-11-11 08:37:38.424 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
2024-11-11 08:40:52.540 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2024-11-11 09:10:54.124 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2024-11-11 09:24:37.333 INFO (MainThread) [supervisor.host.info] Updating local host information
2024-11-11 09:24:38.415 INFO (MainThread) [supervisor.host.services] Updating service information
2024-11-11 09:24:38.431 INFO (MainThread) [supervisor.host.network] Updating local network information
2024-11-11 09:24:38.797 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information
2024-11-11 09:24:38.821 INFO (MainThread) [supervisor.host.manager] Host information reload completed
2024-11-11 09:37:01.628 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
2024-11-11 09:37:01.783 INFO (MainThread) [supervisor.misc.tasks] Found new Supervisor version 2024.11.3, updating
2024-11-11 09:37:01.784 INFO (MainThread) [supervisor.supervisor] Fetching AppArmor profile https://version.home-assistant.io/apparmor_stable.txt
2024-11-11 09:37:01.838 INFO (MainThread) [supervisor.host.apparmor] Adding/updating AppArmor profile: hassio-supervisor
2024-11-11 09:37:02.173 INFO (MainThread) [supervisor.supervisor] Update Supervisor to version 2024.11.3
2024-11-11 09:37:02.175 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/amd64-hassio-supervisor with tag 2024.11.3.
2024-11-11 09:37:12.419 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3")
2024-11-11 09:37:12.420 INFO (MainThread) [supervisor.resolution.module] Create new issue update_failed - supervisor / None
2024-11-11 09:37:12.479 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3")
2024-11-11 09:37:38.426 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state running
2024-11-11 09:37:38.428 INFO (MainThread) [supervisor.resolution.checks.base] Run check for docker_config/system
2024-11-11 09:37:38.429 INFO (MainThread) [supervisor.resolution.checks.base] Run check for no_current_backup/system
2024-11-11 09:37:38.431 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_ipv6_error/dns_server
2024-11-11 09:37:38.432 INFO (MainThread) [supervisor.resolution.checks.base] Run check for ipv4_connection_problem/system
2024-11-11 09:37:38.433 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_failed/dns_server
2024-11-11 09:37:38.435 INFO (MainThread) [supervisor.resolution.checks.base] Run check for trust/supervisor
2024-11-11 09:37:38.464 INFO (MainThread) [supervisor.resolution.checks.base] Run check for pwned/addon
2024-11-11 09:37:38.468 INFO (MainThread) [supervisor.resolution.checks.base] Run check for multiple_data_disks/system
2024-11-11 09:37:38.469 INFO (MainThread) [supervisor.resolution.checks.base] Run check for disabled_data_disk/system
2024-11-11 09:37:38.471 INFO (MainThread) [supervisor.resolution.checks.base] Run check for security/core
2024-11-11 09:37:38.475 INFO (MainThread) [supervisor.resolution.checks.base] Run check for free_space/system
2024-11-11 09:37:38.478 INFO (MainThread) [supervisor.resolution.check] System checks complete
2024-11-11 09:37:38.481 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running
2024-11-11 09:37:38.797 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
2024-11-11 09:37:38.798 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
2024-11-11 09:37:38.798 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
2024-11-11 09:40:54.771 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2024-11-11 09:49:20.149 INFO (MainThread) [supervisor.supervisor] Fetching AppArmor profile https://version.home-assistant.io/apparmor_stable.txt
2024-11-11 09:49:20.280 INFO (MainThread) [supervisor.host.apparmor] Adding/updating AppArmor profile: hassio-supervisor
2024-11-11 09:49:20.494 INFO (MainThread) [supervisor.supervisor] Update Supervisor to version 2024.11.3
2024-11-11 09:49:20.495 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/amd64-hassio-supervisor with tag 2024.11.3.
2024-11-11 09:49:21.428 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 500 Server Error for http+docker://localhost/v1.47/images/create?tag=2024.11.3&fromImage=ghcr.io%2Fhome-assistant%2Famd64-hassio-supervisor&platform=linux%2Famd64: Internal Server Error ("toomanyrequests: retry-after: 18µs, allowed: 44000/minute")
2024-11-11 09:49:21.502 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 500 Server Error for http+docker://localhost/v1.47/images/create?tag=2024.11.3&fromImage=ghcr.io%2Fhome-assistant%2Famd64-hassio-supervisor&platform=linux%2Famd64: Internal Server Error ("toomanyrequests: retry-after: 18µs, allowed: 44000/minute")
2024-11-11 09:49:38.610 INFO (MainThread) [supervisor.supervisor] Fetching AppArmor profile https://version.home-assistant.io/apparmor_stable.txt
2024-11-11 09:49:38.720 INFO (MainThread) [supervisor.host.apparmor] Adding/updating AppArmor profile: hassio-supervisor
2024-11-11 09:49:38.940 INFO (MainThread) [supervisor.supervisor] Update Supervisor to version 2024.11.3
2024-11-11 09:49:38.943 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/amd64-hassio-supervisor with tag 2024.11.3.
2024-11-11 09:49:51.084 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3")
2024-11-11 09:49:51.109 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3")

System Health information

System Information

version core-2024.11.1
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.4
os_name Linux
os_version 6.6.54-haos
arch x86_64
timezone Europe/Amsterdam
config_dir /config
Home Assistant Community Store
GitHub API ok
GitHub Content ok
GitHub Web ok
HACS Data ok
GitHub API Calls Remaining 5000
Installed Version 2.0.1
Stage running
Available Repositories 1461
Downloaded Repositories 8
Home Assistant Cloud
logged_in false
can_reach_cert_server ok
can_reach_cloud_auth ok
can_reach_cloud ok
Home Assistant Supervisor
host_os Home Assistant OS 13.2
update_channel stable
supervisor_version supervisor-2024.11.2
agent_version 1.6.0
docker_version 27.2.0
disk_total 56.0 GB
disk_used 20.0 GB
healthy true
supported true
host_connectivity true
supervisor_connectivity true
ntp_synchronized true
virtualization
board generic-x86-64
supervisor_api ok
version_api ok
installed_addons File editor (5.8.0), Studio Code Server (5.17.3), MariaDB (2.7.1), Terminal & SSH (9.15.0), Duck DNS (1.18.0), NGINX Home Assistant SSL proxy (3.11.0), InfluxDB (5.0.1), Grafana (10.1.3), Tailscale (0.23.2), Samba Backup (5.2.0), Dnsmasq (1.8.1)
Dashboards
dashboards 5
resources 1
views 11
mode storage
Recorder
oldest_recorder_run November 1, 2024 at 9:43 AM
current_recorder_run November 9, 2024 at 1:08 PM
estimated_db_size 1245.77 MiB
database_engine mysql
database_version 10.11.6
Spotify
api_endpoint_reachable ok

Supervisor diagnostics

Is HA OS. See first comment.

Additional information

No response

@jverlaan jverlaan added the bug label Nov 11, 2024
@Fidelity88
Copy link

Same issue, same error! Also using Home Assistant OS

@ketzerkadaver
Copy link

Same problem here!

@agners
Copy link
Member

agners commented Nov 11, 2024

Internal Server Error ("toomanyrequests: retry-after: 18µs, allowed: 44000/minute")

It seems we hit some type of rate limiting 🤔 . This likely will resolve after a while.

@Pussy123a
Copy link

Same problem here!

@eddysteurs
Copy link

me too

@sirlamoid
Copy link

Same problem here

@minmaxat
Copy link

same here, dont think that it is a rate limit:

Die Aktion update/install konnte nicht ausgeführt werden. Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3")

@adamprice56
Copy link

adamprice56 commented Nov 11, 2024

Same issue here, Running HAOS 13.2 and core version 2024.10.3

Just tried pulling the image from another (Non HA) machine and it shows "too many requests". This looks to be a server side/github issue. I guess the 404 is HA checking if the image is already downloaded?

Edit: Looks like the issue is clearing up as I was able to pull the image from my server (Not running HA)

@Fidelity88
Copy link

Fidelity88 commented Nov 11, 2024

Indeed, it's not related to rate limiting. If you visit the URL of the error log multiple times (from the same network/IP), you will notice that you get the same rate limiting error in your local browser:
https://ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3

But when you do not, it links to the github page with images. I guess the required image is missing for some reason. HASSOS keeps trying to get that image, which results in rate limiting.

For that reason #5406 should be combined with this issue.

@Pinni3
Copy link

Pinni3 commented Nov 11, 2024

Same issue here

@Sinyk-NFA
Copy link

Sinyk-NFA commented Nov 11, 2024

Same issue, same error! Also using Home Assistant OS

Ditto!

Over-dramatic and hard put upon sigh of despondency

@N8schatten
Copy link

N8schatten commented Nov 11, 2024

Same Problem

2024-11-11 10:22:23.474 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3")
2024-11-11 10:22:23.475 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3")

@kgiedrius
Copy link

Failed to perform the action update/install. Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.45/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3")

@Bloodpack
Copy link

same here

@billmaterial
Copy link

Same here, has anyone found a fix yet?

Failed to perform the action update/install. Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3")

@TorstenHamper
Copy link

+1
same error message like all above.
Duplicated with #5406

@Otiel
Copy link

Otiel commented Nov 11, 2024

No fix yet, stop replying with "same here", but simply add a 👍 to the original issue, this will stop spamming those of us who subscribed to the issue.

Wait for the team to fix this, you probably have no rush updating Supervisor.

@fourmonks
Copy link

Same issue.

@garv3
Copy link

garv3 commented Nov 11, 2024

Same for me!

@Pinni3
Copy link

Pinni3 commented Nov 11, 2024

Actie update/install kon niet worden uitgevoerd. Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 500 Server Error for http+docker://localhost/v1.47/images/create?tag=2024.11.3&fromImage=ghcr.io%2Fhome-assistant%2Famd64-hassio-supervisor&platform=linux%2Famd64: Internal Server Error ("toomanyrequests: retry-after: 1.203566ms, allowed: 44000/minute")

@pickonedev
Copy link

Same issue. Now I even get this...

image

@werfpsa
Copy link

werfpsa commented Nov 11, 2024

Same issue for me!

@RockFlare
Copy link

Same issue.

@rama31244
Copy link

Same error
Failed to perform the action update/install. Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3")

@drorya1
Copy link

drorya1 commented Nov 11, 2024

Same here
Failed to perform the action update/install. Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3")

@palsbo
Copy link

palsbo commented Nov 11, 2024

Same problem

@TheHuskyCrobats
Copy link

Same issue

@fwartner
Copy link

Same over here. Thought its ratelimited because of a service in my homelab.. Seems i was wrong ^^

@KirovAir
Copy link

KirovAir commented Nov 11, 2024

Failed to perform the action update/install. Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3")

It seems the request limit is hit since this update is needed to update home-assistant core. 44000 downloads / minute is quite a high threshold though so maybe Github is having some internal issues.

@jf-uu
Copy link

jf-uu commented Nov 11, 2024

TIL you out you can subscribe to just close notifications, seems like the best option:
image

@spanzetta
Copy link

TIL you out you can subscribe to just close notifications, seems like the best option: image

this is the best .. I agree (and I didn't know that)
Thanks

@moongear
Copy link

TIL you out you can subscribe to just close notifications, seems like the best option: image

This is good information. Thanks for sharing.

@agners
Copy link
Member

agners commented Nov 11, 2024

We understand what the issue is and are working on a solution. Please be patient. For now, please ignore any warnings about having an "unhealthy" supervisor.

@agners
Copy link
Member

agners commented Nov 11, 2024

What happened

We changed the Supervisor update a while ago to make Supervisor update as soon as it learns about the update (see #5375). This was an intentional change, which has been tested on the beta channel since a couple of weeks, and rolled out already last week to stable.

Today's release was the first time where this new update mechanism got used when we published a new version to stable. The update had the side effect that updates rolled out within a 2h window (instead of 8h window), which was not handled well by the GitHub container registry (causing all the toomanyrequests).

What is the fix

The latest version of Supervisor has been pulled from Supervisor about 2h ago, which means that you should no longer see the update notification. So for most users, the errors in the logs should stop from appearing and the update notification disappears on it's own.

For users who have rebooted the system in the last ~8h they also see a message "Unhealthy system - Supervisor update failed". This message appears because despite a good network connection, we were not able to update the Supervisor. Another reboot of the system should make this message disappear as well.

What if I updated successfully

If you successfully updated to Supervisor 2024.11.3, that is fine too. The version itself is fine. The system won't rollback. We'll eventually release a new version of Supervisor so you'll have the same stable version as anyone else.

What happens from now

We are currently working on solutions to rollout the new Supervisor version without hitting the rate limit.

@home-assistant home-assistant unlocked this conversation Nov 11, 2024
@jj123451
Copy link

I restarted the HA, but still have the message when trying to install addons. Not sure if there is something I should do, or need to wait for the rollout soultion you are working on.
Failed to to call /addons/9074a9fa_cloudflared/install - 'AddonManager.install' blocked from execution, system is not healthy - supervisor

@alexh3o
Copy link

alexh3o commented Nov 11, 2024

I restarted the HA, but still have the message when trying to install addons. Not sure if there is something I should do, or need to wait for the rollout soultion you are working on. Failed to to call /addons/9074a9fa_cloudflared/install - 'AddonManager.install' blocked from execution, system is not healthy - supervisor

I was in the same case, rebooting just the core left me unable to update an add-on, but rebooting ha OS did the trick

@jj123451
Copy link

jj123451 commented Nov 11, 2024

I was in the same case, rebooting just the core left me unable to update an add-on, but rebooting ha OS did the trick

It worked - thank you!

@Dubblbindong
Copy link

For my case its working again - Thank you

@BwezB
Copy link

BwezB commented Nov 11, 2024

If it helps anyone, for a full reboot, you go to System->Power icon in left upper corner->Advanced options->Reboot system

@Poltinnik650
Copy link

Я так же не могу войти в систему... Ну что за проблемы?..

@Acalanum
Copy link

I seem to still be having this issue I believe...maybe I'm in the wrong spot...

Trying to update to Core 2024.11.0

In the supervisor logs, I'm getting this:

2024-11-11 15:48:02.130 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/qemux86-64-homeassistant:2024.11.0: 500 Server Error for http+docker://localhost/v1.47/images/create?tag=2024.11.0&fromImage=ghcr.io%2Fhome-assistant%2Fqemux86-64-homeassistant&platform=linux%2Famd64: Internal Server Error ("Get "https://ghcr.io/v2/": net/http: request canceled (Client.Timeout exceeded while awaiting headers)")

Things I've tried:
Removed HACS completely (thought that was an issue)
Restarted Home assistant
rebooted VM that Home assistant runs on via Proxmox

Unfortunately still get this message. If this is unrelated to this thread, could someone point me to the right spot?

Thanks!

@smarthomefamilyverrips
Copy link

I seem to still be having this issue I believe...maybe I'm in the wrong spot...

Trying to update to Core 2024.11.0

In the supervisor logs, I'm getting this:

2024-11-11 15:48:02.130 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/qemux86-64-homeassistant:2024.11.0: 500 Server Error for http+docker://localhost/v1.47/images/create?tag=2024.11.0&fromImage=ghcr.io%2Fhome-assistant%2Fqemux86-64-homeassistant&platform=linux%2Famd64: Internal Server Error ("Get "https://ghcr.io/v2/": net/http: request canceled (Client.Timeout exceeded while awaiting headers)")

Things I've tried: Removed HACS completely (thought that was an issue) Restarted Home assistant rebooted VM that Home assistant runs on via Proxmox

Unfortunately still get this message. If this is unrelated to this thread, could someone point me to the right spot?

Thanks!

@Acalanum that is a Home Assistant Core issue not a supervisor issue and are most likely not related, you can post your issue here https://github.com/home-assistant/core/issues

@alexlopezit
Copy link

alexlopezit commented Nov 12, 2024

I was in the same case, rebooting just the core left me unable to update an add-on, but rebooting ha OS did the trick

Thanks, this worked for me too and I can now update other depending addons

I ran the reboot from the VM console (I use Proxmox) doing:

host reboot

@jgate
Copy link

jgate commented Nov 12, 2024

I have this issue during Supervisor update to 2024.11.3

2024-11-11 12:28:02.043 INFO (MainThread) [__main__] Running Supervisor
2024-11-11 12:28:02.054 INFO (MainThread) [supervisor.os.manager] Rauc: slot B - activated slot kernel.1, marked slot kernel.1 as good
2024-11-11 12:28:02.061 INFO (MainThread) [supervisor.supervisor] Fetching AppArmor profile https://version.home-assistant.io/apparmor_stable.txt
2024-11-11 12:28:02.130 INFO (MainThread) [supervisor.host.apparmor] Adding/updating AppArmor profile: hassio-supervisor
2024-11-11 12:28:02.195 INFO (MainThread) [supervisor.supervisor] Update Supervisor to version 2024.11.3
2024-11-11 12:28:02.195 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/amd64-hassio-supervisor with tag 2024.11.3.
2024-11-11 12:28:14.567 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3")
2024-11-11 12:28:14.568 INFO (MainThread) [supervisor.resolution.module] Create new issue update_failed - supervisor / None
2024-11-11 12:28:14.592 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3")
2024-11-11 12:28:14.592 CRITICAL (MainThread) [supervisor.core] Can't update Supervisor! This will break some Add-ons or affect future versions of Home Assistant!
2024-11-11 12:28:14.606 INFO (MainThread) [supervisor.addons.manager] Phase 'initialize' starting 0 add-ons
2024-11-11 12:28:14.610 INFO (MainThread) [supervisor.addons.manager] Phase 'system' starting 2 add-ons
2024-11-11 12:28:14.866 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on homeassistant/amd64-addon-mosquitto with version 6.4.1
2024-11-11 12:28:15.103 INFO (MainThread) [supervisor.docker.addon] Starting Docker add-on ghcr.io/hassio-addons/tasmoadmin/amd64 with version 0.30.6
2024-11-11 12:28:17.360 INFO (MainThread) [supervisor.services.modules.mqtt] Set core_mosquitto as service provider for mqtt
2024-11-11 12:28:45.202 INFO (MainThread) [supervisor.addons.manager] Phase 'services' starting 8 add-ons



2024-11-11 12:29:59.497 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information
2024-11-11 12:29:59.503 INFO (MainThread) [supervisor.host.manager] Host information reload completed
2024-11-11 12:30:24.852 INFO (MainThread) [supervisor.supervisor] Fetching AppArmor profile https://version.home-assistant.io/apparmor_stable.txt
2024-11-11 12:30:25.007 INFO (MainThread) [supervisor.host.apparmor] Adding/updating AppArmor profile: hassio-supervisor
2024-11-11 12:30:25.062 INFO (MainThread) [supervisor.supervisor] Update Supervisor to version 2024.11.3
2024-11-11 12:30:25.062 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/amd64-hassio-supervisor with tag 2024.11.3.
2024-11-11 12:30:25.807 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 500 Server Error for http+docker://localhost/v1.47/images/create?tag=2024.11.3&fromImage=ghcr.io%2Fhome-assistant%2Famd64-hassio-supervisor&platform=linux%2Famd64: Internal Server Error ("toomanyrequests: retry-after: 573.992µs, allowed: 44000/minute")
2024-11-11 12:30:25.812 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 500 Server Error for http+docker://localhost/v1.47/images/create?tag=2024.11.3&fromImage=ghcr.io%2Fhome-assistant%2Famd64-hassio-supervisor&platform=linux%2Famd64: Internal Server Error ("toomanyrequests: retry-after: 573.992µs, allowed: 44000/minute")
2024-11-11 12:34:56.675 INFO (MainThread) [supervisor.api.middleware.security] /backups access from de91e161_hassio_onedrive_backup
2024-11-11 12:34:57.241 INFO (MainThread) [supervisor.api.middleware.security] /backups access from de91e161_hassio_onedrive_backup
2024-11-11 12:38:25.675 INFO (MainThread) [supervisor.supervisor] Fetching AppArmor profile https://version.home-assistant.io/apparmor_stable.txt
2024-11-11 12:38:25.866 INFO (MainThread) [supervisor.host.apparmor] Adding/updating AppArmor profile: hassio-supervisor
2024-11-11 12:38:25.919 INFO (MainThread) [supervisor.supervisor] Update Supervisor to version 2024.11.3
2024-11-11 12:38:25.919 INFO (MainThread) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/amd64-hassio-supervisor with tag 2024.11.3.
2024-11-11 12:38:27.614 ERROR (MainThread) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3")
2024-11-11 12:38:27.639 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3/json: Not Found ("No such image: ghcr.io/home-assistant/amd64-hassio-supervisor:2024.11.3")
2024-11-11 12:39:57.252 INFO (MainThread) [supervisor.api.middleware.security] /backups access from de91e161_hassio_onedrive_backup
2024-11-11 12:39:57.834 INFO (MainThread) [supervisor.api.middleware.security] /backups access from de91e161_hassio_onedrive_backup
2024-11-11 12:44:57.628 INFO (MainThread) [supervisor.api.middleware.security] /backups access from de91e161_hassio_onedrive_backup
2024-11-11 12:44:58.019 INFO (MainThread) [supervisor.api.middleware.security] /backups access from de91e161_hassio_onedrive_backup
2


2024-11-12 11:54:58.934 INFO (MainThread) [supervisor.jobs] 'Tasks._update_cli' blocked from execution, system is not healthy - supervisor
2024-11-12 11:57:15.814 INFO (MainThread) [supervisor.api.middleware.security] /backups access from de91e161_hassio_onedrive_backup
2024-11-12 11:57:16.359 INFO (MainThread) [supervisor.api.middleware.security] /backups access from de91e161_hassio_onedrive_backup
2024-11-12 12:00:21.295 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2024-11-12 12:01:03.555 WARNING (MainThread) [supervisor.jobs] 'AddonManager.update' blocked from execution, system is not healthy - supervisor
2024-11-12 12:01:45.199 INFO (MainThread) [supervisor.backups.manager] Found 90 backup files
2024-11-12 12:02:16.269 INFO (MainThread) [supervisor.api.middleware.security] /backups access from de91e161_hassio_onedrive_backup
2024-11-12 12:02:16.763 INFO (MainThread) [supervisor.api.middleware.security] /backups access from de91e161_hassio_onedrive_backup
2024-11-12 12:02:31.069 INFO (MainThread) [supervisor.backups.manager] Found 90 backup files
2024-11-12 12:07:16.572 INFO (MainThread) [supervisor.api.middleware.security] /backups access from de91e161_hassio_onedrive_backup

image

@agners
Copy link
Member

agners commented Nov 12, 2024

You largely can disregard any errors related to container updates in the time frame between 2024-11-11 09:32 and 14:55 GMT+1. This was during the problematic roll out attempt of Supervisor 2024.11.3. The rate limit applies for the whole Home Assistant organization, this means as a side effect during this time frame also other updates were affected (e.g. Core updates).

If your system is still marked unhealthy due to this incident, please read the solutions posted above (essentially, make sure you reboot the host or at least the Supervisor). If despite reboots unhealthy issues remains for you, is unlikely related to this particular rollout issue.

If you see Not Found ("No such image: ghcr.io/home-assistant/aarch64-hassio-supervisor:2024.11.3") or similar issues after the above timeframe, this as well is unlikely related to this incident the ghcr.io infrastructure is back in regular operation. These are often local or provider specific issues. Please open a separate issue if you still believe this is a Supervisor fault.

If in doubt, use our community forum, e.g. for discussions around this issue there is already a thread.

I am locking that issue to avoid the relevant comments getting burried. Please use the community forum, feel free to tag me there if something needs my attention, or open a new issue here in the Supervisor issue tracker.

@home-assistant home-assistant locked as too heated and limited conversation to collaborators Nov 12, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests