🐛 BUG: Possible race condition or failure with static assets upload with wrangler deploy

[sleexyz]

Which Cloudflare product(s) does this pertain to?

Wrangler, Cloudflare workers with Static Assets enabled.

What version(s) of the tool(s) are you using?

3.95.0 [Wrangler]

What version of Node are you using?

20.18.1

What operating system and version are you using?

Ubuntu 22.04.5 LTS

Describe the Bug

Observed behavior

Our CI deploys have all of a sudden been failing non-deterministically. We deploy from Github CI using a custom script that shells out to npx wrangler deploy and it will take anywhere from 2 to 20 retries before the deploy passes. We have observed this happening since Dec 12 4:32 ET.

UPDATE: It seems like a possible race condition or failure related to the static assets upload. The worker gets created, and upon inspection in the Cloudflare Workers UI, I see a single worker.js file with contents:

addEventListener('fetch', e => { e.respondWith(new Response("Assets have not yet been deployed...")) });

The deploy error:

⛅️ wrangler 3.95.0
-------------------

🌀 Building list of assets...
🌀 Starting asset upload...
No files to upload. Proceeding with deployment...
Total Upload: 13672.97 KiB / gzip: 2870.17 KiB
Your worker has access to the following bindings:
- Durable Objects:
  - xxx: xxx
  - xxx: xxx

✘ [ERROR] A request to the Cloudflare API (/accounts/***/workers/scripts/***-api/versions) failed.

  The uploaded script has no registered event handlers. [code: 10068]
  
  If you think this is a bug, please open an issue at: https://github.com/cloudflare/workers-sdk/issues/new/choose

Expected behavior

Deploys work deterministically.

Steps to reproduce

Stripped down wrangler.toml:

name = "foo-api"
main = "src/facade/source.ts"
compatibility_date = "2024-11-06"
compatibility_flags = ["nodejs_compat"]
assets = { directory = "./public" }

[[durable_objects.bindings]]
name = "Foo"
class_name = "Foo"

[[migrations]]
tag = "v1"
new_classes = ["Foo"]

Please provide a link to a minimal reproduction

No response

Please provide any relevant error logs

No response

[sleexyz]

More details

⛅️ wrangler 3.95.0
-------------------
Saving to cache: {"account":{"id":"***","name":""}}
Starting verifyWorkerMatchesCITag() with tag: undefined, name: ***-api-preview-b127521
No WRANGLER_CI_MATCH_TAG variable provided, aborting verifyWorkerMatchesCITag()
-- START CF API REQUEST: GET https://api.cloudflare.com/client/v4/accounts/***/workers/services/***-api-preview-b127521
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API REQUEST
-- START CF API RESPONSE: Not Found 404
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API RESPONSE
-- START CF API REQUEST: GET https://api.cloudflare.com/client/v4/accounts/***/workers/scripts/***-api-preview-b127521/deployments
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API REQUEST
-- START CF API RESPONSE: Not Found 404
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API RESPONSE
Writing additional module to output /home/runner/work/***/***/party/.wrangler/tmp/deploy-b48BUq/e143f8942c2b331ae6a557a71d6dc5dac980db63-RELEASE_SYNC.wasm
Writing additional module to output /home/runner/work/***/***/party/.wrangler/tmp/deploy-b48BUq/21685ebdca0741d0a9662e14fc27a10[50](https://github.com/foo/bar/actions/runs/12357297856/job/34485210080#step:6:51)a4a30bd-photon_rs_bg.wasm
-- START CF API REQUEST: GET https://api.cloudflare.com/client/v4/accounts/***/workers/scripts
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API REQUEST
-- START CF API RESPONSE: OK 200
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API RESPONSE
🌀 Building list of assets...
🌀 Starting asset upload...
-- START CF API REQUEST: POST https://api.cloudflare.com/client/v4/accounts/***/workers/scripts/***-api-preview-b127[52](https://github.com/foo/bar/actions/runs/12357297856/job/34485210080#step:6:53)1/assets-upload-session
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API REQUEST
-- START CF API RESPONSE: OK 200
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API RESPONSE
🌀 Found 83 new or modified static assets to upload. Proceeding with upload...
+ /xxx/xxx.js
+ /xxx/xxx.js
+ /xxx/xxx.js
+ /xxx/xxx.js
+ /xxx/xxx.js
...
+ /xxx/xxx.js
-- START CF API REQUEST: POST https://api.cloudflare.com/client/v4/accounts/***/workers/assets/upload?base64=true
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
BODY: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API REQUEST
-- START CF API REQUEST: POST https://api.cloudflare.com/client/v4/accounts/***/workers/assets/upload?base64=true
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
BODY: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API REQUEST
-- START CF API REQUEST: POST https://api.cloudflare.com/client/v4/accounts/***/workers/assets/upload?base64=true
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
BODY: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API REQUEST
-- START CF API RESPONSE: Accepted 202
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API RESPONSE
Uploaded 27 of 83 assets
-- START CF API RESPONSE: Accepted 202
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API RESPONSE
Uploaded [55](https://github.com/foo/bar/actions/runs/12357297856/job/34485210080#step:6:56) of 83 assets
-- START CF API RESPONSE: Created 201
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API RESPONSE
Uploaded 83 of 83 assets
✨ Success! Uploaded 83 files (4.73 sec)
Total Upload: 13679.81 KiB / gzip: 2871.13 KiB
-- START CF API REQUEST: PUT https://api.cloudflare.com/client/v4/accounts/***/workers/scripts/***-api-preview-b127521?excludeScript=true
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
BODY: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API REQUEST
-- START CF API RESPONSE: Bad Request 400
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API RESPONSE
-- START CF API REQUEST: PUT https://api.cloudflare.com/client/v4/accounts/***/workers/scripts/***-api-preview-b127521?excludeScript=true
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
BODY: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API REQUEST
-- START CF API RESPONSE: Bad Request 400
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API RESPONSE
-- START CF API REQUEST: PUT https://api.cloudflare.com/client/v4/accounts/***/workers/scripts/***-api-preview-b127521?excludeScript=true
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
INIT: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
BODY: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API REQUEST
-- START CF API RESPONSE: Bad Request 400
HEADERS: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
RESPONSE: omitted; set WRANGLER_LOG_SANITIZE=false to include sanitized data
-- END CF API RESPONSE
Your worker has access to the following bindings:
- Durable Objects:
  - PARTYKIT_DURABLE: PartyKitDurable
  - site: site
  - project: project
  - session: session
  - fetch_api: fetch_api
  - project_revision: project_revision
✘ [ERROR] A request to the Cloudflare API (/accounts/***/workers/scripts/***-api-preview-b127521) failed.
  The uploaded script has no registered event handlers. [code: 10068]
  
  If you think this is a bug, please open an issue at: https://github.com/cloudflare/workers-sdk/issues/new/choose
Cloudflare collects anonymous telemetry about your usage of Wrangler. Learn more at https://github.com/cloudflare/workers-sdk/tree/main/packages/wrangler/telemetry.md
Metrics dispatcher: Posting data {"deviceId":"637c361a-afb7-4f3c-b9b9-8fdf993665f6","event":"wrangler command errored","timestamp":1734367780701,"properties":{"amplitude_session_id":17343677[56](https://github.com/foo/bar/actions/runs/12357297856/job/34485210080#step:6:57)791,"amplitude_event_id":1,"wranglerVersion":"3.95.0","osPlatform":"Linux","osVersion":"#20-Ubuntu SMP Tue Oct 22 03:43:13 UTC 2024","nodeVersion":20,"packageManager":"npm","isFirstUsage":false,"configFileType":"toml","isCI":true,"isInteractive":false,"argsUsed":["c","config","define","j","name","xGradualRollouts","xJsonConfig","xVersions"],"argsCombination":"c, config, define, j, name, xGradualRollouts, xJsonConfig, xVersions","command":"wrangler deploy","args":{"c":"<REDACTED>","config":"<REDACTED>","name":"<REDACTED>","define":["<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTE
Retrying...
🪵  Writing logs to "/home/runner/.config/.wrangler/logs/wrangler-2024-12-16_16-49-43_870.log"
Metrics dispatcher: Posting data {"deviceId":"637c3[61](https://github.com/foo/bar/actions/runs/12357297856/job/34485210080#step:6:62)a-afb7-4f3c-b9b9-8fdf993665f6","event":"wrangler command started","timestamp":1734367784009,"properties":{"amplitude_session_id":1734367784008,"amplitude_event_id":0,"wranglerVersion":"3.95.0","osPlatform":"Linux","osVersion":"#20-Ubuntu SMP Tue Oct 22 03:43:13 UTC 2024","nodeVersion":20,"packageManager":"npm","isFirstUsage":false,"configFileType":"toml","isCI":true,"isInteractive":false,"argsUsed":["c","config","define","j","name","xGradualRollouts","xJsonConfig","xVersions"],"argsCombination":"c, config, define, j, name, xGradualRollouts, xJsonConfig, xVersions","command":"wrangler deploy","args":{"c":"<REDACTED>","config":"<REDACTED>","name":"<REDACTED>","define":["<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTED>","<REDACTE
.env file not found at ".env". Continuing... For more details, refer to https://developers.cloudflare.com/workers/wrangler/system-environment-

[petebacondarwin]

I don't think this is a race condition. The way that asset uploads work is that they need at least one deployed worker to attach the assets to. So on a brand new Worker we create a placeholder.
From your logs it looks like we successfully created the placeholder worker, and then successfully uploaded the assets.
The error comes when we try to deploy the actual worker. This is the relevant error message:

The uploaded script has no registered event handlers. [code: 10068]

Can you check that your Worker is actually valid? Did you forget to provide a default export containing a fetch() handler or similar? I see you have a bunch of DOs. Perhaps you are only exporting those and not the Worker that fronts them?

[sleexyz]

We resolved the issue by moving two packages 1 2 to be lazily imported (via dynamic imports).

My guess is there was some sort of non-determinism between the event handler existence check and module initialization side-effects?