HTTP Error: 503 in functions emulator adding trigger when using host 0.0.0.0

[Daltron]

[REQUIRED] Environment info

firebase-tools: 8.11.0

Platform: macOS

[REQUIRED] Test case

In my firebase.json, I have the following:

"emulators": {
    "firestore": {
     "host": "0.0.0.0",
     "port": 8090
    },
    "functions": {
     "host": "0.0.0.0",
     "port": 5001
    }
   },

[REQUIRED] Steps to reproduce

I then try running the emulators:

npx firebase emulators:start --only functions,firestore

I eventually get the error message Error: HTTP Error: 503, io exception and then the emulators stop automatically.

If I remove the host value entirely from the firebase.json or if I add my computer's IP address like 192.168.1.58, it will work perfectly, just not with 0.0.0.0

[REQUIRED] Expected behavior

The emulators run without errors

[REQUIRED] Actual behavior

The emulators run for about 10 seconds then stop with the following error: Error: HTTP Error: 503, io exception

[debug] [2020-09-22T21:14:22.445Z] [worker-~diagnostic~-b2802790-9d42-42d7-ab88-a952d1dba926]: IDLE {"metadata":{"emulator":{"name":"functions"},"message":"[worker-~diagnostic~-b2802790-9d42-42d7-ab88-a952d1dba926]: IDLE"}}
[debug] [2020-09-22T21:14:22.485Z] Sep 22, 2020 4:14:22 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
 {"metadata":{"emulator":{"name":"firestore"},"message":"Sep 22, 2020 4:14:22 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[debug] [2020-09-22T21:14:52.620Z] Sep 22, 2020 4:14:52 PM io.gapi.emulators.netty.HttpHandler$1 onError
INFO: Exception when handling request: UNAVAILABLE: io exception
 {"metadata":{"emulator":{"name":"firestore"},"message":"Sep 22, 2020 4:14:52 PM io.gapi.emulators.netty.HttpHandler$1 onError\nINFO: Exception when handling request: UNAVAILABLE: io exception\n"}}
[debug] [2020-09-22T21:14:52.627Z] <<< HTTP RESPONSE 503 {"content-type":"application/json","content-length":"70"}
[debug] [2020-09-22T21:14:52.627Z] <<< HTTP RESPONSE BODY {"error":{"code":503,"message":"io exception","status":"UNAVAILABLE"}}
[warn] ⚠  Error adding trigger: FirebaseError: HTTP Error: 503, io exception {"metadata":{"emulator":{"name":"functions"},"message":"Error adding trigger: FirebaseError: HTTP Error: 503, io exception"}}
[info] i  emulators: Shutting down emulators. {"metadata":{"emulator":{"name":"hub"},"message":"Shutting down emulators."}}
[info] i  hub: Stopping emulator hub {"metadata":{"emulator":{"name":"hub"},"message":"Stopping emulator hub"}}
[info] i  ui: Stopping Emulator UI {"metadata":{"emulator":{"name":"ui"},"message":"Stopping Emulator UI"}}
[info] i  logging: Stopping Logging Emulator {"metadata":{"emulator":{"name":"logging"},"message":"Stopping Logging Emulator"}}
[info] i  functions: Stopping Functions Emulator {"metadata":{"emulator":{"name":"functions"},"message":"Stopping Functions Emulator"}}
[info] i  firestore: Stopping Firestore Emulator {"metadata":{"emulator":{"name":"firestore"},"message":"Stopping Firestore Emulator"}}
[warn] ⚠  Emulator UI has exited upon receiving signal: SIGINT 
[debug] [2020-09-22T21:14:52.634Z] *** shutting down gRPC server since JVM is shutting down
 {"metadata":{"emulator":{"name":"firestore"},"message":"*** shutting down gRPC server since JVM is shutting down\n"}}
[debug] [2020-09-22T21:14:52.638Z] *** server shut down
 {"metadata":{"emulator":{"name":"firestore"},"message":"*** server shut down\n"}}
[debug] [2020-09-22T21:14:52.640Z] [worker-~diagnostic~-b2802790-9d42-42d7-ab88-a952d1dba926]: exited {"metadata":{"emulator":{"name":"functions"},"message":"[worker-~diagnostic~-b2802790-9d42-42d7-ab88-a952d1dba926]: exited"}}
[debug] [2020-09-22T21:14:52.640Z] [worker-~diagnostic~-b2802790-9d42-42d7-ab88-a952d1dba926]: FINISHED {"metadata":{"emulator":{"name":"functions"},"message":"[worker-~diagnostic~-b2802790-9d42-42d7-ab88-a952d1dba926]: FINISHED"}}
[error] 
[error] Error: HTTP Error: 503, io exception
[debug] [2020-09-22T21:14:52.984Z] Error Context: {
  "body": {
    "error": {
      "code": 503,
      "message": "io exception",
      "status": "UNAVAILABLE"
    }
  },
  "response": {
    "statusCode": 503,
    "body": "{\"error\":{\"code\":503,\"message\":\"io exception\",\"status\":\"UNAVAILABLE\"}}",
    "headers": {
      "content-type": "application/json",
      "content-length": "70"
    },
    "request": {
      "uri": {
        "protocol": "http:",
        "slashes": true,
        "auth": null,
        "host": "0.0.0.0:8090",
        "port": "8090",
        "hostname": "0.0.0.0",
        "hash": null,
        "search": null,
        "query": null,
        "pathname": "/emulator/v1/projects/myproject/triggers/myproject/join_notifications-replies-triggers-sendNotification",
        "path": "/emulator/v1/projects/myproject/triggers/myproject/join_notifications-replies-triggers-sendNotification",
        "href": "http://0.0.0.0:8090/emulator/v1/projects/myproject/triggers/myproject-join_notifications-replies-triggers-sendNotification"
      },
      "method": "PUT"
    }
  }
}

[natebot13]

I think I have a related issue, only for the RTDB emulator. I have functions, firestore and database emulators enabled, each with host set to "0.0.0.0". Running firebase emulators:start would initialize and load everything, then shut down with "unexpected error". I checked the database-debug.log and it shows:
[FirebaseWorkerPool-1-2] WARN com.firebase.core.namespace.NamespaceActorSystem - Request for ns: project-{omitted} has incorrect domain. Current domain: 0.0.0.0, expected domain: firebaseio-staging.com.

When I remove the "host" line for the database in firebase.json, then everything works fine over localhost, but I need to be able to connect my physical device on the same network to the emulators, which doesn't work when using localhost.

firebase --version: 8.13.1

[samtstern]

Hmmm I am not able to reproduce this on my mac, I can get all the emulators to start on 0.0.0.0:

┌───────────────────────────────────────────────────────────────────────┐
│ ✔  All emulators ready! View status and logs at http://localhost:4000 │
└───────────────────────────────────────────────────────────────────────┘

┌───────────┬──────────────┬─────────────────────────────────┐
│ Emulator  │ Host:Port    │ View in Emulator UI             │
├───────────┼──────────────┼─────────────────────────────────┤
│ Functions │ 0.0.0.0:9003 │ http://localhost:4000/functions │
├───────────┼──────────────┼─────────────────────────────────┤
│ Firestore │ 0.0.0.0:9002 │ http://localhost:4000/firestore │
├───────────┼──────────────┼─────────────────────────────────┤
│ Database  │ 0.0.0.0:9001 │ http://localhost:4000/database  │
└───────────┴──────────────┴─────────────────────────────────┘

Clicking around the Emulator UI things seem to be working normally, so I'm not sure how to reproduce the issues you're having.

[samtstern]

Got another report of this in #2742 so clearly something is wrong ... still don't know what.

[StorminGorman]

I think a big clue for me is

WARN com.firebase.core.namespace.NamespaceActorSystem - Request for ns: checkit-portico has incorrect domain. Current domain: 0.0.0.0, expected domain: firebaseio-staging.com\n"}}

In the log ... why would it expect firebaseio-staging.com when the host is configured as 0.0.0.0 ?

[natebot13]

This is still happening with the latest tools release from today. Here are some more details of my setup:
npm --version: 6.14.8
node --version: v12.19.0
firebase --version: 8.14.1

I'm running this on Ubuntu, but through WSL2, which might be part of the problem, since there are some networking caveats. However, every other emulator works with host set to "0.0.0.0", including the new auth emulator.

[blechatellier]

I'm running into the same issue with the database when running in Docker and trying to map to 0.0.0.0 - all the other emulators are working as expected. @samtstern any ideas what's going on? Do you need help reproducing?

[markgarrigan]

Same issue for me. Functions and hosting emulators work when mapping to 0.0.0.0 host. But database.debug.log says...

13:55:17.788 [NamespaceSystem-akka.actor.default-dispatcher-5] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
13:55:18.181 [main] INFO com.firebase.server.forge.App$ - Listening at 0.0.0.0:9000
13:55:21.727 [FirebaseWorkerPool-1-3] WARN com.firebase.core.namespace.NamespaceActorSystem - Request for ns: REDACTED has incorrect domain. Current domain: 0.0.0.0, expected domain: firebaseio-staging.com
13:55:21.987 [Thread-1] INFO com.firebase.server.forge.App$ - Attempting graceful shutdown.
13:55:22.016 [Thread-1] INFO com.firebase.server.forge.App$ - Graceful shutdown complete.

[markgarrigan]

My Firebase CLI runs in a docker container. Here's the Dockerfile that creates the Firebase CLI image.

FROM node:latest
WORKDIR /usr/local/bin
RUN apt update \
  && apt-get -y install default-jre \
  && /bin/bash -c "npm i -g npm firebase-tools npm-check-updates"

[masikmos]

i think we can use nginx or haproxy in docker container

[SeanZom]

I had a similar issue (got HTTP Error: 503, io exception) when starting the emulators to run unit tests from Github actions script. The same script worked before but kept failing recently.
I removed all host:0.0.0.0 from firebase.json then it's back to work again.

[warting]

same thing for me as for SeanZom, removing host 0.0.0.0 made it work on github actions. Is there any trick to set the host from command line when starting emulators?

[samtstern]

@warting right now there's no way to pass --host from the command line but you can use --config=firebase.json to pass an entirely different firebase.json file from the command line, which is a workaround for now.

[warting]

@samtstern Thanks! That will help!

[phcoliveira]

I have the same or at least similar problem, but perhaps the bug is in the Firebase Web SDK 9.
The host 0.0.0.0 should be used to simply accept requests from the local network. However, it is passed when the first websocket is connected. From then, the Web SDK uses this address 0.0.0.0:${your-port} instead of the URL informed when the Firebase App is initialized (problem 1).

By not specifying a host, Firebase Tools does the right thing and returns the server's IP in the local network, so the Web SDK uses the right address. However, because there is no host, the server does not accept a request outside of localhost (problem 2).

As mentioned above, my solution was to use my local IP both when initializing the app with Web SDK and as the host option to start the emulators.

IMO, the Web SDK should continue to use the URL passed when it is initialized.

If that is not technically possible, then Firebase Tools should continue to provide the right IP when no host is specified. But it should also provide an option like acceptAllHosts: boolean, to make it possible for another device in the local network to reach it.

[idudinov]

for someone like me who suddenly started to experience such problem: in my case today it was just because of network configuration – my laptop connected to a WiFi extender which I assume somehow changes network topology or something like that.

I just disconnected from WiFi, started emulators using 0.0.0.0 as host for firestore (that took a bit longer initialize than usual), and then re-connected to WiFi again.

0.0.0.0 interface is required for devices in my local network to be able to connect to emulators.

Hopefully this information helps.

[yuchenshi]

Hi all, thanks for your continued input to this dated thread. I'm a bit late to the party, but let me try to distill some of the issues mentioned here.

1: We've fixed an issue that prevented Emulator UI from accessing emulators listening on 0.0.0.0 a while ago: firebase/firebase-tools-ui#286 (please see the issue description for a detailed description of why 0.0.0.0 is particularly problematic. tl;dr this only happens on some platform / browsers but not others, which is why it is hard to repro)

2: We believe the RTDB has incorrect domain issue has since been fixed, but there there has been an ongoing (related) issue with accessing RTDB emulators from 0.0.0.0, and a workaround is provided at #2870 (comment). (I've reopened the issue for visibility)

3: We haven't yet reproduced OP's HTTP Error: 503 issue, and we'd appreciate any help with a more self-contained repro. (I think this may also be platform/OS specific, but I may be wrong here.)

If you run into any other issue not listed in 1-3, please open a new issue instead and we'll help you over there. Let's keep this issue focused on the 503 issue as described in the OP.

[google-oss-bot]

Hey @Daltron. We need more information to resolve this issue but there hasn't been an update in 7 weekdays. I'm marking the issue as stale and if there are no new updates in the next 3 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[google-oss-bot]

Hey @Daltron. We need more information to resolve this issue but there hasn't been an update in 7 weekdays. I'm marking the issue as stale and if there are no new updates in the next 3 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[google-oss-bot]

Since there haven't been any recent updates here, I am going to close this issue.

@Daltron if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.