How to start prefect server from a running docker container? #3449

newskooler · 2020-10-07T13:07:03Z

newskooler
Oct 7, 2020

I am trying to start the prefect server from a running docker container, but I get an error which seems related to docker-compose, but I could not solve it.

I am starting my docker container like so:

docker run -t -d -p 4200:4200 -p 8080:8080 --name my_cont registry.location.com/my_cont

Then when I run docker exec -it my_cont prefect server start, I get the following error:

Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1252, in request
  File "http/client.py", line 1298, in _send_request
  File "http/client.py", line 1247, in endheaders
  File "http/client.py", line 1026, in _send_output
  File "http/client.py", line 966, in send
  File "docker/transport/unixconn.py", line 43, in connect
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "requests/adapters.py", line 449, in send
  File "urllib3/connectionpool.py", line 727, in urlopen
  File "urllib3/util/retry.py", line 403, in increment
  File "urllib3/packages/six.py", line 734, in reraise
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1252, in request
  File "http/client.py", line 1298, in _send_request
  File "http/client.py", line 1247, in endheaders
  File "http/client.py", line 1026, in _send_output
  File "http/client.py", line 966, in send
  File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docker/api/client.py", line 205, in _retrieve_server_version
  File "docker/api/daemon.py", line 181, in version
  File "docker/utils/decorators.py", line 46, in inner
  File "docker/api/client.py", line 228, in _get
  File "requests/sessions.py", line 543, in get
  File "requests/sessions.py", line 530, in request
  File "requests/sessions.py", line 643, in send
  File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 67, in main
  File "compose/cli/main.py", line 123, in perform_command
  File "compose/cli/command.py", line 69, in project_from_options
  File "compose/cli/command.py", line 132, in get_project
  File "compose/cli/docker_client.py", line 43, in get_client
  File "compose/cli/docker_client.py", line 170, in docker_client
  File "docker/api/client.py", line 188, in __init__
  File "docker/api/client.py", line 213, in _retrieve_server_version
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
[18] Failed to execute script docker-compose
Exception caught; killing services (press ctrl-C to force)
Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1252, in request
  File "http/client.py", line 1298, in _send_request
  File "http/client.py", line 1247, in endheaders
  File "http/client.py", line 1026, in _send_output
  File "http/client.py", line 966, in send
  File "docker/transport/unixconn.py", line 43, in connect
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "requests/adapters.py", line 449, in send
  File "urllib3/connectionpool.py", line 727, in urlopen
  File "urllib3/util/retry.py", line 403, in increment
  File "urllib3/packages/six.py", line 734, in reraise
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1252, in request
  File "http/client.py", line 1298, in _send_request
  File "http/client.py", line 1247, in endheaders
  File "http/client.py", line 1026, in _send_output
  File "http/client.py", line 966, in send
  File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docker/api/client.py", line 205, in _retrieve_server_version
  File "docker/api/daemon.py", line 181, in version
  File "docker/utils/decorators.py", line 46, in inner
  File "docker/api/client.py", line 228, in _get
  File "requests/sessions.py", line 543, in get
  File "requests/sessions.py", line 530, in request
  File "requests/sessions.py", line 643, in send
  File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 67, in main
  File "compose/cli/main.py", line 123, in perform_command
  File "compose/cli/command.py", line 69, in project_from_options
  File "compose/cli/command.py", line 132, in get_project
  File "compose/cli/docker_client.py", line 43, in get_client
  File "compose/cli/docker_client.py", line 170, in docker_client
  File "docker/api/client.py", line 188, in __init__
  File "docker/api/client.py", line 213, in _retrieve_server_version
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
[22] Failed to execute script docker-compose
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/prefect/cli/server.py", line 332, in start
    ["docker-compose", "pull"], cwd=compose_dir_path, env=env
  File "/usr/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker-compose', 'pull']' returned non-zero exit status 255.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/prefect/cli/server.py", line 366, in start
    ["docker-compose", "down"], cwd=compose_dir_path, env=env
  File "/usr/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker-compose', 'down']' returned non-zero exit status 255.

This is how i install docker compose in the Dockerfile:

FROM ubuntu:18.04

# Upgrade installed packages
RUN apt-get update && apt-get upgrade -y && apt-get clean

# Install other libraries.
RUN apt -y install curl unzip build-essential wget

# Set the timezone to UTC
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update -y && apt-get install -y tzdata

# Installing Docker Compose
RUN curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
RUN chmod +x /usr/local/bin/docker-compose

these are the prefect diagnostics from inside the container:

{
  "config_overrides": {
    "server": {
      "ui": {
        "apollo_url": true
      }
    }
  },
  "env_vars": [],
  "system_information": {
    "platform": "Linux-5.4.0-45-generic-x86_64-with-Ubuntu-18.04-bionic",
    "prefect_backend": "server",
    "prefect_version": "0.13.9",
    "python_version": "3.7.5"
  }
}

Any idea how I might fix this?

Answered by joshmeek

Oct 7, 2020

@snenkov I haven't tried this before (and I wouldn't necessarily recommend attempting to run the entire server off of docker compose in a single container) but my guess is for your use case you might need to mount a Docker daemon in order to use Docker-based commands from inside the container. Again, this isn't a pattern we support and Prefect Server is broken up into separate services, served over a compose network, for reasons like this.

View full answer

joshmeek · 2020-10-07T13:28:15Z

joshmeek
Oct 7, 2020

@snenkov I haven't tried this before (and I wouldn't necessarily recommend attempting to run the entire server off of docker compose in a single container) but my guess is for your use case you might need to mount a Docker daemon in order to use Docker-based commands from inside the container. Again, this isn't a pattern we support and Prefect Server is broken up into separate services, served over a compose network, for reasons like this.

9 replies

newskooler Oct 10, 2020
Author

Thanks @NoName115. I tried to do just that (run only the agent in a container), but I get an error (pasted below). I guess the error is related to the localhost / docker network situation you are referring to.
Can you please send some guidance on where I can read about fixing this issue.

Here is what I do now and the error I get:

I start my container like so docker run -t -d --name cont_name registry.gitlab.com/my_repo/images:latest
I run docker exec -it cont_name prefect backend server and get Backend switched to server (so it's successful)
I run docker exec -it cont_name prefect agent start and get the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/dist-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.7/dist-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.7/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 172, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff223cfccd0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/dist-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff223cfccd0>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/prefect/cli/agent.py", line 263, in start
    no_cloud_logs=no_cloud_logs,
  File "/usr/local/lib/python3.7/dist-packages/prefect/agent/local/agent.py", line 78, in __init__
    no_cloud_logs=no_cloud_logs,
  File "/usr/local/lib/python3.7/dist-packages/prefect/agent/agent.py", line 153, in __init__
    self.client = Client(api_server=config.cloud.api, api_token=token)
  File "/usr/local/lib/python3.7/dist-packages/prefect/client/client.py", line 124, in __init__
    tenant_info = self.graphql({"query": {"tenant": {"id"}}})
  File "/usr/local/lib/python3.7/dist-packages/prefect/client/client.py", line 281, in graphql
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/dist-packages/prefect/client/client.py", line 237, in post
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/dist-packages/prefect/client/client.py", line 401, in _request
    session=session, method=method, url=url, params=params, headers=headers
  File "/usr/local/lib/python3.7/dist-packages/prefect/client/client.py", line 319, in _send_request
    response = session.post(url, headers=headers, json=params, timeout=30)
  File "/usr/local/lib/python3.7/dist-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff223cfccd0>: Failed to establish a new connection: [Errno 111] Connection refused'))

: (

NoName115 Oct 10, 2020

Yes, that's exactly to what I was refering to. I recommend to you read these sources https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach & https://docs.docker.com/network/ to get a overview how docker-network works.

Basic facts you should know, localhost address in docker has nothing to do with your actual localhost adress at your host machine. Every docker-network has it's own DNS server basically, by which the container's addresses/IPs are translated and the localhost refers to the container itself.

Let's put it into example of prefect, running prefect server start at your host-machine will create 5 containers (5 or 6, not sure) where your postgres, apollo, graphql, UI, and other stuff are running

CONTAINER ID        IMAGE                           COMMAND                  CREATED              STATUS              PORTS                            NAMES
e9aa08e2aa25        prefecthq/ui:core-0.13.10       "/docker-entrypoint.…"   About a minute ago   Up About a minute   80/tcp, 0.0.0.0:8080->8080/tcp   tmp_ui_1
b79d48a3467a        prefecthq/apollo:core-0.13.10   "bash -c './post-sta…"   About a minute ago   Up About a minute   0.0.0.0:4200->4200/tcp           tmp_apollo_1
86d6f1f65db3        prefecthq/server:core-0.13.10   "python src/prefect_…"   About a minute ago   Up About a minute                                    tmp_towel_1
0d21e942db35        prefecthq/server:core-0.13.10   "bash -c 'prefect-se…"   About a minute ago   Up About a minute   0.0.0.0:4201->4201/tcp           tmp_graphql_1
26dc201a3d01        hasura/graphql-engine:v1.3.0    "graphql-engine serve"   About a minute ago   Up About a minute   0.0.0.0:3000->3000/tcp           tmp_hasura_1
1e15dbf5a956        postgres:11                     "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:5432->5432/tcp           tmp_postgres_1

Command docker ps above. From this you can see that your apollo container is called tmp_apollo_1 and the internal port of the container is 4200 (from 0.0.0.0:4200) and these containers are running in dedicated docker-network called prefect-server, you can see that in docker-compose file from prefect https://github.com/PrefectHQ/prefect/blob/master/src/prefect/cli/docker-compose.yml at the end or by running docker network ls, also try to run docker network inspect prefect-server to see details about this docker-network, you will also find there container names and also their inner IPs inside this prefect-server docker-network (sometimes useful when you're running different services and you need to internally connect to them).

But back to your container with agent, from your log you can see the agent is trying to connect to localhost:4200 address, which will not work in docker as localhost refers to the container itself, but running prefect agent start --api http://tmp_apollo_1:4200/graphql should work, but you have to run your container in prefect-server network! so just use docker run --network prefect-server ....
This should work as your container will run in internal docker-network and by refering to the apollo-container name tmp_apollo_1 the internal docker DNS server will translate this hostaddress into corret IP.

I hope it's helpfull answer for you :)

newskooler Oct 10, 2020
Author

That’s exactly the explanation I was (unknowingly) looking for. Thanks a lot! I will dig into it and may come back if I have another setback with this. I appreciate the help!

newskooler Oct 12, 2020
Author

Hi @NoName115 I read what you suggested and it works well.
Now I ran to a new (or related issue), which since you have performed the above, maybe you can help on it as well.

When I run prefect agent start inside a docker image, now I face the issue where the flows are registered locally on the server (not on the docker image), so when I try to execute a flow, the agent naturally does not find it.

In such a scenario, what is the best practice?
I guess it's not to add the flows to the image used to run the agent, since this will involve rebuilding the image each time a flow is added and/or updated.

What would you suggest is a good path to choose?

NoName115 Oct 12, 2020

In my case I'm registering flows inside the docker where my agent is running and I mounted docker volume to my physical host-machine, so they share the same disk-space which mean that even if you rebuild your image, your flows are saved at your physical disk, so they will be automatically loaded by agent.

newskooler · 2020-10-12T15:27:17Z

newskooler
Oct 12, 2020
Author

So I guess I do a similar thing, but the opposite way. I have NFS and mount it to the docker image. I register the flows from the server to the NFS and then Run the flow via the agent (which is running on docker), but the I get the following errr:
Failed to load and execute Flow's environment: ValueError('unsupported pickle protocol: 5')
Any idea why?

this is how I mount it when I run the docker image:

--mount type=bind,source="/nfs/location/prefect/flows",target="/nfs/location/prefect/flows"

1 reply

newskooler Oct 12, 2020
Author

I found the erro. I was not using python 3.8. Now it's fixed. Thanks! : )

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to start prefect server from a running docker container? #3449

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 10 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to start prefect server from a running docker container? #3449

newskooler Oct 7, 2020

Replies: 2 comments · 10 replies

joshmeek Oct 7, 2020

newskooler Oct 10, 2020 Author

NoName115 Oct 10, 2020

newskooler Oct 10, 2020 Author

newskooler Oct 12, 2020 Author

NoName115 Oct 12, 2020

newskooler Oct 12, 2020 Author

newskooler Oct 12, 2020 Author

newskooler
Oct 7, 2020

Replies: 2 comments 10 replies

joshmeek
Oct 7, 2020

newskooler Oct 10, 2020
Author

newskooler Oct 10, 2020
Author

newskooler Oct 12, 2020
Author

newskooler
Oct 12, 2020
Author

newskooler Oct 12, 2020
Author