Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker build #815

Closed
wants to merge 7 commits into from
Closed

Docker build #815

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
505d66c
changed build to low level, added api_client
ccharest93 Feb 1, 2024
9ff7f09
linting
ccharest93 Feb 3, 2024
f965c86
typing for pyre, uneeded error case, renaming output->build_msg
ccharest93 Feb 6, 2024
474478f
Merge branch 'pytorch:main' into docker-build
ccharest93 Feb 10, 2024
5e7052c
Merge branch 'pytorch:main' into docker-build
ccharest93 Feb 28, 2024
0a6c777
Merge branch 'pytorch:main' into docker-build
ccharest93 Apr 10, 2024
62c6eef
reverted some changes to lose dependency on #819
ccharest93 Apr 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 29 additions & 2 deletions torchx/workspace/docker_workspace.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
from torchx.workspace.api import walk_workspace, WorkspaceMixin

if TYPE_CHECKING:
from docker import DockerClient
from docker import APIClient, DockerClient

log: logging.Logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -71,10 +71,12 @@ def __init__(
self,
*args: object,
docker_client: Optional["DockerClient"] = None,
docker_api_client: Optional["APIClient"] = None,
**kwargs: object,
) -> None:
super().__init__(*args, **kwargs)
self.__docker_client = docker_client
self.__docker_api_client = docker_api_client

@property
def _docker_client(self) -> "DockerClient":
Expand All @@ -86,6 +88,16 @@ def _docker_client(self) -> "DockerClient":
self.__docker_client = client
return client

@property
def _docker_api_client(self) -> "APIClient":
api_client = self.__docker_api_client
if api_client is None:
import docker

api_client = docker.APIClient()
self.__docker_api_client = api_client
return api_client

def workspace_opts(self) -> runopts:
opts = runopts()
opts.add(
Expand Down Expand Up @@ -121,7 +133,7 @@ def build_workspace_and_update_role(
f"failed to pull image {role.image}, falling back to local: {e}"
)
log.info("Building workspace docker image (this may take a while)...")
image, _ = self._docker_client.images.build(
build_events = self._docker_api_client.build(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't we simply use the second value in the tuple that self._docker_client.images.build() returns? It's the same build_events Iterable, isn't it? Then we don't have to copy the logic to retrieve the image id.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it done so that we can print all the output leading to the error?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If so we don't seem to print error events

Copy link
Contributor Author

@ccharest93 ccharest93 Apr 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if i recall correctly, the first call is blocking and returns all the events at the same time, the second isnt and returns a stream which we can then print/process in real time

fileobj=context,
custom_context=True,
dockerfile=TORCHX_DOCKERFILE,
Expand All @@ -134,7 +146,22 @@ def build_workspace_and_update_role(
labels={
self.LABEL_VERSION: torchx.__version__,
},
decode=True,
)
build_msg: Dict[str, str] = {}
while True:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we use a simple for loop?

for build_msg in build_events:
    if "stream" in build_msg:
        output_str = build_msg["stream"].strip("\r\n").strip("\n")
        if output_str:
            log.info(output_str)
    if "aux" in build_msg:
        image_id = build_msg["aux"]["ID"]
        break

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build events is a stream, until stop_iteration is given we need to wait for further input on the stream. A for_loop wouldn't wait for the stream to complete and would only print events up to this point

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it how Iterables/generators work in for loop in Python? If you give a for loop an Iterable/generator (something that has __next__()) it will call __next__() until it faces StopIteration and exists the loop.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seem to be working fine, please check the latest version of #874

try:
build_msg = build_events.__next__()
if "stream" in build_msg:
output_str = build_msg["stream"].strip("\r\n").strip("\n")
if output_str != "":
log.info(output_str)
if "aux" in build_msg:
image = build_msg["aux"]
break
except StopIteration:
break

if len(old_imgs) == 0 or role.image not in old_imgs:
role.image = image.id
finally:
Expand Down