docker-compose reaching end of support, migrate to docker compose

[crstamps2]

According to this https://docs.docker.com/compose/migrate/ , Compose V1 is reaching end of support. In fact, a warning at the bottom of this page indicates they had even stopped security updates back in 2021. The recommendation is to migrate to using the built in docker version of compose (aka "v2"). Generally the change is to just go from docker-compose based commands to docker compose (no hyphen) based commands.

[toote]

At this time the plugin supports both the old command (docker-compose, aka v1 , the default) as well as the new way through a docker plugin (docker compose, aka v2). The latter can be turned on through the cli-version option but it is important to make users aware of that so I have pinned the issue for everyone to know about it.

Eventually the plugin will need to default to v2 (which will be a major change). At that time it may be a good idea to start priting a warning message when v1 is used. I am not 100% sure we will ever completely remove the compatibilty layer to support those that can not update but maybe decide to stop actively supporting it to simplify the plugin code, if that is the way we decide to go on we should create a documentation page to document the necessary changes and workarounds for those that still need that.

Any comments, opinions and suggestions will be more than welcome :)

[Frug]

I help maintain a large number of pipelines and recently updated our build workers. Adding cli-version to every pipeline step that makes use of this plugin is quite a lot of boilerplate to add. Alternatively I can install v1, or a shim like compose switch, but I'm sure new users will wonder why this plugin can't find their docker compose install.

Personally I think it's time to switch the default to using v2, as this will affect more and more new users in favor of people who aren't keeping things up to date.

The only issue I've come across making the switch to v2 is that the output from buildkit is messy on CI pipelines unless I specify BUILDKIT_PROGRESS: "plain" as an env var. It might be nice to have that as a default, too.

[crstamps2]

Let me know if you would prefer I open another issue about this, but it sort of has to do with this issue.

I can't get v2 docker compose working. I updated all of our buildkite docker compose plugins to the latest version 4.12.0 and added cli-version option to 2. But I get this error on a build:

:docker: Running plugin docker-compose command hook | 0s
:docker: Found a pre-built image for test | 0s
:docker: Creating docker-compose override file for prebuilt services | 0s
:docker: Pulling services test | 6s
  | $ docker compose -f docker/docker-compose.yml -p buildkite0188584a93f04c5d98170382c6991212 -f docker-compose.buildkite-160126-override.yml pull test
  | unknown shorthand flag: 'f' in -f
  | See 'docker --help'.
  |  
  | Usage:  docker [OPTIONS] COMMAND
  |  
  | A self-sufficient runtime for containers

I tried forking this repo to see if I could get it working. Seems like it is some kind of whitespace issue, the whitespace between docker and compose. Obviously, this is the new preferred way. I tried forking the repo and trying some minor tweaks to get around it, but haven't been successful yet. Any thoughts? Found out this was all a part of my stack not having the updated version of docker.

[toote]

@crstamps2 I have used version 2 of docker compose and the plugin works just fine with it. Sounds like your scenario is a bug or issue in your setup, can you create a different issue to discuss it further?

[toote]

I help maintain a large number of pipelines and recently updated our build workers. Adding cli-version to every pipeline step that makes use of this plugin is quite a lot of boilerplate to add. Alternatively I can install v1, or a shim like compose switch, but I'm sure new users will wonder why this plugin can't find their docker compose install.

Another alternative would be to define the appropriate variable BUILDKITE_PLUGIN_DOCKER_COMPOSE_VERSION=2 as an agent environment variable (or through other hooks/plugins you are using) so that it applies across all jobs executed by that agent irrespective of the configuration. That way you don't have to touch any pipeline and you can concentrate the changes to a single place: the instances running the agents and docker :)

Personally I think it's time to switch the default to using v2, as this will affect more and more new users in favor of people who aren't keeping things up to date.

I wouldn't be so fast to critique those not keeping up with the latest and greatest. As someone who has been working on several areas of the multi-faceted beast that can be a professional development enterprise, sometimes it is not the fault of the person using the plugin that they are unable to update the underlying infrastructure. Other times it is about coordination with several other teams that need to make changes at the same time or depend on things that you want to change. And yet other times it is just a matter of the urgent not leaving room for the important, like making money to keep a company afloat instead of keeping with the new shiny thing on something that still works.

I'm not saying that was your intention, just that making such big changes on code that lots of people use can cause more harm than good as well. You don't want yet another thing that is out of date and has no support :(

The discussion on this thread is my attempt to try to hear as many voices as possible. That is why I pinned this issue to the repository, and want to give people a lot of time to find it and add their opinion. Honestly, my current idea is not to make the change before the end of 2023. But nothing is set in stone.

The only issue I've come across making the switch to v2 is that the output from buildkit is messy on CI pipelines unless I specify BUILDKIT_PROGRESS: "plain" as an env var. It might be nice to have that as a default, too.

Good idea!

[deanpcmad]

Any news on this?

[pzeballos]

Hey folks, sorry for the delay 🙏🏻. We are planning on setting v2 as default by the end of Q4. We'll update this thread when there's more details

[pzeballos]

We did the major release (v5.0.0) with this change and also other backward incompatible changes. Thanks everyone!