Make container image building part of Copilot

[Fodoj]

I've been thinking a lot about how Copilot handles building images. I think it could be improved a lot by moving the actual image building and publishing to be a part of Copilot source code.

Right now

Image-specific options are passed to a build spec Golang template. The result is basically a big shell script, that depends both on Docker and AWS CLI, as well as lots of commands to fetch values from Copilot manifest with jq and pass it to other tools. Problems I see there are:

1. Hard to update. If Copilot changes this shell script, users of Copilot need to re-generate the buildspec and then re-embed custom build steps (like running tests) in there.
2. Dependency on there tools. In terms of build pipeline, Copilot is used only to bind together multiple other tools, like AWS CLI and Docker. This means that both build environment and local environment require those tools to be installed. As a user, I can't just take Copilot and use it end to end - I also need to install aforementioned tools.
3. Related to the previous point - if I would not use pipeline feature of Copilot and decide to use GitHub actions, Jenkins or something else, I need to take care of installing all the extra tooling to make it work, Copilot alone won't do. I need whole Docker daemon (and Docker in Docker setup if I want to run by builds in container environment) to be able to use it - imagine if I have my builds running on a secured Kubernetes cluster, without privileged pods or pods with host access (or even hosts with cri-o instead of Docker daemon as a runtime).

I was thinking how to implement this #1943 or this #1583 and the only way right now is to update that shell script template.

Proposal

Building and publishing container images is not tool-specific, all of there tools are OCI compliant and as long as Copilot is also OCI compliant, it can as well just take care of building and publishing images itself. There are many ways to do it and multiple libraries that can handle it. One of the most feature-complete ones is Buildah - it could be embedded into Copilot (see https://github.com/containers/buildah/blob/master/docs/tutorials/04-include-in-your-build-tool.md as an example, or Podman that basically bundles Buildah to do image management).

By moving image building and pushing to Copilot codebase, this part becomes programmable, much more extensible (there are limits to shell script, and after reaching this limit the script becomes write-only :-) ) and last but not least - testable. It will also enable implementing features like #1081 - right now it would require once again extending the shell script template.

Downside of course is much larger Copilot binary, as it needs more dependencies bundled. But I would argue it's not a big deal.

[efekarakus]

Hi @Fodoj

This is a really neat idea! I agree completely about the benefits, we ran into several situations where folks didn't have the docker daemon running and the getting started experience was interrupted because of it.

For benefit#1, I see you already commented on #1583 :) which would free folks from not having to re-generate the buildspec and then copying the custom build steps on newer versions of the CLI. So I think we can still support that at least without having to integrate with buildah.

For using a different tool for building and enabling benefit#2,3, we would need to get it security reviewed first with our AppSec engineers to make sure we're doing the right thing for customers.

So I expect the latter improvement to take some time, but I think we can abstract away the buildspec earlier than that.