Excluding files and folders from a result container

[xSAVIKx]

Hey! I've been using builder and run images and have a task of building a Cloud Functions and Cloud Run compatible containers.

So I decided to go with buildpacks and have a Python Functions Framework app. Now, while this is a real world and not some example, I do have tons of different folders/files in the repo that I don't want to be included. E.g. the .git and .venv or the tests folder.

I was going over the buildpacks docs and looks like I should be using the exclude (or include) configs in my project.toml, but I struggle to make it work.

Here's an example project.toml I'm trying to use:

[project]
id = "com.example.id"
name = "App Name"
authors = ["some authors"]
version = "1.2.3"

[io.buildpacks]
builder = "gcr.io/buildpacks/google-22/builder:latest"
include = [
    "dist/",
    "app_package/",
    "main.py",
    "project.toml",
    "pyproject.toml",
    "poetry.lock",
    "requirements.txt",
    "poetry.toml"
]

[[build.env]]
name = "GOOGLE_FUNCTION_TARGET"
value = "handle"

[[build.env]]
name = "GOOGLE_FUNCTION_SIGNATURE_TYPE"
value = "http"

[[build.env]]
name = "GOOGLE_FUNCTION_SOURCE"
value = "main.py"

[[build.env]]
name = "GOOGLE_RUNTIME"
value = "python"

[[build.env]]
name = "GOOGLE_RUNTIME_VERSION"
value = "3.11.3"

and here's how the file tree looks like:

├── .git/
├── .mypy_cache/
├── .pytest_cache/ 
├── app_package/
├── call-service.http
├── deploy-personal-service.sh
├── dist/
├── main.py
├── mypy.ini
├── poetry.lock
├── poetry.toml
├── __pycache__
├── pyproject.toml
├── README.md
├── _reports
├── requirements.txt
├── tests/
└── update-container.sh

And then I run my pack as:

pack build gcr.io/<project>/app-name --descriptor project.toml --builder gcr.io/buildpacks/google-22/builder:latest --run-image gcr.io/buildpacks/google-22/run:latest --clear-cache --tag gcr.io/<project>/app-name:latest --tag gcr.io/<project>/app-name:<hash> --tag gcr.io/<project>/app-name:v1.2.3 --publish

[kennethye1]

Here's an example of the project.toml https://buildpacks.io/docs/app-developer-guide/using-project-descriptor/#example. I don't think you can add builder= in the project.toml file.

[xSAVIKx]

OK, but I guess builder is not an issue. Nothing complains here and I also specify builder in the cmd.

I'm worried about exclude doing nothing.

[xSAVIKx]

Also, looks like the spec allows it: https://github.com/buildpacks/spec/blob/main/extensions/project-descriptor.md#iobuildpacksbuilder-optional

[kennethye1]

FYI I was able to reproduce this. Renaming io.buildpacks to something like build resolved this issue (I didnt test the build env variables, only the exclude).

[xSAVIKx]

So you're saying having [build] over [io.buildpacks] solves it for you and the content is actually excluded? I believe I tried this couple of times, but will try once again

[xSAVIKx]

OK, so I have built an example repository where I tried both approaches with [io.buildpacks] and [build]. You can check it out here: https://github.com/xSAVIKx/python-functions-cloud-run-buildpacks

Unfortunately, none works as expected. The [build] fails to compile at all, see https://github.com/GoogleCloudPlatform/buildpacks/issues/302.

The [io.buildpacks] compiles, but the resulting build workspace still has tons of files that were not included:

[kennethye1]

It works for me. I only edited the project.toml file.

1. Running git diff

@@ -4,9 +4,10 @@ name = "Cloud Run Buildpacks Example"
 authors = ["Yurii Serhiichuk <[email protected]>"]
 version = "1.0.0"
 
-[io.buildpacks]
+[build]
+
 include = [
-    "python_functions_cloud_run_buildpacks/",
+    "python_functions_cloud_run_buildpacks",
     "main.py",
     "project.toml",
     "pyproject.toml",

2. Run pack build test --descriptor project.toml --builder gcr.io/buildpacks/google-22/builder:latest --run-image gcr.io/buildpacks/google-22/run:latest --clear-cache
3. Then docker run -it --entrypoint=/bin/bash test and ls after shows

/workspace$ ls
main.py  poetry.lock  poetry.toml  project.toml  __pycache__  pyproject.toml  python_functions_cloud_run_buildpacks

[kennethye1]

Anyways this is probably more of a question for the CNB team.

[xSAVIKx]

Can you maybe provide the output of ls -la?

And you're doing this from my example repo, right?

[jama22]

@xSAVIKx I managed to get it working as well. Looks like you were mixing up syntaxes from spec v0.1 and 0.2. Had to modify your [project] line to [_] in your project.toml as per https://github.com/buildpacks/spec/blob/main/extensions/project-descriptor.md#top-level-tables

@kennethye1 's fix would also work, but instead of specifying the schema-version = "0.2" you would convert your enteries to be compatible with 0.1 e.g. [io.buildpacks]' should be [build]`

[_]
schema-version = "0.2"
id = "com.github.xsavikx.python_functions_cloud_run_buildpacks"
name = "Cloud Run Buildpacks Example"
authors = ["Yurii Serhiichuk <[email protected]>"]
version = "1.0.0"

[io.buildpacks]
include = [
    "python_functions_cloud_run_buildpacks/",
    "main.py",
    "project.toml",
    "pyproject.toml",
    "poetry.lock",
    "requirements.txt",
    "poetry.toml"
]
...

[xSAVIKx]

OK, @jama22, thx for pointing this out. The buildpacks docs are really confusing regarding this. I've managed to make include work, but the compilation thing is still a problem when a .venv folder is there.

So here's a build that fails: xSAVIKx/python-functions-cloud-run-buildpacks@a648829. The only thing added is the poetry install call that creates a .venv folder. I've also verified that local build also fails.

And here's a build without poetry install command that was successful: https://github.com/xSAVIKx/python-functions-cloud-run-buildpacks/actions/runs/5004179296/jobs/8966440142.

So https://github.com/xSAVIKx/python-functions-cloud-run-buildpacks/blob/master/project.toml currently has a valid example of the buildpacks project config file with the 0.2 schema and can be used as a reference.