Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pyproject.toml adapter for uv writes to the filesystem within the project root, breaks down in read-only situations #3292

Open
1 task done
sanmai-NL opened this issue Nov 14, 2024 · 2 comments
Open
1 task done

pyproject.toml adapter for uv writes to the filesystem within the project root, breaks down in read-only situations #3292

sanmai-NL opened this issue Nov 14, 2024 · 2 comments
Labels
🐛 bug Something isn't working

Comments

@sanmai-NL
Copy link
Contributor

Describe the bug

To support uv (indeed, a feature not yet production ready in the current implementation), the pyproject.toml file gets moved away and the contents at its path mutate. After uv completes, pdm restores the original pyproject.toml.

This doesn't work when the filesystem is read-only (in some place). This design also creates some noise in IDEs that watch the filesystem and thing the VCS working copy is undergoing changes.

I haven't studied the current implementation in detail, but creating a temporary directory and/or files using e.g., the tempfile module is more likely to get past this filesystem constraint.

To reproduce

024-11-14T21:50:48.973170+0100 - DEBUG - aimtools.cli - execute - [stdout] + pdm install --fail-fast --frozen-lockfile --no-editable --no-isolation --production --verbose

2024-11-14T21:50:48.973527+0100 - DEBUG - aimtools.cli - execute - [stdout] STATUS: Resolving packages from lockfile...

2024-11-14T21:50:48.973912+0100 - DEBUG - aimtools.cli - execute - [stdout] WARNING: inherit_metadata strategy is not supported by uv resolver, it will be ignored

2024-11-14T21:50:48.974240+0100 - DEBUG - aimtools.cli - execute - [stdout] Traceback (most recent call last):

2024-11-14T21:50:48.974540+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/bin/pdm", line 8, in <module>

2024-11-14T21:50:48.974899+0100 - DEBUG - aimtools.cli - execute - [stdout]     sys.exit(main())

2024-11-14T21:50:48.975196+0100 - DEBUG - aimtools.cli - execute - [stdout]              ^^^^^^

2024-11-14T21:50:48.975485+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/core.py", line 379, in main

2024-11-14T21:50:48.975756+0100 - DEBUG - aimtools.cli - execute - [stdout]     return core.main(args or sys.argv[1:])

2024-11-14T21:50:48.976070+0100 - DEBUG - aimtools.cli - execute - [stdout]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2024-11-14T21:50:48.976367+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/core.py", line 267, in main

2024-11-14T21:50:48.976664+0100 - DEBUG - aimtools.cli - execute - [stdout]     raise cast(Exception, err).with_traceback(traceback) from None

2024-11-14T21:50:48.976963+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/core.py", line 262, in main

2024-11-14T21:50:48.977263+0100 - DEBUG - aimtools.cli - execute - [stdout]     self.handle(project, options)

2024-11-14T21:50:48.977591+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/core.py", line 194, in handle

2024-11-14T21:50:48.977969+0100 - DEBUG - aimtools.cli - execute - [stdout]     command.handle(project, options)

2024-11-14T21:50:48.978298+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/cli/commands/install.py", line 103, in handle

2024-11-14T21:50:48.978603+0100 - DEBUG - aimtools.cli - execute - [stdout]     actions.do_sync(

2024-11-14T21:50:48.978947+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/cli/actions.py", line 284, in do_sync

2024-11-14T21:50:48.979219+0100 - DEBUG - aimtools.cli - execute - [stdout]     packages = list(resolve_from_lockfile(project, requirements, groups=list(selection)))

2024-11-14T21:50:48.979505+0100 - DEBUG - aimtools.cli - execute - [stdout]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2024-11-14T21:50:48.979794+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/cli/actions.py", line 218, in resolve_from_lockfile

2024-11-14T21:50:48.980090+0100 - DEBUG - aimtools.cli - execute - [stdout]     return resolver.resolve().packages

2024-11-14T21:50:48.980446+0100 - DEBUG - aimtools.cli - execute - [stdout]            ^^^^^^^^^^^^^^^^^^

2024-11-14T21:50:48.980818+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/resolver/uv.py", line 173, in resolve

2024-11-14T21:50:48.981154+0100 - DEBUG - aimtools.cli - execute - [stdout]     builder.build_pyproject_toml()

2024-11-14T21:50:48.981447+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/formats/uv.py", line 64, in build_pyproject_toml

2024-11-14T21:50:48.981775+0100 - DEBUG - aimtools.cli - execute - [stdout]     path = self._enter_path(self.project.root / "pyproject.toml")

2024-11-14T21:50:48.982066+0100 - DEBUG - aimtools.cli - execute - [stdout]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2024-11-14T21:50:48.982373+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/home/unprivileged/.local/share/pdm/venv/lib/python3.12/site-packages/pdm/formats/uv.py", line 72, in _enter_path

2024-11-14T21:50:48.982694+0100 - DEBUG - aimtools.cli - execute - [stdout]     backup = path.rename(name)

2024-11-14T21:50:48.983082+0100 - DEBUG - aimtools.cli - execute - [stdout]              ^^^^^^^^^^^^^^^^^

2024-11-14T21:50:48.983370+0100 - DEBUG - aimtools.cli - execute - [stdout]   File "/usr/lib/python3.12/pathlib.py", line 1363, in rename

2024-11-14T21:50:48.983645+0100 - DEBUG - aimtools.cli - execute - [stdout]     os.rename(self, target)

2024-11-14T21:50:48.983981+0100 - DEBUG - aimtools.cli - execute - [stdout] OSError: [Errno 16] Resource busy: '/opt/app/pyproject.toml' -> '/opt/app/pyproject.toml.w1buacou'

Expected Behavior

Don't assume renames etc. are possible and avoid filesystem mutations as much as possible.

Environment Information

2024-11-14T21:50:48.867099+0100 - DEBUG - aimtools.cli - execute - [stdout] + pdm info

2024-11-14T21:50:48.867460+0100 - DEBUG - aimtools.cli - execute - [stdout] PDM version:

2024-11-14T21:50:48.867931+0100 - DEBUG - aimtools.cli - execute - [stdout]   2.20.1

2024-11-14T21:50:48.868263+0100 - DEBUG - aimtools.cli - execute - [stdout] Python Interpreter:

2024-11-14T21:50:48.868593+0100 - DEBUG - aimtools.cli - execute - [stdout]   /opt/app/.venv/bin/python (3.12)

2024-11-14T21:50:48.868933+0100 - DEBUG - aimtools.cli - execute - [stdout] Project Root:

2024-11-14T21:50:48.869294+0100 - DEBUG - aimtools.cli - execute - [stdout]   /opt/app

2024-11-14T21:50:48.869603+0100 - DEBUG - aimtools.cli - execute - [stdout] Local Packages:

2024-11-14T21:50:48.869939+0100 - DEBUG - aimtools.cli - execute - [stdout]   

2024-11-14T21:50:48.870314+0100 - DEBUG - aimtools.cli - execute - [stdout] + pdm info --env

2024-11-14T21:50:48.870597+0100 - DEBUG - aimtools.cli - execute - [stdout] {

2024-11-14T21:50:48.870940+0100 - DEBUG - aimtools.cli - execute - [stdout]   "implementation_name": "cpython",

2024-11-14T21:50:48.871360+0100 - DEBUG - aimtools.cli - execute - [stdout]   "implementation_version": "3.12.7",

2024-11-14T21:50:48.871707+0100 - DEBUG - aimtools.cli - execute - [stdout]   "os_name": "posix",

2024-11-14T21:50:48.872006+0100 - DEBUG - aimtools.cli - execute - [stdout]   "platform_machine": "aarch64",

2024-11-14T21:50:48.872331+0100 - DEBUG - aimtools.cli - execute - [stdout]   "platform_release": "6.11.3-200.fc40.aarch64",

2024-11-14T21:50:48.872724+0100 - DEBUG - aimtools.cli - execute - [stdout]   "platform_system": "Linux",

2024-11-14T21:50:48.873115+0100 - DEBUG - aimtools.cli - execute - [stdout]   "platform_version": "#1 SMP PREEMPT_DYNAMIC Thu Oct 10 22:53:48 UTC 2024",

2024-11-14T21:50:48.873409+0100 - DEBUG - aimtools.cli - execute - [stdout]   "python_full_version": "3.12.7",

2024-11-14T21:50:48.873669+0100 - DEBUG - aimtools.cli - execute - [stdout]   "platform_python_implementation": "CPython",

2024-11-14T21:50:48.873960+0100 - DEBUG - aimtools.cli - execute - [stdout]   "python_version": "3.12",

2024-11-14T21:50:48.874211+0100 - DEBUG - aimtools.cli - execute - [stdout]   "sys_platform": "linux"

2024-11-14T21:50:48.874470+0100 - DEBUG - aimtools.cli - execute - [stdout] }

pdm -v output

No response

Additional Context

When pyproject.toml is a read-only bind mount, it's simply impossible to rename as far as I know.

Are you willing to submit a PR to fix this bug?

  • Yes, I would like to submit a PR.
@sanmai-NL sanmai-NL added the 🐛 bug Something isn't working label Nov 14, 2024
@frostming
Copy link
Collaborator

Usually, building or installing the current project needs to be done in place, otherwise, you would need to copy the entire project to a temporary directory, causing unnecessary overhead. Moreover, if your project references packages from the parent directory as dependencies, copying becomes infeasible either.

@sanmai-NL
Copy link
Contributor Author

I think if we can define more strictly what mutations/write transactions are needed where, we can in practice work around this without significant disadvantages. In my case, only pyproject.toml, pdm.toml and pdm.lock are bind mounts and the rest of the filesystem tree isn't, avoiding some of the essential limitations you point out.

Arguments in favor of such a more refined design, are that read-only filesystems or parts thereof (such as some mount points) are common in Linux-containerized scenarios. These environments allow for more performance-efficient and hermetic builds. See source identity on https://bazel.build/basics/hermeticity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🐛 bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sanmai-NL @frostming