Skip to content

Commit

Permalink
Engine: Fix bug introduced when refactoring upload_calculation
Browse files Browse the repository at this point in the history
In 6898ff4 the implementation of the
processing of the `local_copy_list` in the `upload_calculation` method
was changed. Originally, the files specified by the `local_copy_list`
were first copied into the `SandboxFolder` before copying its contents
to the working directory using the transport. The commit allowed the
order in which the local and sandbox files were copied, so the local
files were now no longer copied through the sandbox. Rather, they were
copied to a temporary directory on disk, which was then copied over
using the transport. The problem is that if the latter would copy over a
directory that was already created by the copying of the sandbox, an
exception would be raised.

For example, if the sandbox contained the directory `folder` and the
`local_copy_list` contained the items `(_, 'file', './folder/file')`
this would work just fine in the original implementation as the `file`
would be written to the `folder` on the remote folder. The current
implementation, however, would write the file content to `folder/file`
in a local temporary directory, and then iterate over the directories
and copy them over. Essentially it would be doing:

    Transport.put('/tmpdir/folder', 'folder')

but since `folder` would already exist on the remote working directory
the local folder would be _nested_ and so the final path on the remote
would be `/workingdir/folder/folder/file`.

The correct approach is to copy each item of the `local_copy_list` from
the local temporary directory _individually_ using the `Transport.put`
interface and not iterate over the top-level entries of the temporary
directory at the end.
  • Loading branch information
sphuber committed May 4, 2024
1 parent 4934c20 commit c2a5903
Showing 1 changed file with 1 addition and 3 deletions.
4 changes: 1 addition & 3 deletions src/aiida/engine/daemon/execmanager.py
Original file line number Diff line number Diff line change
Expand Up @@ -366,9 +366,7 @@ def _copy_local_files(logger, node, transport, inputs, local_copy_list):
with data_node.base.repository.open(filename_source, 'rb') as source:
shutil.copyfileobj(source, handle)

# Now copy the contents of the temporary folder to the remote working directory using the transport
for filepath in dirpath.iterdir():
transport.put(str(filepath), filepath.name)
transport.put(str(filepath_target), target)


def _copy_sandbox_files(logger, node, transport, folder):
Expand Down

0 comments on commit c2a5903

Please sign in to comment.