Optimize file transfer

[chrisjsewell]

The key usage of the transport in aiida-core is in:

• file uploads, with transport.get: https://github.com/aiidateam/aiida-core/blob/9dfad2efbe9603957a54d0123a3cec2ee48b54bd/aiida/engine/daemon/execmanager.py#L205-L207
• file downloads, with transport.put: https://github.com/aiidateam/aiida-core/blob/9dfad2efbe9603957a54d0123a3cec2ee48b54bd/aiida/engine/daemon/execmanager.py#L516
• Also copy and symlink are in the same file

Currently, the get and put methods are "simple" in that (via also gettree/puttree) they make isolated/synchronous calls to getfile/putfile, to download/upload each file.

For small files (< 5Mb) this is maybe not so bad, since anyway these are transferred in a single GET/POST request (although for async PUT, see e.g. https://github.com/chrisjsewell/fireflow/blob/941ab9631537bcb8103331f6fc2aaecc720ab6da/src/fireflow/process.py#L307)

Certainly, though for large files, upload/download is a multi-step process, for download (and inverse for upload):

1. Initiate a transfer task of the target file to the "staging area" object store
2. Poll this task until it is completed
3. Download from the object store, via the supplied URL

For multi-file downloads, it would be ideal to initiate all transfers first, then poll all the tasks together, before starting to download from the object-store.

It is also of note that the code in execmanager does not provide all the files at once, rather in a loop (of either file or folder),
and so ideally there would be an upstream change there also