Get sandbox in schedd #8645
Get sandbox in schedd #8645
Conversation
This reverts commit 57840ae.
|
Jenkins results:
|
| Check if file exist in S3. Raise exception if wget is exit with non-zero. | ||
| Usually, you will see stderr with http response `404 Not Found` if file does not exists. | ||
| Note that presigned url from GetObject API could not use by HeadObject API. | ||
| Use 'head -c1000' to fetch few bytes instead, and need to wrap inside bash |
There was a problem hiding this comment.
?? can you explain ? I have not found a use of head -c100 in the code.
There was a problem hiding this comment.
Sorry, it is the leftover from previous attempt to avoid extra code on REST side.
It does not work. So switching to HeadObject api instead.
| @@ -727,6 +727,16 @@ def prepareLocal(self, dagSpecs, info, kw, inputFiles, subdags): | |||
| finally: | |||
| tf.close() | |||
|
|
|||
| # also upload InputFiles.tar.gz to s3 | |||
| # Wa: Now (2024) I am still not sure if we need to poll uploaded file | |||
There was a problem hiding this comment.
well, here we upload from a CERN node a <2MB file, not a large user sandbox from a remote client.
Maybe we can add a call to the new checkS3Object() at the end of uploadToS3
There was a problem hiding this comment.
But sandbox should finish uploading (from client) before reaching this line, no?
Or maybe I misread the code 🤔
There was a problem hiding this comment.
we agree to
add a call to the new checkS3Object() at the end of uploadToS3
No description provided.