clean up local tmp if it has been saved to a tarball #267
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Instance
|
boegel
reviewed
Jun 22, 2023
|
bot: show_config |
Updates by the bot instance
|
Updates by the bot instance
|
|
Instance
|
|
Instance
|
|
bot: show_help |
Updates by the bot instance
|
Updates by the bot instance
|
|
bot: show_config |
Updates by the bot instance
|
Updates by the bot instance
|
|
Instance
|
|
Instance
|
|
bot: show_config |
Updates by the bot instance
|
Updates by the bot instance
|
|
Instance
|
|
Instance
|
trz42
pushed a commit
to trz42/software-layer
that referenced
this pull request
Feb 3, 2024
{2023.06}[foss/2023a] TensorFlow v2.13.0
bedroge
changed the base branch from
2023.06-pilot.eessi-hpc.org
to
2023.06-software.eessi.io
bedroge
reviewed
Apr 16, 2024
| # double-check that at least one tarball of the temporary storage was created | ||
| if check_tmp_tarball ${TARBALL_TMP_BUILD_STEP_DIR} || check_tmp_tarball ${TARBALL_TMP_TARBALL_STEP_DIR}; then | ||
| echo "Removing temporary storage under '${STORAGE}'" | ||
| rm -rf ${STORAGE} |
There was a problem hiding this comment.
If I understand things correctly, $STORAGE refers to the base directory where all jobs create their unique subdirectory? In that case, shouldn't we only remove ${JOB_STORAGE} here, as there may be other (running) jobs on the same machine?
There was a problem hiding this comment.
I think, that's right.
Surprisingly, this ${JOB_STORAGE} is just created (a subdirectory under ${STORAGE}) but it is never used.
We should do two things
- replace uses of
${STORAGE}by${JOB_STORAGE}when runningeessi_container.shand test properly if that still works (maybe a separate PR would be beneficial) - replace uses of
${STORAGE}by${JOB_STORAGE}in this PR here such that only job specific job temporary storage is removed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
bot/build.shrunseessi_container.shtwice and each run results in a tarball containing the temporary folder. If a job runs on a machine where this temporary folder is not cleaned up automatically, disk usage can pile up leading to failing build jobs.This PR checks if a temporary folder is accessible (
${STORAGE}). If so it removes it if there is any tarball created for it.Improves situation that resulted in creating EESSI/eessi-bot-software-layer#186