Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Script for exporting a stack to a set of tarballs #141

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

bedroge
Copy link
Collaborator

@bedroge bedroge commented Jan 6, 2023

Currently trying this to export the 2021.06 stack before actually removing it, see #140.

@bedroge
Copy link
Collaborator Author

bedroge commented Jan 6, 2023

This is the resulting tarball for aarch64:

-rw-r--r-- 1 root root  23G Jan  6 10:46 eessi-2021.06-linux-aarch64.tar.gz

And a listing of its contents:
eessi-2021.06-linux-aarch64.tar.gz.list.gz

@ocaisa
Copy link
Member

ocaisa commented Jan 6, 2023

Should we provide the tarball via the new data repo? We could also replace the version via a symlink to somewhere under host_injections, that would allow us to add logic to check if they've unpacked the tarball and use the archived stuff in the initialisation (and advise them what to do if they haven't)

@bedroge
Copy link
Collaborator Author

bedroge commented Jan 6, 2023

I definitely like the idea with the symlink! Storing the tarballs in the data repo doesn't really work at the moment, as they're way too large for the current setup. Then we should definitely use the approach with some external storage (S3 bucket) and only use CVMFS on top for the metadata / filesystem layer. So, for now, I'd propose that we just store it somewhere in an S3 bucket, and maybe we can set up the CVMFS repo that way later on (or make something like archive(d).eessi.io)?

@boegel
Copy link
Contributor

boegel commented Jan 6, 2023

+1 for just storing the exports in an S3 bucket for now

@bedroge
Copy link
Collaborator Author

bedroge commented Jan 9, 2023

I've stored them in s3://eessi-archive/2021.06 on AWS for now.

@ocaisa
Copy link
Member

ocaisa commented Jan 9, 2023

Can we get S3 storage for free somewhere? I bet we can...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants