-
Notifications
You must be signed in to change notification settings - Fork 7
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add docs for users who want to tape data
- Loading branch information
Showing
1 changed file
with
16 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
# Longterm Data Storage | ||
|
||
To keep our storage costs low our lab likes to move data that isnt used frequently on to tape for long term storage. If you have data you won't be using for a few months or longer and don't want to delete you should move it off our projects/scratch and on to tape to help us save money :) This page describes how to do this and how to get your data back when you need it again. | ||
|
||
## Moving data to tape | ||
1. Organize all the data you want moved into a folder named `to_tape` in either your projects or scratch. | ||
2. Add a README file to this folder explaining the contents for any future people who might try to use or identify it. | ||
3. Contact a lab employee. This can be anyone who works in the lab but preferentially should be Dawn ([email protected]) or Kevin ([email protected]). Just shoot an email or slack message asking for the data to be moved to tape (include the path to your `to_tape` folder). | ||
|
||
## Retrieving data from tape | ||
1. **Identify what you want pulled back from tape**. The records for what is stored on tape can be found at `/archive/tapes`. User data is stored by user id under the 'users' folder and specific datasets are under the 'datasets' folder. Everything is organized by *date that it was packaged* and there will be .ncdu files and .sha512 files for each item. The .sha512 files can be used to grep for a specific file/folder if you know what you're looking for and the .ncdu files can be opened and browsed with `ncdu -f <ncdu file path here>` if you're not sure what something will be named. Use the arrow keys to navigate and 'q' to quit. | ||
|
||
In some cases, data packaged on a certain date may have been split into parts (large tar files don't do well on tape!). Usually if this is the case, there will also be a file ending with `-all.ncdu` that shows the entirety of what was packaged on that date to make it easier to find things. | ||
|
||
2. **Contact a lab employee with a list of files/folders you want**. Once you've found everything you need, send a list of files/folders to either Dawn ([email protected]) or Kevin ([email protected]). If you can, include the the name of the .sha512 or .ncdu you found them in to make it easier on us :) | ||
|