-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
89f98dd
commit 9901571
Showing
6 changed files
with
42 additions
and
19 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,12 +6,13 @@ You can display your current usage with the command 'uquota'. | |
|
||
When you exceed your quota, the system will not let you write any more data and you have to either remove some files or request more quota. The 'uquota' command will also show the date and to what limit your quota will change to, if you have been given a larger quota. | ||
|
||
To get more quota, send a mail to support ([email protected]) and state how much, for how long time, and why you need it. See the storage project application page for more information on how we handle and prioritise storage requests. | ||
To get more quota, send a mail to support (`[email protected]`) and state how much, for how long time, and why you need it. See the storage project application page for more information on how we handle and prioritise storage requests. | ||
|
||
Here are two commands. The first results in a list of subdirectories ordered by size and proportion of total size. The second produces a list of the files in the current directory that take up most space. These may take a long time to complete, use 'ctrl-c' to stop the process if you change your mind. | ||
|
||
``` | ||
```bash | ||
du -b $PWD | sort -rn | awk 'NR==1 {ALL=$1} {print int($1*100/ALL) "% " $0}' | ||
find $PWD -print0 -type f | xargs -0 stat -c "%s %n" | sort -rn | ||
``` | ||
|
||
You should also read the Disk Storage Guide. **LINK** |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,7 +10,24 @@ Parallel network file systems are very fast when accessed from many nodes, but c | |
|
||
For this reason, jobs that perform a lot of file accesses, especially on temporary files, should use the compute node's local hard drive. If you do, then any slow-down due to file I/O is limited to the node(s) on which these jobs are running. | ||
|
||
The hard drive of the node is located at /scratch, and each job that runs on a node gets a folder created automatically with the same name as the jobid, /scratch/<jobid>. This folder name is also stored in the environment variable $SNIC_TMP for ease of use. The idea is that you copy files that you will be reading randomly, such as indices and databases but not files of reads, to $SNIC_TMP first thing in the job. Files that you read as a stream from beginning to end, like files of reads, should remain in project storage and read from there. You then run your analysis and have all the output files written to $SNIC_TMP as well. After the analysis is done, you copy back all the output files you want to keep to your project storage folder. Everything in /scratch/<jobid> will be deleted as soon as the job is finished, and you have no hope of recovering it after the job is completed. | ||
The hard drive of the node is located at `/scratch`, | ||
and each job that runs on a node gets a folder created automatically | ||
with the same name as the jobid, `/scratch/<jobid>`. | ||
This folder name is also stored in the environment variable `$SNIC_TMP` | ||
for ease of use. | ||
The idea is that you copy files that you will be reading randomly, | ||
such as indices and databases but not files of reads, | ||
to `$SNIC_TMP` first thing in the job. | ||
Files that you read as a stream from beginning to end, like files of reads, | ||
should remain in project storage and read from there. | ||
You then run your analysis and | ||
have all the output files written to `$SNIC_TMP` as well. | ||
After the analysis is done, | ||
you copy back all the output files you want to keep | ||
to your project storage folder. | ||
Everything in `/scratch/<jobid>` will be deleted | ||
as soon as the job is finished, | ||
and you have no hope of recovering it after the job is completed. | ||
|
||
An example would be a script that runs bwa to align read. Usually they look something like this: | ||
|
||
|
@@ -62,4 +79,10 @@ cp $SNIC_TMP/sample.bam /proj/snic2022-X-YYY/nobackup/results/ | |
|
||
It's not harder than that. This way, the index files are copied to $SNIC_TMP in a single operation, which is much less straining for the file system than small random read/writes. The network filesystem is used when gathering reads for alignment, and streaming reads are easy for that filesystem. When the alignment is finished the results is copied back to project directory so that it can be used in other analysis. | ||
|
||
One problem that can happen is if your files and the results are too large for the node's hard drive. The drive is 2TiB on Rackham and 4TiB on Bianca, so it is unusual for the hard drive to be too small for the results of such analyses. If you run into this problem, please email UPPMAX at [email protected] and we can look into the problem. | ||
One problem that can happen is if your files and the results are too large | ||
for the node's hard drive. | ||
The drive is 2TiB on Rackham and 4TiB on Bianca, | ||
so it is unusual for the hard drive to be too small | ||
for the results of such analyses. | ||
If you run into this problem, | ||
please email UPPMAX at `[email protected]` and we can look into the problem. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters