Storage on FRCE
There are several options for file storage on the FRCE cluster, each with its own advantages and disadvantages. Shares available to all users include
| Directory | Quota | Availability | Data Protection |
|---|---|---|---|
| /home/username | 256GB | cluster-wide | Volume snapshots are kept for 30 days |
| /scratch/cluster_scratch/username | 5TB | cluster-wide | no protection |
| /scratch/local or /tmp | no quota, limited to volume size | not shared between hosts | no protection |
/home
Each user has a home directory called /home/username which is accessible from every HPC system. The /home directories have a quota of 256GB which cannot be increased. It is commonly used for config files, code, scripts, and analysis results. The share is relatively slow and applications should not process data on the share. Volume-level snapshots (backups) of the directories are made daily and kept for 30 days. Users can restore their own files from the past 14 days from the directory /home/.snapshot/timestamp/user. Restores from later than that will require a Service Now ticket directed to the EIT Storage group.
/scratch/cluster_scratch
This share is mounted on all FRCE nodes and is a high-performance network share. While the intent is for transient storage of raw data there is no defined lifetime for storage. The system does not ever delete data from this share. It is the responsibility of the user to manage file storage.
/scratch/local
This share is also mounted as /tmp and is local to each server. Files stored on one system will not be available on other systems, including the head node. Intended for data needed only during the duration of a single job and the files may be deleted when the systems reboot. Performance is comparable to that of /scratch/cluster_scratch.
Snapshots of most NFS shares are made daily and kept for a varying number of days depending on the requirements of the share owner. These snapshots are user accessible, and any user can retrieve earlier versions of their files. Substitute your account name for username in this example.
# First, list the available snapshots by date $ ls -lc /home/.snapshot total 504 drwxr-xr-x 500 root root 257536 Aug 30 23:30 2263164_shared-home_shared-home drwxr-xr-x 500 root root 257536 Aug 31 23:30 2266190_shared-home_shared-home drwxr-xr-x 500 root root 257536 Sep 1 23:30 2269261_shared-home_shared-home drwxr-xr-x 500 root root 257536 Sep 2 23:30 2272677_shared-home_shared-home drwxr-xr-x 500 root root 257536 Sep 3 23:30 2276645_shared-home_shared-home drwxr-xr-x 500 root root 257536 Sep 4 23:30 2281308_shared-home_shared-home drwxr-xr-x 500 root root 257536 Sep 5 23:30 2285730_shared-home_shared-home drwxr-xr-x 500 root root 257536 Sep 6 23:30 2289821_shared-home_shared-home drwxr-xr-x 500 root root 257536 Sep 7 23:30 2293117_shared-home_shared-home drwxr-xr-x 500 root root 257536 Sep 8 23:30 2296740_shared-home_shared-home drwxr-xr-x 502 root root 258560 Sep 9 23:30 2301115_shared-home_shared-home drwxr-xr-x 502 root root 258560 Sep 10 23:30 2305897_shared-home_shared-home drwxr-xr-x 503 root root 259072 Sep 11 23:30 2310713_shared-home_shared-home drwxr-xr-x 504 root root 259584 Sep 12 23:30 2315649_shared-home_shared-home # Select the new snapshot taken before the date you need to recover from $ cd /home/.snapshot/2301115_shared-home_shared-home/username # Get a list of files/directories that can be recovered $ ls -l < ... file list ...> # Copy the needed file back to your directory $ cp -a .bashrc /home/username/.bashrc-recovered $ cd # Compare the dates on the new and old files $ ls -l .bashrc* -rw-r--r-- 1 username username 1496 Sep 13 09:36 .bashrc -rw-r--r-- 1 username username 1446 Jul 25 09:19 .bashrc-recovered