Transferring Files

Transferring large amounts of data can be CPU intensive and a 10-CPU-minute limit is enforced on the FRCE head node. It is recommended that you either log into fsitgl-xfer03p.ncifcrf.gov or set up an interactive batch login for data transfers.

File transfer clients

Several Windows clients are available to transfer files from a desktop to the cluster. A popular multi-platform file transfer is Filezilla. It runs on Windows, Mac, and Linux and has a intuitive interface. For Windows platforms, WinSCP also provides a windowed file transfer package.

For Macs and Linux systems, scp is a reasonable client.

rsync as a transfer agent

rsync will work with any remote Linux/UNIX system. It allows for the transfer of individual file or entire directory trees and if the transfer is interrupted, it can be restarted from where the interruption occurred, avoiding having to transfer files that have already been copied. In the case where data is often synced between sites, rsync is capable of only copying files that have been changed. As an example for a transfer:

$ cd /scratch/cluster_scratch/username
$ rsync -azP username@biowulf.nih.gov:/data/username/foo .

Globus ftp

Globus ftp is a file transfer agent that provided unattended file/directory transfers between sites within the Globus network. An account needs to be authorized to use the Frederick Globus endpoint. Some, but not all file shares available to the FRCE cluster are mounted on the Globus servers so transferred files may need to be moved between shares after they are transferred. The FRCE Globus page provides more information.