Copying Files

Transferring files to and from Alliance clusters should be done through data transfer nodes (data movers) rather than login nodes for better performance and to avoid overloading login resources. If you need a secure and efficient way to move files from your personal computer, you can use command-line tools such as SCP, SFTP, or higher-level programs like WinSCP and MobaXterm on Windows.

When transferring data between different clusters, Globus is the recommended tool due to its speed and reliability. It automatically uses data transfer nodes and can also be configured to sync files instead of overwriting them, ensuring only new or changed files get transferred. Refer to the Globus documentation for setup details.

To synchronize directories or large datasets, consider using rsync. By default, it compares file modification time and size, only sending differences. For large or numerous files, use options like --partial and --progress to resume interrupted transfers and see progress bars. Avoid preserving group ownership (i.e., don’t use -p/-g) when sending files to /project directories to prevent Disk quota exceeded errors.

For downloads from the web, tools like wget and curl can resume interrupted transfers using their respective command-line flags. If you need to move data from cloud services, rclone is a good alternative, supporting services such as Google Drive and Google Photos.

If you want to verify files after transferring or if syncing tools are insufficient, you can generate checksums with commands like sha1sum. Comparing checksums on the source and destination ensures file integrity across different systems.

For full details, including additional transfer examples, consult the Alliance Transferring Data Guide.

When transferring data, it's important to use data transfer nodes (also called data mover nodes) instead of login nodes. These nodes are specifically designed for efficient data transfer and are available for most Alliance clusters.

For transferring data between your personal computer and Alliance systems, several options are available:

1. Command-line tools: scp and sftp for Linux and Mac users.

2. Graphical tools: MobaXterm and WinSCP for Windows users.

3. Globus Personal Connect: Recommended for transfers taking more than a minute.

For transfers between Alliance resources, Globus is the preferred tool. However, other methods like scp can also be used.

To download data from websites, wget and curl are standard tools available on Alliance clusters. For cloud services like Google Drive, consider using rclone.

Synchronizing files between two locations can be done using several methods:

1. Globus: Offers the best performance and reliability.

2. rsync: A popular tool for ensuring two datasets are the same.

3. Checksums: Useful for verifying file integrity when other methods are unavailable.

For secure file transfers, SFTP and SCP are reliable options. Both use the SSH protocol to encrypt data during transfer.

Remember to always initiate transfers from your local computer when using SCP, regardless of the transfer direction. Also, be cautious when using scp -r to transfer data into /project directories, as it may lead to quota-related issues[1][2].

Citations: [1] https://docs.alliancecan.ca/wiki/Transferring_data [2] https://docs.alliancecan.ca/wiki/Transferring_data