Data transfer¶

This guide covers how to transfer files to and from our HPC clusters. All transfer methods use SSH, so your existing account and SSH key setup will work without additional configuration.

Which tool should I use?

Most users: use rsync: it handles everything well, resumes interrupted transfers, and only sends what has changed.
Quick one-off transfers: use scp for simplicity.
Interactive browsing: use sftp or a GUI client like WinSCP.

File systems overview¶

Before transferring data, make sure you know where your files need to go:

HyperionAtlas EDRAtlas FDR

File system	Path	Quota	Compute nodes	Notes
Home	`/dipc`	50 GB / 300K files	Read-write	Config files, dotfiles, small data
Scratch	`/scratch`	2 TB / 10M files	Read-write (BeeGFS)	Job working directory, all I/O
Data	`/data`	10 TB / 20M files	Read-only	Long-term storage of results
Lscratch	`/lscratch`	None	Read-write (local disk)	Fast I/O for single-node jobs

Note

On Hyperion, /dipc is a symbolic link to /home. Both paths work, but we recommend using /dipc consistently across all clusters.

File system	Path	Quota	Compute nodes	Notes
Home	`/dipc`	None	Not accessible	Config files, backed up daily
Scratch	`/scratch`	1.5 TB / 3M files	Read-write	Job working directory, all I/O
Lscratch	`/lscratch`	None	Read-write (local disk, ~750 GB)	Fast I/O for single-node jobs

/dipc is not accessible from compute nodes

On Atlas EDR, /dipc is only mounted on login nodes. On compute nodes, /dipc is a symlink to /scratch. Always submit jobs from and direct output to /scratch.

File system	Path	Quota	Compute nodes	Notes
Home	`/dipc`	None	Not accessible	Config files, backed up daily
Scratch	`/scratch`	500 GB / 3M files	Read-write	Job working directory, all I/O
Lscratch	`/lscratch`	None	Read-write (local disk, ~700 GB)	Fast I/O for single-node jobs

/dipc is not accessible from compute nodes

On Atlas FDR, /dipc is only mounted on login nodes. On compute nodes, /dipc is a symlink to /scratch. Always submit jobs from and direct output to /scratch.

Clusters have separate file systems

Atlas and Hyperion have separate home and scratch directories. Files in your home on Atlas are not visible from Hyperion and vice versa. To move data between clusters you have two options:

Via your local machine: download from one cluster, then upload to the other. This works with your existing SSH key setup.
Direct transfer between clusters: you can rsync/scp directly between Atlas and Hyperion, but this requires generating an SSH key pair on the source cluster and registering the public key on the authentication server or in ~/.ssh/authorized_keys on the destination.

Hostnames¶

Use these hostnames in all the examples below, replacing username with your assigned username:

Cluster	Hostname
Hyperion	`hyperion.sw.ehu.es`
Atlas EDR	`atlas-edr.sw.ehu.es`
Atlas FDR	`atlas-fdr.sw.ehu.es`

The examples on this page use atlas-fdr.sw.ehu.es, but you can substitute any of the hostnames above.

rsync (recommended)¶

rsync is the most versatile transfer tool. It synchronizes files efficiently by only transferring the differences between source and destination, which makes it ideal for large datasets, repeated transfers, and backups.

Common examples¶

Upload a directory to the cluster:

rsync -avz /local/path/ username@atlas-fdr.sw.ehu.es:/dipc/username/target_dir/

Download a directory from the cluster:

rsync -avz username@atlas-fdr.sw.ehu.es:/dipc/username/source_dir/ /local/path/

Preview what would be transferred (dry run):

rsync -avz --dry-run /local/path/ username@atlas-fdr.sw.ehu.es:/dipc/username/target_dir/

Transfer with progress and bandwidth limit:

rsync -avz --progress --bwlimit=10000 /local/path/ username@atlas-fdr.sw.ehu.es:/dipc/username/target_dir/

Sync and delete files on the destination that no longer exist locally:

rsync -avz --delete /local/path/ username@atlas-fdr.sw.ehu.es:/dipc/username/target_dir/

Warning

The --delete flag will remove files on the remote side that are not present locally. Always run with --dry-run first to verify what will be deleted.

Useful flags¶

Flag	Description
`-a`	Archive mode (preserves permissions, timestamps)
`-v`	Verbose output
`-z`	Compress data during transfer
`--progress`	Show transfer progress
`--dry-run`	Simulate the transfer without making changes
`--delete`	Delete files on destination not present in source
`--exclude 'pattern'`	Exclude files matching a pattern
`--bwlimit=KBPS`	Limit bandwidth usage (in KB/s)

Trailing slashes matter

In rsync, /local/path/ (with trailing slash) copies the contents of the directory, while /local/path (without) copies the directory itself. This is a common source of confusion. When in doubt, use --dry-run first.

SCP¶

scp works like cp but over SSH. It is simple and effective for quick, one-off transfers but does not support resuming or incremental sync.

Upload a file:

scp /local/path/file.dat username@atlas-fdr.sw.ehu.es:/dipc/username/target_dir/

Download a file:

scp username@atlas-fdr.sw.ehu.es:/dipc/username/file.dat /local/path/

Upload a directory recursively:

scp -r /local/path/directory username@atlas-fdr.sw.ehu.es:/dipc/username/target_dir/

Transfer with compression:

scp -C /local/path/largefile.dat username@atlas-fdr.sw.ehu.es:/dipc/username/target_dir/

SFTP¶

sftp provides an interactive session for browsing, uploading, and downloading files. It is useful when you want to explore the remote file system before transferring.

Start a session:

sftp username@atlas-fdr.sw.ehu.es

Common commands within the session:

Command	Description
`ls`	List files on the remote server
`lls`	List files on your local machine
`cd directory`	Change remote directory
`lcd directory`	Change local directory
`put localfile`	Upload a file
`get remotefile`	Download a file
`mkdir directory`	Create a remote directory
`rm file`	Delete a remote file
`exit`	End the session

GUI clients (Windows)¶

If you prefer a graphical interface, WinSCP and FileZilla are good options. Both support SFTP and allow drag-and-drop file transfers.

To connect with WinSCP:

Set the protocol to SFTP (or SCP).
Enter the hostname (e.g. atlas-fdr.sw.ehu.es) and your username.
If using SSH key authentication, go to Advanced → Authentication and select your private key file.
Click Login and drag and drop files between the local and remote panels.

Quick comparison¶

Feature	`rsync`	`scp`	`sftp`
Best for	Large/repeated transfers	Quick one-off copies	Interactive browsing
Incremental sync	Yes	No	No
Resume interrupted	Yes	No	Yes
Compression	Yes (`-z`)	Yes (`-C`)	No
Complexity	Moderate	Simple	Simple (interactive)