Data transfer operations

Data transfer operations

Cargo, for Dahu, Bigfoot and Luke

For all large data volume transfer operations on the Dahu, Bigfoot and Luke clusters it is important to use the Cargo machine and never launch them on cluster head nodes.

The machine is a dedicated server for all large data volume transfer operations on the Luke and Dahu clusters.

Cargo is only accessible via ssh key authentication. It is therefore necessary to have copied your ssh key to Dahu in order to access it. You can’t access it by password. You can’t copy your ssh key directly to Cargo either, you have to copy it to the associated cluter first. If you have already set up the transparent connection to the clusters with the alias “proxycommand” .ciment, you can copy your ssh key to Dahu with the command ssh-copy-id dahu.ciment.

Do not try to connect to cargo if you have not already copied your SSH public key on the Dahu cluster.

If you try to connect without having beforehand copied your SSH public key on Dahu, you will get a "Permission denied (publickey)" error.

It provides access to:

  • Your home directory on the Dahu cluster, mounted on /home/$login/
  • Your home directory on the Bigfoot cluster, mounted on /home-bigfoot/$login/
  • Your home directory on the Luke cluster, mounted on /home-luke/$login/
  • The Bettik storage space, mounted on /bettik/
  • The Silenus storage space, mounted on /silenus/
  • The Mantis storage space, accessible via the iRODS commands
  • SUMMER storage volumes, mounted at user request under the /summer/ directory

It is accessible via the ssh and ftp protocols from any IP address. It therefore allows to run data transfer via rsync, scp or ftp between storage spaces on this server and any distant machine.

It is very important to use this server for all large data transfer operations, as it is a machine dedicated for this function, rather than cluster head nodes. Any large transfer operation launched on cluster head nodes is susceptible to be killed without warning.

As Cargo is accessible via ssh from any IP address. You can connect directly, via the command ssh, without having to pass through the bastions or cluster head nodes and run any file transfer between the different storage spaces from there.

An example of call allowing to synchronise a directory in your home space to Bettik is :

my-perseus-login@dahu|luke:~$ rsync -avxH /home/{mon-login}/{mon-repertoire}/ /bettik/{mon-login}/{mon-repertoire}/

NB: the / at the end of the paths are important.

Screen and tmux are available on Cargo, allowing you to create a command line terminal in which you can run your file transfer operations without having to stop them when you must shut down your connexion.

You can also transfer data between your local machine and Bettik using the cargo server :

my-local-login@ma-machine:~: rsync -avxH /{local-storage-path}/{path-to-my-data}/

Do not add a line loading an environment such as guix, nix, module or other (e.g. source /applis/site/ via the .bashrc or .profile file, you will probably get unexpected side-effects which usually prevent you from doing data transfers.