Accessing the computing clusters

In a nutshell

Préambule pour les utilisateurs de systèmes Windows

Pour les utilisateurs Windows, quelques étapes sont nécessaires afin de pouvoir utiliser les commandes d’accès aux clusters. Ce protocole a été testé sous Windows 10.

1 Client OpenSSH

Vérifiez que vous avez le client OpenSSH. Pour ce faire :

  • Allez dans Paramètres > Applications > Fonctionnalités facultatives
  • Vérifiez la présence du Client OpenSSH
  • S’il n’y est pas, ajoutez-le

2 Activation client

Pour activer le client OpenSSH (ceci permettra de le laisser actif à chaque connexion) :

  • Allez dans Services dans le menu Démarrer
  • Cherchez OpenSSH Authentication Agent
  • Clic droit puis propriétés
  • Changez le mode de démarrage en Automatique

3 Vérification

Vérifiez l’activation du client :

  • Ouvrez cmd et tapez where ssh
  • Vérifiez que le chemin est dans System32 (par exemple C:\Windows\System32\OpenSSH\ssh.exe)

4 Bien joué

Vous avez maintenant accès aux commandes ssh-agent, ssh-add et plusieurs autres commandes ssh. Vous pouvez maintenant suivre les instructions comme si vous étiez sous Linux, en utilisant, au choix, cmd ou PowerShell.

La commande ssh-copy-id n’est pas accessible. Cela pourra être contourné en utilisant cat.

For Unix/Linux/MacOS X users

Follow these five steps in order to set up your cluster access.

1 PERSEUS account

Get a PERSEUS account and join an active project. Wait 24 hours after these steps for your account to become active, during that time prepare your setup to access the clusters.

2 Initial connection to the bastions

Make an initial ssh connection to both the bastions using your PERSEUS login and password in order to register their ssh fingerprints on your local account.

ssh <your-perseus-login>@rotule.univ-grenoble-alpes.fr

ssh <your-perseus-login>@trinity.univ-grenoble-alpes.fr

Replace <your-perseus-login> with the login of your PERSEUS account.

3 Initial connection to the clusters

Make an initial connection from both the bastions to the clusters you wish to use, again in order to register their ssh fingerprints, but this time in your accounts on the bastions. Before you can do this you have to wait 24 hours at most after you have joined an active project.

ssh <cluster>

Replace <cluster> with the name of the clusters you wish to use such as dahu, bigfoot, luke…

4 Transparent SSH connexion configuration

Add the following lines to your .ssh/config.

Host *
  ServerAliveInterval 30

Host *.ciment
  User <perseus-login>
  ProxyCommand ssh -q <your-perseus-login>@access-gricad.univ-grenoble-alpes.fr "nc -w 60 `basename %h .ciment` %p"

ProxyCommand can be used once you have set up ssh key access.

The ServerAliveInterval is mandatory and should not be set to a higher value.

5 SSH key access setup

Create an ssh key which you will use to access the clusters. Copy the public key to both bastions:

ssh-copy-id -i </the/path/to/your/key> <your-perseus-login>@rotule.univ-grenoble-alpes.fr:

and:

ssh-copy-id -i </the/path/to/your/key> <your-perseus-login>@trinity.univ-grenoble-alpes.fr:

and to the clusters:

ssh-copy-id -i </the/path/to/your/key> <your-perseus-login>@<the-cluster-you-will-user>.ciment:

Repeat for each cluster you need. This is required if you wish to use transparent access through the bastions.


Getting an account

Prior to accessing the clusters you have to get an account by registering via PERSEUS: PERsonal Space for cimEnt USers. You also need to be member of an active project before you can do anything on the computing clusters. Please refer to the PERSEUS documentation for further details.

Common authentication

PERSEUS provides a central authentication service used by multiple services offered by GRICAD.

When you create your PERSEUS account, you define a login and a password. These can be used to give you access, via ssh, to the computing clusters such as Luke and Dahu.

Why is the access to the clusters filtered?

Cluster access is normally done using an SSH client. However, SSH servers are very vulnerable to scans and attacks. For security reasons, it is not possible to let the clusters be directly accessed from the Internet.

We therefore provide two dedicated SSH gateways. They are more secure than cluster frontends. They also provide a common access point to all the clusters.

The classical method is to first login to an SSH gateway and then to login to the target cluster. The downside is that this requires you to enter your password twice. We will see, in the next sections, how to make this process transparent and much less troublesome..

SSH cluster access schema

graph LR; A[Your machine] -->|ssh| B[SSH gateways] B -->|ssh| C[Dahu head node] B -->|ssh| E[Bigfoot head node] B -->|ssh| D[Luke head node] B -->|ssh| I[CiGri head node] C -->|oar| F[Dahu compute nodes] E -->|oar| H[Bigfoot compute nodes] D -->|oar| G[Luke compute nodes] I -->|cigri| J[Cluster head nodes] J -->|oar| K[Cluster compute nodes]

SSH gateway hosts

There are 2 ssh bastions to access the GRICAD computing facilities. They are grouped under a single DNS name:

  • access-gricad.univ-grenoble-alpes.fr

This allows for load balancing on these two machines via a DNS round-robin mechanism. Moreover, if one of the servers ever came to fail, the other one remains available. In this situation, access via the round-robin name will fail once in every two successive connexion attempts and you can connect using the name of the remaining functionnol server if you wish to avoid this. These servers are:

  • rotule.univ-grenoble-alpes.fr
  • trinity.univ-grenoble-alpes.fr

It offers load balancing and, in addition, if one fails, you just have to try a second time and you’ll be directed to the other providing no one else has used the same command in the meantime. DNS round robin works by cycling through the possible values each time a new request is done regardless who the client is. Therefore if someone else queries access-gricad.univ-grenoble-alpes.fr immediately after you have done, the next query will fall again on the same host as the one you queried.

Password authentication

This is the most basic method to connect to a cluster. It is functional but is not the most efficient method possible. However you must do it at least once to test if everything is working:

monlogin@monpc:~$ ssh login-perseus@access-gricad.univ-grenoble-alpes.fr
The authenticity of host 'access-gricad.univ-grenoble-alpes.fr (129.88.196.128)' can't be established.
ECDSA key fingerprint is SHA256:fLweOXXtjaQ+Vr8PpmZTNlZIwb93oO/VmQh62qPPr34.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'access-gricad.univ-grenoble-alpes.fr,129.88.196.128' (ECDSA) to the list of known hosts.


******************************************************
                RESTRICTED ACCESS
      If you can't access, please, make sure 
            you've accepted the charter 
      https://perseus.univ-grenoble-alpes.fr/
******************************************************



login-perseus@access-gricad.univ-grenoble-alpes.fr's password: 
Linux rotule 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64

[...]

login-perseus@rotule:~$ 

Of course, replace login-perseus with your own PERSEUS login. Note also that login-perseus may be different from your local machine login mylogin.

When you connect for the first time to the SSH gateways your client will ask you if you recognize the the key fingerprint and if you want to continue. The SSH key allows you to make sure you are connecting to the proper and are not the victim of a man in the middle attack. The SSH gateway key fingerprints are given below. If it corresponds to one of these values you may safely accept and continue.


SSH gateway key fingerprints

  • RSA EDOYGE0LQIkZeyOO2CS8ZhR6/o3GthCGV7neNuqMgY4
  • ECDSA fLweOXXtjaQ+Vr8PpmZTNlZIwb93oO/VmQh62qPPr34
  • ED25519 7LNVN2unYEDLLVH4FZyeFJ66w5XBT4SsAAgBtn97pkE

As you used the name access-gricad.univ-grenoble-alpes.fr, you may be connected to rotule or trinity. Then try to connect to a cluster’s head node:

login-perseus@rotule:~$ ssh <ciment-cluster>
Password:  <type your paswword>
Last login: Wed Aug 26 12:18:18 2009 from 129.88.34.220

[...]

[login-perseus@ciment-cluster ~]$

Replace here by the real name of the cluster you are trying to connect to.

Passwords cannot be changed on the clusters. It can be done on the PERSEUS account management interface or via the AGALAN account management interace if your PERSEUS account is linked to your AGALAN account.

The same way that you had to check the SSH gateway fingerprint upon first connexion and answer that you recognized it before proceeding, you have to verify the fingerprint of the cluster head node upon first connexion. It is important to do this on both access hosts for each cluster otherwise the connexion will fail as it will be expecting your input. You therefore have to do at least one ‘ssh ’ for each cluster you will be using on each ssh bastion. The SSH cluster head node fingerprints are given below in order to allow you to check them to make sure you are safe from a man in the middle attack.

Dahu head node key fingerprints

  • 256 SHA256:LvTJwHNh6VYHa2WcHkiDJ7/5uMRurE4vPgryByHMSvE root@f-dahu (ECDSA)
  • 256 SHA256:frSjHEJyxO2XJO2RofBkhqLx9loPv4Ji4pEF60QewVk root@f-dahu (ED25519)
  • 2048 SHA256:++iF71wRwbYtE5zFx8uNY3Ahqv38gVHLlGxLjJJc5Bs root@f-dahu (RSA)

Bigfoot head node key fingerprints

  • 256 SHA256:Y85zqzkCu3dBEOyrDgTR5gnbU47r48VB9RSrTaueP40 root@bigfoot (ECDSA)
  • 256 SHA256:vHFGhKXTyD/64gkZWFO+uhCgwp5NQLqP2R9uI2fT2Lw root@bigfoot (ED25519)
  • 2048 SHA256:uUFMSzk2lpqHQEvVTD1NX7ESMwYSOnKWEfVKfJT17RQ root@bigfoot (RSA)

Luke head node key fingerprints

  • 1024 SHA256:neEKeikIL2064PpEfmDaO6qQzRcUiHl79cfoOh+MXNw root@luke (DSA)
  • 256 SHA256:aRuf2qZ6yHvc1tp0zP/kFS9RRdjdP2pf2pd79KqwE0k root@luke (ECDSA)
  • 2048 SHA256:2SEIkZ78qRUy9TU2l9HgYKSYqPfbFVqoUR1EeeljdmM root@luke (RSA)

CiGri head node key fingerprints

  • 256 SHA256:RPHa9C4fj/94AvqOEjZjOqO9N4CuCJAAp1yLvzJKjhc root@killeen3 (ECDSA)
  • 256 SHA256:mVsTwYE1qtlQXfwLNGPRCviRlRSG18/VR3iZJbYewzM root@killeen3 (ED25519)
  • 2048 SHA256:3oPbQmE/jXpxEGJQvhUob+KC0eioRlKkFGCRQA1A4O4 root@killeen3 (RSA)

Connecting to the gateways without a password

SSH keys can be used to connect to the gateways and clusters. This has several advantages which we will see further on.

This method is done by generating a public/private key pair and placing the public key on the target server.

The private key is very important and must, at all costs, always remain secret. It must be protected by a robust password and must never be shared.

Generating a key pair

You only need to do this once. It must be done on your personal computer.

It can be done using the ssh-keygen on UNIX/Linux systems.

mylogin@myhost:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/mylogin/.ssh/id_rsa): <enter>
Enter passphrase (empty for no passphrase): <enter a passphrase here>
Enter same passphrase again: <enter a passphrase here>
Your identification has been saved in /home/mylogin/.ssh/id_rsa.
Your public key has been saved in /home/mylogin/.ssh/id_rsa.pub.
The key fingerprint is:
21:f2:fe:d2:60:3e:26:e1:7b:a2:bd:48:6c:59:59:75 mylogin@myhost
The key's randomart image is:
+--[ RSA 2048]----+
|        . E      |
|       . .       |
|    . o .        |
|     = . .       |
|    o . S        |
| . o..o          |
|  =. +.o         |
| o o+ *..        |
|  o.+B o.        |
+-----------------+
mylogin@myhost:~$

You must give a passphrase. This is critical. Do not hesitate to give a complicated passphrase. As we will see later on, you will only need to give this passphrase once when you start your session. The key will then remain loaded in your agent until you close your session.

To wrap things up, we have:

  • created a private key, stored on your computer in the .ssh/ directory in your home
  • created a public key corresponding to the private key, normaly also stored in the .ssh/ directory in your home
  • encrypted your private key with a strong passphrase

SSH agent usage

The SSH agent allows you to load your key once and keep it available for all new connexions as long as your session is open and the agent is active.

On most modern systems, the ssh-agent is automatically started upon opening your session. If this is not the case, you may need to start it by running mylogin@myhost:~$ ssh-agent bash.

You can now load your private ssh key in the agent. If you have only one key in your .ssh/ directory, you can do this by running mylogin@myhost:~$ ssh-add:

mylogin@myhost:~$ ssh-add
Enter passphrase for /home/mylogin/.ssh/id_rsa: <enter your passphrase>
Identity added: /home/mylogin/.ssh/id_rsa (/home/mylogin/.ssh/id_rsa)
mylogin@myhost:~$

If you have more than one key in your .ssh/ directory, you have to specify its' path:

mylogin@myhost:~$ ssh-add .ssh/my_ssh_key
Enter passphrase for /home/mylogin/.ssh/my_ssh_key: <enter your passphrase>
Identity added: /home/mylogin/.ssh/my_ssh_key (/home/mylogin/.ssh/my_ssh_key)
mylogin@myhost:~$

The private key is now decrypted and loaded in memory. You won’t have to type your passphrase again until you leave the shell started with the agent or exit your session.

Uploading the public key to the access hosts and clusters

You now have to upload your public key to the 2 access hosts using the ssh-copy-id command. It’s the last time you’ll have to give your password (your PERSEUS password, not to be confused with your ssh key passphrase!):

mylogin@myhost:~$ ssh-copy-id login-perseus@rotule.univ-grenoble-alpes.fr
login-perseus@rotule.univ-grenoble-alpes.fr's password: <type your ciment password>

Now try logging into the machine, with ssh 'login-perseus@rotule.univ-grenoble-alpes.fr', and check in .ssh/authorized_keys to make sure we haven’t added extra keys that you weren’t expecting.

mylogin@myhost:~$ ssh-copy-id login-perseus@trinity.univ-grenoble-alpes.fr
login-perseus@trinity.univ-grenoble-alpes.fr's password: <type your ciment password>

Now try logging into the machine, with ssh 'login-perseus@trinity.univ-grenoble-alpes.fr', and check in .ssh/authorized_keys to make sure we haven’t added extra keys that you weren’t expecting.

If the command ssh-copy-id is not available, you can use the following command:
cat ~/.ssh/id_rsa.pub | ssh login-perseus@rotule.univ-grenoble-alpes.fr 'cat > .ssh/authorized_keys'
Beware, if the .ssh/authorized_keys already exists it will be squashed by this command, without any possibility of recovery!
If it already exists and you wish to keep it, use this syntax instead:
cat ~/.ssh/id_rsa.pub | ssh login-perseus@rotule.univ-grenoble-alpes.fr 'cat >> .ssh/authorized_keys'

If you have multiple ssh keys on your computer, ssh-copy-id will copy all of them to the remote host. You may want to edit your .ssh/authorized_keys on the remote host to remove the ones not intended for connexion to the clusters.
It may however be better practice to select the key you wish to copy for this specific connexion using the -i option of ssh-copy-id to specify the identity file from which to copy the public key.
Also, if you have too many keys to copy, ssh-copy-id will be rejected by the host for too many failed connexion attempts before you can even copy a single key.

If you have copied your ssh keys to the bastions, you can then also copy it to the cluster head nodes in order to access the clusters without any password usage. This operation is explained in greater detail below in the section on making the access host transparent.

Testing the setup

Try to connect to the first access host:

mylogin@myhost:~$ ssh login-perseus@rotule.univ-grenoble-alpes.fr

[...]

login-perseus@rotule:~$

Then try connecting to the second one:

mylogin@myhost:~$ ssh login-perseus@trinity.univ-grenoble-alpes.fr

[...]

login-perseus@trinity:~$

If you didn’t need to enter your SSH key password, then your configuration is correct, good job!

You can now try to connect and disconnect several times using ssh login-perseus@access-gricad.univ-grenoble-alpes.fr. This should connect you alternatively to rotule and trinity without having to give a password.

If the function of the SSH agent does not appear to be clear from the previous steps, log out from the current shell (exit) and re-try to connect to the access hosts. You’ll see that you are prompted for your passphrase each time as your private key needs to be uncrypted before use. The role of the agent is to keep your key uncrypted during your session so that you don’t have to type your passphrase again and again.

SSH agent

Gnome/KDE

On some Linux distributions, if you are using Gnome or KDE, the window manager may be configured to automatically open a window asking you for the ssh passphrase and you just have to check a box for this passphrase to be memorized for the whole session. If it is not working automatically, try to add the ssh-add command at the starting of your session. For KDE it is as simple as:

ln -s /usr/bin/ssh-add ~/.kde/Autostart/

For gnome, go into the System -> Preferences -> More Preferences -> Sessions -> Startup options menu and add /usr/bin/ssh-add.

Console ssh-agent

You can add the following code at the bottom of the .bash_profile file in your home directory:

ssh-add
if [ $? -eq 2 ]
then
  echo lancement ssh-agent
  eval $(ssh-agent)
  ssh-add
fi

This will start automatically an ssh-agent if it not already loaded.

Making the access transparent using SSH proxycommands

Now that we can access to access-gricad.univ-grenoble-alpes.fr without giving a password, here is a “magical” configuration to make this access host completely invisible and the access to the clusters totally transparent! Put these 3 lines into your .ssh/config file:

Host *.ciment
  User login-perseus
  ProxyCommand ssh -q login-perseus@access-gricad.univ-grenoble-alpes.fr "nc -w 60 `basename %h .ciment` %p"

Replace login-perseus with your PERSEUS login.

The ‘-w 60’ parameter in the command is useful to keep the connexion active. It is important to keep this value. Lowering it can cause unwanted disconnexions or even an impossibility to connect to the clusters.

Now you can connect to every GRICAD cluster using the form .ciment. For example, you can connect to Dahu using:

mylogin@myhost:~$ ssh dahu.ciment

Because the access hosts know the clusters domain names, you can use the short names.

Therefore, you can use:

  • dahu instead of f-dahu.univ-grenoble-alpes.fr

  • bigfoot instead of bigfoot.univ-grenoble-alpes.fr

  • luke instead of luke.univ-grenoble-alpes.fr

But you still have to give a password? Unless you also copied your ssh key to the cluster head node as suggested previously, yes you still do, but you can correct this very simply using this command:

mylogin@myhost:~$ ssh-copy-id dahu.ciment

As seen earlier, you have to copy your ssh public key to the cluster head node, because now, you are accessing it directly (via access-rr-ciment!)

Now you can connect to all the ciment clusters just by suffixing their name with .ciment. You can even use “scp” to make file transferts:

mylogin@myhost:~$ scp my_file dahu.ciment:

Or even sftp:

mylogin@myhost:~$ sftp dahu.ciment

I receive broadcast messages when I am connected, how can I stop them?

If you receive broadcast messages, you can enter the mesg n command in your terminal to stop them.

You can add the mesg n command in your .profile or .bash_profile file to make it systematic.

If you receive such messages, don’t hesitate to report it to us by writing to sos-calcul-gricad@univ-grenoble-alpes.fr or by going to our ticket manager. Don’t forget to specify the message you receive and the machine on which you have noticed them.