Data management, storage infrastructures

Good practises

TL;DR

  • draft a DMP (ask for help from sos-calcul or from the UGA Data Stewardship Unit;
  • document the data;
  • designate a permanent member as data manager for each project;
  • use the project shared directories;
    • /silenus/PROJECTS/ on Dahu and Bigfoot;
    • /bettik/PROJECTS/ on Dahu, Bigfoot and Luke;
    • /scratch/PROJECTS/ on Froggy;
  • do not store project data in your home directory, particularly for temporary members;
  • use the best suited storage space for your needs.

Recommendations

Some generalrules concerning data management apply particularly to the data storage systems of the GRICAD computing infrastructures.

In particular, it is strongly recommended to use the shared project storage spaces available on the different clusters, detailed below. This practice is generally preferable as it allows for more flexibility and efficiency within a project. It is especially important in the case of temporary collaborators, trainees, doctoral students, etc., for whom the data contained in the personal directory will be deleted after the expiry of the Perseus account.

These common storage spaces for projects, as proposed on the GRICAD computing infrastructures, offer the advantage of correctly managing access rights, by automatically authorising them to all members of the group but refusing them to people who are not members of the project, which avoids having to do this operation oneself since, by default, the data is authorised in read mode to all people who have access to the clusters.

The drafting of a DMP, even if it may seem restrictive and time-consuming, makes it possible to enhance the value of the data produced within the framework of a research project and is therefore very quickly profitable, even for small projects. Moreover, GRICAD and the UGA Data Stewardship Unit can help you create this DMP.

It is important to think about the life of the data, beyond the duration of the participation in a project of a temporary collaborator, trainee, doctoral student or post-doctoral student, and to think about what will happen to them after the end of the collaboration and the closing of the Perseus account, which will make them inaccessible if they are not properly managed. In this respect, support on this subject from the project managers is very important for these users who are not necessarily familiar with the structure of the storage systems set up on the computing clusters. To ensure the best possible management of this function, do not hesitate to contact us by writing to sos-calcul so that we can guide you or confirm you in your current uses.

Data storage infrastructures

Due to the volumes and high throughput of data on the GRICAD computing facilities, please be aware that there are NO BACKUPS of any data anywhere!

There are different storage infrastructures available to GRICAD HPC users:

High performance distributed scratch

Cloud storage

Cluster wide shared storage

  • 90Tb Lustre scratch on Froggy
  • Accessible via /scratch.
  • Per cluster home directories
  • These home directories spaces are relatively small compared to the number of users on the clusters. They are not made to store data;
  • They are meant for operational code storage;
  • For data, please use other available storage spaces.

Local scratch

  • Some nodes of the Luke cluster have local scratch spaces. These scratch spaces are dedicated to projects with specific needs which are not met by other existing solutions.

SUMMER

  • It is possible to mount SUMMER storage volumes as NFS shares on the Luke and Dahu clusters. If you have SUMMER volumes you wish to have mounted on our clusters please contact us. Please get in touch with the teams of project managers of the shares for volumes already mounted on the clusters.

These different infrastructures serve different needs and are tuned to offer different characteristics.

As there is NO BACKUP of any data, users are advised to use available solutions to secure their data.

For codes it is strongly advised to use a versioning system, preferably one with a high level of tolerance to failure. For example GRICAD provides such a service via gricad-gitlab.

For computing data which will be hosted on the Mantis infrastructure, you are strongly encouraged to use a backup system either within your laboratory or to use institutionnal services such as SUMMER


Data safety

Although all our infrastructures are designed to be as tolerant to failure as possible within the limits imposed by available technologies and operating constraints, due to the very large scale of the infrastructures hardware failures cannot be entirely mitigated and data loss cannot entirely be guaranteed never to happen. There also is no backup and therefore no protection against human error. We cannot guarantee there will never be any loss of data. You are therefore strongly advised to take appropriate measures.