High Performance Computing infrastructures description

Computing clusters

Dahu, HPCDA platform

Head node: dahu.univ-grenoble-alpes.fr or dahu from the bastions.

  • HPCDA platform
  • 3256 Xeon Skylake cores
    • 2112 Xeon SKL Gold 6130 @ 2.10GHz, nodes 33 to 72 and 82 to 107
    • 896 Xeon SKL Gold 5218 @ 2.30GHz, nodes 108, 109 and 112 to 137
    • 216 Xeon SKL Gold 6126 @ 2.60GHz, nodes 76 to 81
    • 32 Xeon SKL Gold 6244 @ 3.60GHz, nodes 110 and 111
  • Omnipath 100Gb semi blocking network
  • Local SSD and HDD scratch space
  • Full list of characteristics displayed via the Message Of The Day

Bigfoot, GPU platform

Head node: bigfoot.univ-grenoble-alpes.fr or bigfoot from the bastions.

  • IA and Deep Learning platform
  • 3 nodes with 4 Tesla V100 GPUs with NV-link per node
  • 4 nodes with 4 Tesla V100 GPUs with PCIe link per node
  • 5 nodes with 2 Tesla A100 GPUs including one available as 7 independant MIGs
  • Omnipath 100Gb semi blocking network
  • 35 “Virgo” nodes with a T4 GPU available during night time (reserved for training during the day) in collaboration with UGA teaching services
  • Full list of characteristics available at any time by calling the recap.py instruction

Recapitulative table of the Dahu and Bigfoot clusters

 ========================================================================================
|   node   | cpumodel  |n_cpus n_cores| scratch1_type                           | hasgpu |
|          |           |     total_mem|                           scratch2_type |        |
 ========================================================================================
| dahu33   | Gold 6130 | 2 | 32 | 192 |dedicated_ssd       | dedicated_hdd      | NO     |
|    [ + 38 more node(s) ]                                                               |
| dahu72   | Gold 6130 | 2 | 32 | 192 |dedicated_ssd       | dedicated_hdd      | NO     |
| dahu73   | Gold 6126 | 2 | 24 | 192 |system_ssd          | dedicated_hdd      | NO     |
|    [ + 2  more node(s) ]                                                               |
| dahu76   | Gold 6126 | 2 | 24 | 192 |system_ssd          | dedicated_hdd      | NO     |
| dahu77   | Gold 6126 | 2 | 24 | 192 |dedicated_ssd       | dedicated_hdd      | NO     |
|    [ + 3  more node(s) ]                                                               |
| dahu81   | Gold 6126 | 2 | 24 | 192 |dedicated_ssd       | dedicated_hdd      | NO     |
| dahu82   | Gold 6130 | 2 | 32 | 192 |dedicated_ssd       | dedicated_hdd      | NO     |
|    [ + 23 more node(s) ]                                                               |
| dahu106  | Gold 6130 | 2 | 32 | 192 |dedicated_ssd       | dedicated_hdd      | NO     |
| dahu107  | Gold 6130 | 2 | 32 | 192 |dedicated_ssd       | none               | NO     |
| dahu108  | Gold 5218 | 2 | 32 | 192 |dedicated_ssd       | dedicated_hdd      | NO     |
| dahu109  | Gold 5218 | 2 | 32 | 192 |dedicated_ssd       | dedicated_hdd      | NO     |
| dahu110  | Gold 6244 | 2 | 16 | 192 |dedicated_ssd       | none               | NO     |
| dahu111  | Gold 6244 | 2 | 16 | 192 |dedicated_ssd       | none               | NO     |
| dahu112  | Gold 5218 | 2 | 32 | 192 |dedicated_ssd       | dedicated_hdd      | NO     |
|    [ + 24 more node(s) ]                                                               |
| dahu137  | Gold 5218 | 2 | 32 | 192 |dedicated_ssd       | dedicated_hdd      | NO     |
| dahu138  | Gold 5218 | 2 | 32 | 192 |system_ssd          | dedicated_hdd      | NO     |
| dahu139  | Gold 5218 | 2 | 32 | 192 |system_ssd          | dedicated_hdd      | NO     |
| dahu140  | Gold 6244 | 2 | 16 | 192 |system_ssd          | dedicated_hdd      | NO     |
| dahu-fat1| Gold 6244 | 2 | 16 |1133 |dedicated_raid0_ssd | dedicated_raid5_hdd| NO     |
| dahu-visu| Silver 4216| 2 | 32 | 192 |system_hdd          | none               | NO     |
 ========================================================================================
 ============================================================================
|   node   | cpumodel  | gpumodel  | gpus | cpus | cores| mem | mem/gpu |MIG|
 ============================================================================
| bigfoot1 | Gold 6130 | V100      |   4  |   2  |   32 | 192 |   96  |  NO |
|    [ + 1  more node(s) ]                                                  |
| bigfoot3 | Gold 6130 | V100      |   4  |   2  |   32 | 192 |   96  |  NO |
| bigfoot4 | Gold 5218R| V100      |   4  |   2  |   40 | 192 |   96  |  NO |
|    [ + 1  more node(s) ]                                                  |
| bigfoot6 | Gold 5218R| V100      |   4  |   2  |   40 | 192 |   96  |  NO |
| bigfoot7 | EPYC 7452 | A100      |   2  |   2  |   64 | 192 |   96  | YES |
| bigfoot8 | Gold 5218R| V100      |   4  |   2  |   40 | 192 |   48  |  NO |
| bigfoot9 | EPYC 7452 | A100      |   2  |   2  |   64 | 192 |   96  |  NO |
|    [ + 2  more node(s) ]                                                  |
| bigfoot12| EPYC 7452 | A100      |   2  |   2  |   64 | 192 |   96  |  NO |
| virgo1   | vcpu      | T4        |   1  |   1  |    2 |   4 |    4  |  NO |
|    [ + 33 more node(s) ]                                                  |
| virgo35  | vcpu      | T4        |   1  |   1  |    2 |   4 |    4  |  NO |
 ===========================================================================

The path for the scratch1 space is /var/tmp.

The path for the scratch2 space, when it exists, is /var/tmp2.

Luke, heterogeneous platform

Head node: luke.univ-grenoble-alpes.fr or luke from the bastions.

  • Data processing platform
  • Heterogeneous architecture
  • Constantly evolving
  • One petabyte of local scratch space
  • 10Gbe network
  • Visualisation node
  • Full list of characteristics displayed via the Message Of The Day

Froggy, HPC platform

Head node: froggy.ujf-grenoble.fr or froggy from the bastions.

  • HPC platform
  • 3200 Xeon E5 cores
  • 18 K20m GPUs
  • 90Tb Lustre high performance distributed storage
  • Infiniband FDR non blocking network
  • Visualisation nodes

Infrastructures de stockage

Bettik, high performance distributed storage

  • High performance distributed storage
  • Accessible as local filesystem from Luke and Dahu
  • Filesystem mounted on /bettik
  • User folder and file creation and management
  • Default access rights to be adjusted by the user
  • Usage information on the CiGri grid server under “Bettik BeeGFS Storage”

Bettik infrastructure hardware table

nodemodeltotal memorystorage
bettik-meta1PowerEdge R64048Gb4 * 480Gb metadata SSD
[ + 1 more identical node(s) ]
bettik-meta1PowerEdge R64064Gb4 * 480Gb metadata SSD
[ + 1 more identical node(s) ]
bettik-data1PowerEdge R730xd64Gb73Tb
[ + 3 more identical node(s) ]
bettik-data5PowerEdge R740xd64Gb33Tb
bettik-data6PowerEdge R740xd64Gb95Tb
bettik-data7PowerEdge R740xd64Gb98Tb
bettik-data8PowerEdge R740xd64Gb80Tb
bettik-data9PowerEdge R740xd64Gb95Tb
bettik-data10PowerEdge R740xd64Gb80Tb
bettik-data11PowerEdge R740xd64Gb98Tb
bettik-data12PowerEdge R740xd64Gb80Tb
bettik-data13PowerEdge R740xd64Gb98Tb
bettik-data14PowerEdge R740xd64Gb80Tb
[ + 1 more identical node(s) ]
bettik-data16PowerEdge R740xd64Gb98Tb

Mantis, cloud storage

  • Two storage infrastructures
    • Mantis 1, legacy platform reaching end of life
    • Mantis 2, new platform currently being deployed
  • Cloud block mode storage
  • Accessible from all clusters
  • Access rights management and sharing with other users
  • Accessible from the IDRIS AdaPP machine
  • Usage information on the CiGri grid server under “Mantis iRODS Storage”

Mantis 1 infrastructure hardware table

nodemodeltotal memorystorage
quathPowerEdge R63064Gb700Gb system SSD
quath-icatPowerEdge R43064Gb180Gb system SSD
quath2PowerEdge R510 + MD120024Gb18Tb + 18Tb
quath4PowerEdge R510 + MD120024Gb18Tb + 18Tb
shibo4PowerEdge R730xd32Gb73Tb
shibo5PowerEdge R730xd + MD120032Gb73Tb + 36Tb
shibo6PowerEdge R730xd32Gb73Tb
shibo7PowerEdge R730xd32Gb73Tb
shibo8PowerEdge R740xd32Gb90Tb
shibo9PowerEdge R740xd32Gb90Tb
cargoPowerEdge R730xd64Gb10Tb

Mantis 2 infrastructure hardware table

nodemodeltotal memorystorage
nigel-1PowerEdge R740xd2192Gb250Tb
nigel-2PowerEdge R740xd2192Gb250Tb
nigel-3PowerEdge R740xd2192Gb250Tb