Job workflow

Job workflow management

If you need to run a process outside of a job, for example a workflow manager, do not do it on the cluster head node.

For this usage we provide several dedicated workflow host, dahu-workflow[N], from which you can submit oar jobs on any of the clusters, for example:

$ ssh dahu-workflow1.ciment

        WELCOME TO THE WORKFLOW HOST FOR GRICAD CLUSTERS !   

   ********* You are logged on a secondary dahu frontend ********
   This host is reserved to workflow processes that submit jobs 
   to Dahu nodes. You can also submit jobs on bigfoot or luke 
   with ssh. For example:
       ssh bigfoot oarsub ...

      NOTE: Bigfoot home is mounted into /home-bigfoot

   WARNING: - Other users may create high workloads
            - Workflow hosts are provided in a best-effort way
            - Workflow hosts are susceptible to breakdown
            - Use at your own risks!

   **************************************************************

Such hosts are useful for workflow software, post-processing, or devel tools that need too much resources to run on the offcial frontends which are dedicated to only job submissions and jobs monitoring.

The home directory by default is the one of the Dahu cluster. But you can also access to your Bigfoot’s home directory under the /home-bigfoot mountpoint. The Bettik and Silenus fast storages are available in their usual mountpoints /bettik and /silenus.

Currently available workflow frontends:

  • dahu-workflow1: dual Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz (10 cores) with 32GB of RAM. WARNING: /silenus is available, but only through a slow Ethernet connexion.
  • dahu-workflow2: dual Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz (10 cores) with 64 GB of RAM and a 700GB SSD scratch mounted into /scratch. This host has a fast access to Silenus (Omnipath networking).

Workflow hosts are provided in a best-effort way: we use old recycled servers and we do less monitoring. Such hosts are succeptible to overload (due to you or other users launching heavy tools) or even hardware failures in some extreme circumstances.