Defining host templates

You can create your own host templates in addition to the ones shipped with the MDBenchmark. We use the jinja2 Python package for these host templates. Please refer to the official Jinja2 documentation for further information on formatting and functionality.

To be detected automatically, a template file must have the same filename as returned by the UNIX command hostname. If this is not the case, you can point MDBenchmark to a specific template by providing its name via the --host option.

Assuming you created a new host template in your home directory ~/.config/MDBenchmark/my_custom_hostfile:

mdbenchmark generate --host my_custom_hostfile

Sun Grid Engine (SGE)

This example shows a HPC running SGE with 30 CPUs per node.

#!/bin/bash
### join stdout and stderr
#$ -j y
### change to currend work dir
#$ -cwd
#$ -N {{ job_name }}
# Number of nodes and MPI tasks per node:
#$ -pe impi_hydra {{ 30 * n_nodes }}
#$ -l h_rt={{ formatted_time }}

module unload gromacs
module load {{ module }}
module load impi
module load cuda

# The below configuration was kindly provided by Dr. Klaus Reuter
# --- edit the following two lines to configure your job ---
{% if hyperthreading %}
USE_HT=1                                # use hyperthreading, 0 (off) or 1 (on)
{%- else %}
USE_HT=0                                # use hyperthreading, 0 (off) or 1 (on)
{%- endif %
N_TASKS_PER_HOST={{ number_of_ranks }}  # number of MPI tasks to be started per node


# --- no need to touch the lines below ---
N_SLOTS_TOTAL=$NSLOTS
N_TASKS_TOTAL=$((N_TASKS_PER_HOST*NHOSTS))
N_SLOTS_PER_HOST=$((NSLOTS/NHOSTS))
N_THREADS_PER_PROCESS=$((N_SLOTS_PER_HOST/N_TASKS_PER_HOST))
N_THREADS_PER_PROCESS=$(((1+USE_HT)*N_THREADS_PER_PROCESS))
export OMP_NUM_THREADS=$N_THREADS_PER_PROCESS
if [ $USE_HT ]; then
    export OMP_PLACES=threads
else
    export OMP_PLACES=cores
fi

# Edit again below, as you see fit

# Run gromacs/{{ version }} for {{ time - 5 }} minutes
mpiexec -n $N_TASKS_TOTAL -ppn $N_TASKS_PER_HOST mdrun_mpi -ntomp $N_THREADS_PER_PROCESS -v -maxh {{ time / 60 }} -resethway -noconfout -deffnm {{ name }}

Slurm

The following showcases the job template for the MPCDF cluster draco using Slurm.

#!/bin/bash -l
# Standard output and error:
#SBATCH -o ./{{ job_name }}.out.%j
#SBATCH -e ./{{ job_name }}.err.%j
# Initial working directory:
#SBATCH -D ./
# Job Name:
#SBATCH -J {{ job_name }}
#
# Queue (Partition):
{%- if gpu %}
#SBATCH --partition=gpu
#SBATCH --constraint='gpu'
{%- else %}
{%- if time is lessthan 30 or time is equalto 30 %}
#SBATCH --partition=express
{%- elif time is greaterthan 30 and time is lessthan 240 or time is equalto 240 %}
#SBATCH --partition=short
{%- else %}
#SBATCH --partition=general
{%- endif %}
{%- endif %}
#
# Request {{ n_nodes }} node(s)
#SBATCH --nodes={{ n_nodes }}
# Set the number of tasks per node (=MPI ranks)
#SBATCH --ntasks-per-node={{ number_of_ranks }}
# Set the number of threads per rank (=OpenMP threads)
#SBATCH --cpus-per-task={{ number_of_threads }}
{% if hyperthreading %}
# Enable hyperthreading
#SBATCH --ntasks-per-core=2
{%- endif %}
# Wall clock limit:
#SBATCH --time={{ formatted_time }}

module purge
module load impi
module load cuda
module load {{ module }}

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
{% if hyperthreading %}
export OMP_PLACES=threads
export SLURM_HINT=multithread
{%- else %}
export OMP_PLACES=cores
{%- endif %}

# Run {{ module }} for {{ time  }} minutes
srun gmx_mpi mdrun -v -ntomp $OMP_NUM_THREADS -maxh {{ time / 60 }} -resethway -noconfout -deffnm {{ name }}

# Running multiple simulations on a single node (multidir)
# If you want to run multiple simulations on the same node, use the `multidir`
# variable, like so:
#
# srun gmx_mpi mdrun -v -ntomp $OMP_NUM_THREADS -maxh {{ time / 60 }} -resethway -noconfout -deffnm {{ name }} {{ multidir }}
#
# MDBenchmark will set up the folder structure as required by GROMACS and
# replace the variable.

LoadLeveler

Here is an example job template for the decomissioned MPG cluster hydra (LoadLeveler).

# @ shell=/bin/bash
#
# @ error = {{ job_name }}.err.$(jobid)
# @ output = {{ job_name }}.out.$(jobid)
# @ job_type = parallel
# @ node_usage = not_shared
# @ node = {{ n_nodes }}
# @ tasks_per_node = {{ number_of_threads }}
{%- if gpu %}
# @ requirements = (Feature=="gpu")
{%- endif %}
# @ resources = ConsumableCpus(1)
# @ network.MPI = sn_all,not_shared,us
# @ wall_clock_limit = {{ formatted_time }}
# @ queue

module purge
module load {{ module }}

# run {{ module }} for {{ time }} minutes
poe gmx_mpi mdrun -deffnm {{ name }} -maxh {{ time / 60 }} -resethway -noconfout

Options passed to job templates

MDBenchmark passes the following variables to each template:

Value Description
name Name of the TPR file
job_name Job name as specified by the user, if not specified same as name
gpu Boolean that is true, if GPUs are requested
module Name of the module to load
n_nodes Maximal number of nodes to run on
number_of_ranks The number of MPI ranks
number_of_threads The number of OpenMP threads
hyperthreading Whether to use hyperthreading
time Benchmark run time in minutes
formatted_time Run time for the queuing system in human readable format (HH:MM:SS)
multidir Run multiple simulations on a single node (GROMACS only)

To ensure correct termination of jobs formatted_time is 5 minutes longer than time.

MDBenchmark will look for user templates in the xdg config folders defined by the environment variables XDG_CONFIG_HOME and XDG_CONFIG_DIRS which by default are set to $HOME/.config/MDBenchmark and /etc/xdg/MDBenchmark, respectively. If the variable MDBENCHMARK_TEMPLATES is set, the script will also search in that directory.

MDBenchmark will first search in XDG_CONFIG_HOME and XDG_CONFIG_DIRS for a suitable template file. This means it is possible to overwrite system-wide installed templates or templates shipped with the package.