LSF + Apptainer backend

Sprocket contains a High-Performance Computing (HPC) backend targeting environments that use LSF 10.1.0 or later for job scheduling and Apptainer 1.3.6 or later as a container runtime.

For a step-by-step walkthrough of setting up Sprocket on an LSF cluster, see the LSF + Apptainer guide.

To execute WDL workflows and tasks using the LSF + Apptainer backend, you must be running Sprocket on a Linux system with the LSF command-line tools available locally. The nodes where LSF dispatches jobs must have the Apptainer command-line tools available. With these prerequisites met, add configuration like the following example to your sprocket.toml to execute tasks using the HPC:

# Set the default backend to LSF + Apptainer.
[run.backends.default]
type = "lsf_apptainer"

# The LSF queue used by default for task execution.
#
# This parameter is optional. If it's absent and no other applicable queues
# are specified, jobs will be submitted to your LSF cluster's default queue.
# default_lsf_queue.name = "standard"
# The largest number of CPUs and memory that can be reserved for a single job
# on this queue.
#
# These parameters are optional, and should be set according to site-specific
# information about the hosts available to dispatch work from the queue. They
# can also be set for the other types of queues, but this example leaves them
# unconstrained by default.
default_lsf_queue.max_cpu_per_task = 64
default_lsf_queue.max_memory_per_task = "96 GB"

# The LSF queue used for short tasks.
#
# This parameter is optional, and overrides `default_lsf_queue`.
short_task_lsf_queue.name = "short"

# The LSF queue used for GPU tasks.
#
# This parameter is optional, and overrides `default_lsf_queue` and
# `short_task_lsf_queue`.
gpu_lsf_queue.name = "gpu"

# The LSF queue used for FPGA tasks.
#
# This parameter is optional, and overrides `default_lsf_queue` and
# `short_task_lsf_queue`.
fpga_lsf_queue.name = "fpga"

# The maximum number of concurrent `bsub` processes the backend will spawn to
# queue tasks. Defaults to `10`. Consider raising this for large-scale
# workflow execution.
max_concurrency = 10

# Prefix added to every LSF job name. Useful for identifying Sprocket jobs
# in `bjobs` output (e.g., `bjobs -J "sprocket*"`).
job_name_prefix = "sprocket"

# Task monitor polling interval in seconds. Defaults to `30`.
interval = 30

# Settings related to `bsub`.
[run.backends.default.bsub]
# Additional command-line arguments to pass to `bsub` when submitting jobs to
# LSF.
args = ["-app", "my_app_profile"]

# Settings related to `apptainer`.
[run.backends.default.apptainer]
# Additional command-line arguments to pass to `apptainer exec` when executing
# tasks.
extra_args = ["--hostname=\"my_host\""]

# Path to the Apptainer (or Singularity) executable. Defaults to `"apptainer"`.
# Set to `"singularity"` or a full path if the executable is not on `PATH`.
executable = "apptainer"

# Shared directory for caching pulled `.sif` images across runs. When unset,
# images are stored per-run and not shared.
image_cache_dir = "/shared/containers/cache"

If you run into problems or have other feedback, please reach out to us in the #sprocket channel on the WDL Slack.

Conditional bsub Arguments

The LSF + Apptainer backend supports conditional arguments to bsub when queuing a task.

The bsub.conditional setting of the backend is a collection of conditional arguments and may be specified using the array of tables syntax, like so:

[[run.backends.default.bsub.conditional]]
condition = "<expr>"
args = ["<arg>", "..."]

The condition is a WDL expression that is evaluated in the context of an individual task and must be of type Boolean.

If the expression evaluates to true, the provided arguments (via the args array) are applied to the bsub command when queuing the task.

Conditional arguments are evaluated in the order defined in the configuration and only the first conditional argument whose expression evaluates to true is applied to the bsub command.

The condition WDL expression has access to the following variables relating to the task being evaluated:

Name	Type	Description
cpu	Float	cpu requirement or default (1) if not present
memory	Int	memory requirement (in bytes) or default (2 GiB) if not present
gpu	Boolean	gpu requirement or default (false) if not present
fpga	Boolean	fpga requirement or default (false) if not present
disks	Int	the sum of all disk requirements (in bytes) or default (1 GiB) if not present
hint	Object	the task's hints section; note: accessing a non-existent key for the object will result in a None value rather than an evaluation error

The expression is statically analyzed when configuration is loaded to catch syntax errors and most type errors prior to evaluating tasks.

Example

An example conditional argument that applies a fictitious bsub argument when a task requests at least 64 CPUs and 16 GiB of memory:

[[run.backends.default.bsub.conditional]]
condition = "cpu >= 64 && memory >= 17179869184"
args = ["--some-bsub-arg"]

Known issues

• Much of the error reporting is based on best-effort inspection of the output of CLI tools and dumping output to files. Error messages presented at the command line will likely be less informative than inspecting the various output files left behind in the runs directory after a failure.

• There are only basic controls and limits applied to LSF jobs: scatter concurrency factor, CPU per task, and memory per task. This has a couple impacts worth noting:

  • The backend may attempt to schedule tasks which will be forever pending due to an unsatisfiable CPU or memory request. Using LSF tooling such as bjobs can help identify when a task is pending for an inordinate amount of time.
  • Too-high scatter concurrency or other task count explosions may overwhelm the LSF queues and cause impact to the cluster depending on how it's administered. Raise this setting with caution, and note that each scattered task may itself request multiple CPUs, which LSF may see as multiple tasks.

• Development and testing of this backend has taken place on a single HPC cluster with specific versions and configurations of the relevant tools. It is likely that other configurations will behave slightly differently. Reports of these types of issues are greatly appreciated in #sprocket channel on the WDL Slack.