Sprocket. The Bioinformatics Workflow Engine.

Sprocket is a high-performance, modern, and open-source workflow engine for bioinformatics. Create, test, and run your analyses locally, then seamlessly move to HPC or the cloud to handle thousands of parallel workflows.

Explore documentation Join us on Slack

Installbash
```
brew install sprocket
```
Installs with Homebrew. Get Homebrew All options

version 1.3

task say_hello {
    input {
        String greeting
    }

    command <<<
        echo "~{greeting}, world!"
    >>>

    output {
        String out = read_string(stdout())
    }

    requirements {
        container: "ubuntu:latest"
    }
}

Runbash
$ sprocket run example.wdl --target say_hello greeting="Hello"
{
"say_hello.out": "Hello, world!"
}
$
```
$ sprocket run example.wdl --target say_hello greeting="Hello"
{
  "say_hello.out": "Hello, world!"
}
```

Check

Validate every document before it runs.

sprocket lint points to the exact line of each problem and suggests a fix. Invalid WDL exits with a non-zero code, so the same check can guard your CI, and sprocket format keeps every file in one style.

Read about check and lint

Terminal

$ sprocket lint example.wdl
note[ContainerUri]: container URI uses a mutable tag
   ┌─ example.wdl:18:20
   │
18 │         container: "ubuntu:latest"
   │                    ^^^^^^^^^^^^^^^
   │
   = fix: replace the mutable tag with its SHA256 equivalent

warning[UnusedInput]: unused input `color`
   ┌─ example.wdl:30:16
   │
30 │         String color = "green"
   │                ^^^^^

Edit
See the same checks while you write.
The Visual Studio Code extension runs sprocket analyzer, Sprocket's language server, and lists validation and lint problems as you edit. It also adds WDL syntax highlighting and snippets. For Neovim, use sprocket.nvim.
Set up the editor extension
Problems
example.wdl~/Desktop4 problems
- unused input `color`Sprocket(UnusedInput)[Ln 30, Col 16]
- task `say_hello` is missing both `meta` and `parameter_meta` sectionsSprocket(MetaSections)[Ln 3, Col 6]
  example.wdl[Ln 3, Col 6]: fix: add both the `meta` and `parameter_meta` sections
- container URI uses a mutable tagSprocket(ContainerUri)[Ln 18, Col 20]
  example.wdl[Ln 18, Col 20]: fix: replace the mutable tag with its SHA256 equivalent (e.g., `ubuntu@sha256:foobar` instead of `ubuntu:latest`)
- workflow `main` is missing both `meta` and `parameter_meta` sectionsSprocket(MetaSections)[Ln 22, Col 10]
  example.wdl[Ln 22, Col 10]: fix: add both the `meta` and `parameter_meta` sections
Run locally
Develop on your own machine first.
Write, run, and debug a workflow on your laptop before it goes anywhere near a cluster. sprocket inputs writes a template of the inputs a workflow expects, sprocket validate checks your values, and sprocket run executes it right there with the local Docker backend.
- macOS
- Linux
- Windows
Take the guided tour
TerminalYour machine · Docker
```
$ sprocket run example.wdl --target main name="World"
{
  "main.messages": [
    "Hello, World!",
    "Hallo, World!",
    "Hej, World!"
  ]
}
```

Scale

Move to a cluster by changing configuration, not code.

Point sprocket.toml at a Slurm or LSF cluster with Apptainer, or at a Task Execution Service (TES) server. Sprocket moves inputs and outputs to and from cloud storage as the workflow needs them. It is built toward a target of 20,000+ concurrent jobs.

Amazon S3
Azure Blob Storage
Google Cloud Storage

Configure a backend

sprocket.toml

# Run every task in a local Docker container.
[run.backends.default]
type = "docker"

# Set the default backend to Slurm + Apptainer.
[run.backends.default]
type = "slurm_apptainer"

# The Slurm partition used by default for task execution.
default_slurm_partition.name = "gpu"
default_slurm_partition.max_cpu_per_task = 64
default_slurm_partition.max_memory_per_task = "96 GB"

The Slurm + Apptainer backend is experimental.

# Set the default backend to LSF + Apptainer.
[run.backends.default]
type = "lsf_apptainer"

# The LSF queue used by default for task execution.
default_lsf_queue.name = "standard"
default_lsf_queue.max_cpu_per_task = 64
default_lsf_queue.max_memory_per_task = "96 GB"

The LSF + Apptainer backend is experimental.

[run.backends.default]
type = "tes"
# The URL of the TES API server
url = "<tes-server-url>"
# The cloud storage URL where Sprocket will upload inputs
inputs = "<cloud-storage-url>"
# The cloud storage URL where the TES API server will upload outputs
outputs = "<cloud-storage-url>"

Reproduce

Every run leaves a complete record.

Each run gets its own directory with its inputs, outputs, logs, and every task attempt, and a SQLite database keeps the history. Turn on the call cache and a rerun after a failure reuses the tasks that already succeeded.

How provenance tracking works

./out

./out/
├── sprocket.db              # SQLite provenance database
├── runs/
│   └── main/
│       ├── <timestamp>/
│       │   ├── inputs.json
│       │   ├── outputs.json
│       │   ├── output.log
│       │   └── calls/        # One directory per task call
│       └── _latest -> <timestamp>/
└── index/                   # Optional output indexing

sprocket.toml

[run.task]
cache = "on"

Extend

Use our supporting tools, or build your own.

Lint every pull request with the Sprocket GitHub Action. To build your own tooling, analyze WDL from Python with sprocket-bio, use the wdl crate in Rust, or submit runs to the experimental Sprocket server over its REST API.

Start with the Python bindings

.github/workflows/wdl.yml

steps:
  # Lint every WDL document in the repository.
  - uses: stjude-rust-labs/sprocket-action@main
    with:
      action: lint
      except: TrailingComma,ContainerUri

# Analyze WDL documents from Python 3.10 or later
$ pip install sprocket-bio

# Use Sprocket's WDL crate in your own Rust tools
$ cargo add wdl

# Start a server that accepts workflows from a directory
$ sprocket dev server start \
    --allowed-file-paths /home/user/workflows \
    --port 8080

# Submit a run from anything that speaks HTTP
$ curl -X POST http://localhost:8080/api/v1/runs \
    -H "Content-Type: application/json" \
    -d '{"source": "/home/user/workflows/hello.wdl", "inputs": {"name": "World"}}'

The Sprocket server is experimental and may change.

In the animation, Sprocket, a workflow engine built on WDL, is the head unit. Below it sits Crankshaft, a language-agnostic execution engine that handles the submission, execution, and reporting of tasks. Head units for other languages, such as CWL, Nextflow, and Snakemake, can drive it too.

Join Our Community

Connect with fellow researchers and developers, stay updated on the latest features, and share ideas to push the boundaries of bioinformatics.

Join us on Slack

Sprocket. The Bioinformatics Workflow Engine.

One tool from first line to full scale

Check

Edit

Run locally

Scale

Reproduce

Extend

How Sprocket Powers Your Workflows

Join Our Community