Guided Tour
In this guide, we'll cover how to write, verify, and run WDL documents using Sprocket. This should give you a good sense of where and how you might use Sprocket in your day-to-day work.
To follow along, you are encouraged to download this WDL document that has been specifically crafted for this walkthrough. Further, if you have not already, please follow this guide to install Sprocket.
Ensuring high-quality code
Using automation to ensure high-quality code both during development and through continuous integration (CI) processes is one of the hallmarks of modern development. Sprocket contains both the check
and lint
(shortcut for check --lint
) subcommands to help you during your WDL development.
Exploring lint and validation rules
You can see a list of all of the rules Sprocket contains by running sprocket explain -h
. You can dive into any of the rules Sprocket includes by running sprocket explain <RULE>
. For example, if we wanted to learn more about the ImportSorted
rule, we could do the following.
sprocket explain ImportSorted
This command gives the following description of the ImportSorted
rule.
ImportSorted [Style, Clarity, Sorting]
Ensures that imports are sorted lexicographically.
Imports should be sorted lexicographically to make it easier to find
specific imports. This rule ensures that imports are sorted in a
consistent manner. Specifically, the desired sort can be acheived
with a GNU compliant `sort` and `LC_COLLATE=C`. No comments are
permitted within an import statement.
We encourage you to explore the existing validation and linting rules supported by Sprocket along with suggesting new helpful rules on our issues page.
Linting and validation
Both single WDL documents and directories of WDL documents can be validated and linted by using the sprocket lint
subcommand.
sprocket lint example.wdl
This returns a set of validation and linting diagnostics that can/should be addressed by the workflow author. In this case, the following is an abbreviated output from the linting and validation process on example.wdl
.
note[ContainerUri]: container URI uses a mutable tag
┌─ example.wdl:18:20
│
18 │ container: "ubuntu:latest"
│ ^^^^^^^^^^^^^^^
│
= fix: replace the mutable tag with its SHA256 equivalent (e.g., `ubuntu@sha256:foobar` instead of `ubuntu:latest`)
note[MetaSections]: workflow `main` is missing both `meta` and `parameter_meta` sections
┌─ example.wdl:22:10
│
22 │ workflow main {
│ ^^^^ this workflow is missing both `meta` and `parameter_meta` sections
│
= fix: add both the `meta` and `parameter_meta` sections
warning[UnusedInput]: unused input `color`
┌─ example.wdl:35:16
│
35 │ String color = "green"
│ ^^^^^
Specific lint rules can be ignored with multiple invocations of the -e
flag.
sprocket lint example.wdl -e ContainerUri -e MetaSections
This leaves a single diagnostic, which is that color
is an unused workflow input. Before continuing on, we will remove that input from the main
workflow so it doesn't continue showing up in the diagnostics.
Continuous integration
If you use GitHub for source control, you can use the Sprocket GitHub Action to ensure that your WDL documents stay formatted correctly and free of validation/lint errors.
Code editor integration
Rather than running sprocket check
and sprocket lint
continually on the command line, most developers prefer to have these errors and lints show up in their editor of choice. Sprocket makes available a language server protocol (LSP) server under the sprocket analyzer
command.
If you use Visual Studio Code, you can easily get started by simply installing the Sprocket VSCode extension. This automatically downloads the latest version of sprocket
and integrates the various lint and validation warnings into the "Problems" tab of your editor.
Running tasks and workflows
Individual tasks and workflows can be run with the sprocket run
subcommand. By default, the workflow in the document is designated as the target for sprocket run
.
sprocket run example.wdl
After a few seconds, you'll see sprocket
return an error.
error: failed to validate the inputs to workflow `main`
Caused by:
missing required input `name` to workflow `main`
This is because the workflow has a required input parameter called name
that must be provided before the workflow can run.
Understanding inputs
Inputs to a Sprocket run are provided as arguments passed after the WDL document name is provided. Each input can be specified as either
- a key value pair (e.g.,
workflow.foo="bar"
), or - a file containing inputs within JSON (e.g., a
defaults.json
file where the contents are{ "workflow.foo": "bar" }
).
Inputs are incrementally applied, meaning that inputs specified later override inputs specified earlier. This enables you to do something like the following to use a set of default parameters and iterate through sample names in Bash rather than create many individual JSON input files.
sprocket run workflow.wdl defaults.json main.sample_name="Foo"
Here, we can specify the name
parameter as a key-value pair on the command line.
sprocket run example.wdl main.name="World"
After a few seconds, this job runs successfully with the following outputs.
{
"messages": [
"Hello, World!",
"Hallo, World!",
"Hej, World!"
]
}
Congrats on your first successful sprocket run
🎉!
If you wanted to override the greetings
workflow, you could do so by defining the input in a greetings.json
file:
{
"main.greetings": [
"Good morning",
"Good afternoon",
"Good evening"
]
}
and the providing that in the set of inputs to the workflow.
sprocket run example.wdl greetings.json main.name="Sprocket" --overwrite
Notably, the --overwrite
option is now provided to let sprocket
know you're okay with overwriting the results of your last workflow run. This produces the following output.
{
"messages": [
"Good morning, Sprocket!",
"Good afternoon, Sprocket!",
"Good evening, Sprocket!"
]
}
Conclusion
You should now have a clear idea on how the most commonly used commands within Sprocket work. With further questions or feature requests, we ask that you join the #sprocket channel on the WDL Slack or file an issue on the Sprocket repository.