sprocket test ​

The Sprocket unit testing framework does not require any modifications to your source WDL, re-uses your run configuration, and everything required to test is done via YAML. The framework has been designed to get up and running with comprehensive unit tests as quickly and easily as possible.

How to write unit tests ​

Unit tests are written in YAML and stored alongside the WDL source.

Where test definitions are located ​

Tests are defined separately from the WDL, in standalone YAML files. These YAML files need to have the same basename as the source WDL document, up to the trailing .wdl extension which should be replaced with either a .yaml or .yml file extension.

These YAML files must be located either "sibling" to their source WDL or nested under a test/ folder in the same directory. For example:

wdl_source/
├── data_structures
│   ├── flag_filter.wdl
│   ├── flag_filter.yml
│   ├── read_group.wdl
│   ├── test
│   │   └── read_group.yaml
│   └── untested_source.wdl
└── other_source

What test YAML looks like ​

The root of each YAML file should be populated with a YAML mapping of exact matches to the names of any WDL tasks or workflows defined in the corresponding WDL source. Each entrypoint (task or workflow) can appear up to once in the root, but may have any number of tests associated with it. Tests are defined as a sequence beneath the entrypoint name where sequence elements are delimited with a hyphen (-) and each test must have a name: <your test name here>. Inputs are defined as a mapping of WDL input names to sequences of values. Inputs must be specified as sequences, even if the sequence only contains one element; if the sequence has multiple elements, they will be combinatorially expanded with all the other input sequences. This is referred to as an "input matrix" and allows for a large number of individual WDL executions to be run as a single test. For example:

bam_to_fastq:
  - name: kitchen sink
    inputs:
      bam:
        - bams/Aligned.sortedByCoord.chr9_chr22.bam
        - bams/test_rnaseq_variant.bam
        - bams/test.bwa_aln_pe.chrY_chrM.bam
      bitwise_filter:
        - include_if_all: "0x0"
          exclude_if_any: "0x900"
          include_if_any: "0x0"
          exclude_if_all: "0x0"
        - include_if_all: "00"
          exclude_if_any: "0x904"
          include_if_any: "3"
          exclude_if_all: "0"
      paired_end:
        - true
        - false
      retain_collated_bam:
        - true
        - false
      append_read_number:
        - true
        - false
      output_singletons:
        - true
        - false
      prefix:
        - kitchen_sink_test

This is a single test named "kitchen sink" which defines an input matrix that gets expanded into 96 individual WDL executions, all of which have a unique set of inputs and run in parallel!

96 executions come from the cartesian product of each input sequence (3*2*2*2*2*2*1=96).

If not otherwise specified, a test is considered successful so long as the entrypoint runs to completion, exits without an error, and all outputs are evaluated successfully. For tasks, the test framework assumes the expected exit code is 0.

Assertions ​

Assertions are available for ensuring more complex expectations than simple "success" are true. These more complex conditions can be specified with an assertions: section for the YAML definition.

The assertions available depend on whether the entrypoint is a workflow or a task.

The should_fail: <boolean> assertion is available for both workflows and tasks. Specifying false doesn't change test behavior, but if it's specified as should_fail: true the execution is expected to fail. For workflows, any failure satisfies the assertion; for tasks, any nonzero exit code satisfies it.

validate_flag_filter: # this is a workflow
  - name: valid FlagFilter
    inputs:
      flags:
        - include_if_all: "3"
          exclude_if_any: "0xF04"
          include_if_any: "03"
          exclude_if_all: "4095"
  - name: invalid FlagFilter
    inputs:
      flags:
        - include_if_all: "3"
          exclude_if_any: "0xF04"
          include_if_any: "03"
          exclude_if_all: "whoops! I'll trigger a fail :("
    assertions:
      should_fail: true

To unit test a fail case for a task, you can either set should_fail: true to expect any nonzero exit code, or assert a specific non-zero exit_code: <integer>. These two assertions are mutually exclusive.

validate_string_is_12bit_int: # this is a task
  - name: valid numbers
    inputs:
      number:
        - "5" # decimal
        - "0x9AF" # hexadecimal
        - "01" # octal
        - "4095" # decimal
  - name: invalid numbers
    inputs:
      number:
        - "0x1000" # too many bits
        - ""
        - string
        - this is not a number
        - "000000000011" # too many bits (interpreted as octal, not binary)
        - "-1" # only unsigned ints are accepted
        - "+1"
    assertions:
      exit_code: 42

Assertions are shared by all executions of a test. In the example above, there are 4 executions defined for the validate_string_is_12bit_int::valid numbers test. This test is considered passed if all 4 of those executions evaluate with an exit code of 0. The invalid numbers test contains 7 executions, and every one of those executions must exit with a code of 42 for the test to be considered a success.

Other assertions available for tasks are the stdout and stderr assertions. Both of these work very similarly; they expect a YAML sequence of strings that are interpreted as regular expressions which should match on the task's STDOUT/STDERR stream. For example:

validate_string_is_12bit_int:
  - name: too big decimal fails
    inputs:
      number:
        - "4096"
        - "10000"
        - "9999"
    assertions:
      exit_code: 42
      stderr:
        - Input number \(.*\) interpreted as decimal
        - But number must be less than 4096!

Output Assertions ​

It is often important to test for specific properties of the outputs for a WDL task or workflow. This is possible via the outputs section of the assertions YAML block. Just like other assertions, output assertions are expected to pass for every execution of a test.

read_group_to_string:
  - name: valid read groups
    inputs:
      read_group:
        - ID: R1
          SM: sampleFoo
        - ID: R1
          SM: sampleFoo
          LB: spaces are allowed in LB
          BC: barcode with a space
    assertions:
      outputs:
        validated_read_group: # this output is of type `String`
          - Contains: R1
          - Contains: sampleFoo

Each key under outputs should be an exact match to an output of the task or workflow. The assertions available for that output are going to depend on the WDL type of the output. It wouldn't make sense to assert whether a WDL Boolean "contains" a string, so Sprocket will emit an informative error if there are any type mismatches between the assertions used and the expected type of each output.

The currently supported basic output assertions are:

• Defined: <boolean>: is the output defined (i.e. any value other than None)?
  • available for all optional WDL types
• BoolEquals: <boolean>: is the output Boolean equal to this value?
• StrEquals: <string>: is the output String equal to this value?
• IntEquals: <integer>: is the output Int equal to this value?
• FloatEquals: <float>: is the output Float equal to this value?
• Contains: <string>: does the output String contain this substring?
• Name: <string>: does the output File or Directory have this exact match basename?
• Length: <unsigned integer>: does the output String, Array, or Map have this length?
• Empty: <boolean>: is the output String, Array, or Map empty?
  • "empty" for String types means length zero

There are also several compound output assertions which can be combined with the above in order to test properties of compound WDL types:

• All: <inner assertion>: applies the inner assertion to all elements of the output Array
• Any: <inner assertion>: applies the inner assertion to elements of the output Array, passing if at least one element succeeds
• First: <inner assertion>: applies the inner assertion to the first element of an output Array
• Last: <inner assertion>: applies the inner assertion to the last element of an output Array
• Left: <inner assertion>: applies the inner assertion to the left element of an output Pair
• Right: <inner assertion>: applies the inner assertion to the right element of an output Pair

An example combining these:

assertions:
  outputs:
    fusion_bams: # this output is of type `Array[File]`
      - Length: 2
      - First: { Name: fusions_1.bam }
      - Last: { Name: fusions_2.bam }

Custom Assertions ​

The above "builtin" assertions can get you pretty far with your testing, but there are often further checks you may want to perform which are outside the scope or ability of the builtins. For these cases, you can use custom assertions which allow test authors to write arbitrary code in Bash scripts or other executables and call them from test YAML. An example may be wanting to verify that a BAM created as a task output can be parsed by samtools quickcheck.

Custom assertions will call Bash scripts (they must be Bash) which are found within the <workspace>/test/bin directory. Other, non-Bash executables may also be placed in the test/bin directory and will be made available to the called Bash script. These scripts will run in the host environment, so external dependencies like samtools are easily accessed.

An example of a custom assertion Bash script calling samtools quickcheck on a newly created BAM could look like:

set -euo pipefail

output_json="$1"
bam_field="$2"

bam="$(jq -r ".$bam_field" "$output_json")"

samtools quickcheck "$bam"

This script would then be called from a custom block in the assertions section of a YAML test. Each key in the block must be the basename of a Bash script in the test/bin directory. The value of that key is an array of strings, where each string is an argument to pass to the script. One of those arguments must be "$OUTPUTS", and that argument will be replaced with a path to the successful task or workflow's generated outputs.json.

assertions:
  custom:
    quickcheck.sh: [ "$OUTPUTS", "duplicate_marked_bam" ]

In the example, the second argument to the script is the field of the output JSON which holds the BAM we want to quick check. We parse the BAM's path out of the supplied JSON using jq and then run samtools on the new file.

Calling other executables from Bash ​

Bash is great for its portability, but it's often not the language we want to write our tests in. You may want to write a Python script that uses the pysam module to do even more advanced checks on your output BAM files. We'll leave the actual Python to you as an exercise, but let's assume you have a Python script called check_readgroups.py which does something more advanced than what we can easily do in Bash.

You can store check_readgroups.py in the test/bin directory alongside your Bash scripts, and it will be made available at runtime using the Bash environment variable $TEST_BIN. To call it, we may write a Bash script that looks like:

set -euo pipefail

output_json="$1"
output_field="$2"
script_name="$3"

output="$(jq -r ".$output_field" "$output_json")"

python3 "$TEST_BIN/$script_name" "$output"

Then our test YAML could call this pair of executables with:

assertions:
  custom:
    quickcheck.sh: [ "$OUTPUTS", "duplicate_marked_bam" ]
    call_python.bash: [ "$OUTPUTS", "duplicate_marked_bam", "check_readgroups.py" ]

Debugging custom assertions ​

While writing scripts, it's often helpful to be able to inspect the STDOUT and STDERR of what is being run. To view these streams during test execution, you can pass the --show-assertion-stdio flag to test.

To prevent littering your filesystem with temporary files, we run custom assertions inside a temp directory that will be deleted when the custom executables are done running. This may change in future releases.

Enabling custom assertions ​

By default, Sprocket will ignore any custom assertions and skip over them when executing tests. This is to make sure that Sprocket never runs potentially malicious code without your instruction. Custom assertions can be enabled either with the --enable-custom-assertions flag at the command line or in your sprocket.toml in the [test] table with enable_custom_assertions = true.

Blessing executables ​

In addition to enabling custom assertions, all files in the test/bin directory must be "blessed" to run on your host. After finalizing your bin directory, you can run sprocket test bless-executables to store the name and exact hash of every file in that directory in a special blessed_executables.json file. If Sprocket detects any changes to your bin directory that haven't been blessed, no user code will run and the tests will fail. If all the hashes match, they will run as described above. This process prevents running changed files; it does not ensure a file is safe to execute. That remains the user's responsibility.

To ease the development process while writing new tests, you can pass the --trusted flag to the test command. This will bypass the bless check and let anything in test/bin run on your host.

How to use test fixtures ​

WDL tasks and workflows often take file inputs, so the Sprocket unit testing framework has attempted to make it as easy as possible to reference "test fixtures" from test YAML.

Fixtures are located relative to a WDL workspace's root, at <workspace>/test/fixtures/. A WDL workspace is very similar to a git repository (a directory can be both a WDL workspace and a git repository), in that it is simply a directory on your local filesystem with a collection of WDL documents within it. These WDL documents can be arbitrarily nested and Sprocket will recursively search the workspace for all files with a .wdl extension and an associated YAML file. The test/fixtures/ directory will always be loaded as the "origin" from which all files are expected to be located. This means that your test YAML can live near the WDL source it is testing, but test fixtures remain isolated relative to the workspace root, preventing clutter in your source directories.

Let's return to the "kitchen sink" example test defined earlier: one of the inputs sections looks like:

      bam:
        - bams/Aligned.sortedByCoord.chr9_chr22.bam
        - bams/test_rnaseq_variant.bam
        - bams/test.bwa_aln_pe.chrY_chrM.bam

Regardless of where this YAML file is located in the workspace, the 3 BAM files referenced will be found at:

<workspace>/test/fixtures/bams/Aligned.sortedByCoord.chr9_chr22.bam
<workspace>/test/fixtures/bams/test_rnaseq_variant.bam
<workspace>/test/fixtures/bams/test.bwa_aln_pe.chrY_chrM.bam

This makes re-using test fixtures across your workspace easy! Any YAML file in the workspace can reference the same fixtures using the same paths; even if the source WDL moves around, your tests will remain valid.

Specifying groups of inputs ​

Often in bioinformatics, files have separate accessory files that are specific to each other. If these files are provided as distinct inputs, the combinatorial nature of input matrices may mix up these files and cause unintended errors. For example, the following test definition would not produce a useful test matrix:

bam_coverage:
  - name: this will not work
    inputs:
      bam:
        - bams/test.bam
        - bams/test.bwa_aln_pe.chrY_chrM.bam
        - bams/Aligned.sortedByCoord.chr9_chr22.bam
        - bams/test_rnaseq_variant.bam
      bam_index:
        - bams/test.bam.bai
        - bams/test.bwa_aln_pe.chrY_chrM.bam.bai
        - bams/Aligned.sortedByCoord.chr9_chr22.bam.bai
        - bams/test_rnaseq_variant.bam.bai
      prefix:
        - test_bigwig

The problem is that in addition to the correct inputs we want to test (e.g. bam=bams/test.bam, bam_index=bams/test.bam.bai, and prefix=test_bigwig) there will be incorrect file combinations that will fail (e.g. bam=test_rnaseq_variant.bam, bam_index=test.bwa_aln_pe.chrY_chrM.bam.bai, and prefix=test_bigwig).

Groups of inputs that should be combined exactly can be specified with a special syntax that involves nesting the input keys under a key starting with a dollar sign ($). The identifier that follows the dollar sign is unimportant and can be any non-empty string. To fix our test definition, we could re-write to look like:

bam_coverage:
  - name: this will work
    inputs:
      $samples:
        bam:
          - bams/test.bam
          - bams/test.bwa_aln_pe.chrY_chrM.bam
          - bams/Aligned.sortedByCoord.chr9_chr22.bam
          - bams/test_rnaseq_variant.bam
        bam_index:
          - bams/test.bam.bai
          - bams/test.bwa_aln_pe.chrY_chrM.bam.bai
          - bams/Aligned.sortedByCoord.chr9_chr22.bam.bai
          - bams/test_rnaseq_variant.bam.bai
      prefix:
        - test_bigwig

Now, instead of creating a matrix with 16 executions (12 of which would fail), we are only defining 4 executions, all of which should succeed.

Tagging tests ​

It's often helpful to run only a subset of tests defined in a workspace, either by excluding slow tests or by only running the tests you are actively developing. Each test can include a sequence of tags that can be toggled from the command line when executing tests. For example:

build_bwa_db:
  - name: build reference
    tags: [ reference, slow ]
    inputs:
      reference_fasta:
        - reference/GRCh38.chrY_chrM.fa

Running unit tests ​

Choosing which tests run ​

Use --target (-t) to run only tests whose task or workflow name contains a substring, and --filter (-f) to run only tests whose test name contains a substring. Pass --exact to match these filters exactly rather than by substring. These combine with the tag-based selection described above.

The --include-tag (-i) and --exclude-tag (-e) flags can also be used to choose which tests run.

Investigating tests ​

By default, Sprocket will clean up all the files created by successful test executions. The individual executions which fail their assertions will remain on disk to enable investigating why the test did not match expectations. If you instead want Sprocket to remove all test files at the end of a run, you can use the --clean-all flag. Conversely, if you wish to keep all files created, including those created by successful tests, you can use the --no-clean flag.

Generating a test schema ​

sprocket test schema prints a JSON schema for Sprocket test definition YAMLs to standard out. You can use this to validate test definition files or to power editor autocompletion.

sprocket test schema > test.schema.json