Software:Snakemake

Snakemake

Repository	github.com/snakemake/snakemake
Written in	Python, YAML
Platform	Windows, Linux, macOS
License	MIT License
Website	snakemake.github.io

Snakemake is a scientific workflow management framework that uses a domain-specific language (DSL) to specify allowed data transformations for automatic generation of processing pipelines. Subscribing to the declarative paradigm, usage invloves stating desired products instead of an explicit course of action. Individual steps, however, must still be defined as rules through syntax that is inspired from that of GNU Make. Since Snakemake's DSL is implimented as an extension of python, these rules are written in a Snakefile using YAML and python syntax. Execution of the generated pipeline is done by selecting and running each step as prerquisites become satisfied. Steps with no interdependence may be executed in parallel.

The goal of Snakemake is to enhance reproducibility, transparency, and adaptability in bioinformatic analysis. These goals manifest in the following characteristics: Native integration of external tools like Conda and Singularity borrow their functionalities, namely dependency resolution and containerization. Being platform agnostic allows it to run the same workflow on local or high performance compute (HPC) environments. Reports can be generated to describe and log the conditions and actions performed at each step.

Operation

Rules

The set of actions that Snakemake can take towards generating a desired data product is specified in a file called the Snakefile. These actions are called "rules" and include a name, inputs, outputs, and how to start the represented action. Inputs and outputs are declared as explicit file paths, cloud storage, or sets of locations through patterns and wildcards. Several methods exist to start an action, including a shell command, python script, or Snakemake wrapper. A wrapper is a predefined configuration that allows Snakemake to directly use prevalent software such as Samtools. In the Snakefile, YAML adhereing to Snakemake's DSL and native python appears in the syntax, but upon execution, the rules are translated into python via a finite automation.

Example workflow

Example rule:

rule user_defined_name:
    input:
        "path/to/input.file"
    output:
        "path/to/output.file"
    shell:
        "shell_command {input} > {output}"

Workflows and Execution

When given a set of rules and target outputs, Snakemake will first either look for files that match the targets or search for defined rules that produce them. By then redefining the targets to the inputs of found rules and repeating recursively until all inputs are existing files, an implicit workflow is generated in the form of a directed acycic graph (DAG) of steps. The execution order of these steps is computed by optimizing a mixed integer linear program for various parameters such as runtime, total disk space required for temporary files, and user defined priority. Where possible, intermediate data product from previous runs can be used to skip ahead if prerequisite steps are found to be equivalent. Equivalence is determined by comparing the prior steps used to generate the data, recorded through a ledger system analogous to that of block chains.

Goals

Example Snakefile and tool wrapper reuse

Reproducibility

A cornerstone of science is the need to validate experimental results so that confidence can be gained on the result's correctness. Snakemake utilizes Conda to reconstruct the compute environment with required software dependencies and supports containerization via Docker or Singularity. Reports can also be generated to summarize the various parameters and events at each processing step, including an image of the computed DAG.

Transparency

Information gained from science must be shared so that findings can be built upon and utilized, therefore the experiment and results must be comprehensible. In addition to the report generation mentioned above, the developers also claim that a typical Snakefile contains a high proportion of commonly understood terms rather than technical or Snakemake specific language. This supports transparency by reducing the amount of domain knowledge required to understand what an experiment did and what the results mean.

Adaptability

Experimental methods themselves can contribute to science if they can be repurposed, in whole or in part, to new research. To this end, previously written Snakefiles can be referenced in new Snakefiles, thus allowing rules to be reused. Certain rules, called tool wrappers, fully integrate external software as a set of possible steps for workflow generation and are publicly availible.

Comparisons

Snakemake belongs to a family of implicit convention frameworks, members of which include Make and Nextflow. The common characteristic of this group of tools is the ability to generate specific workflows from a collection of discrete steps. Specifically, the GNU derivative of Make used for building source code, is considered to be the source of inspiration for Snakemake and contribute to its model of operation and certain features like wildcards. Computationally, Snakemake was found to have a competitve runtime but failed to consistently utilize the CPU while allowing excess context switching. Usage wise, Snakemake opts for text-based configuration instead of a GUI, like that of Galaxy, and attempts to be environment agnostic.

Usage Examples

Bioinformatics

• SnakeChunks is a set of Snakemake rules used for processing Next-Generation Sequencing data, including quality control, peak calling, and visualization steps. The authors demonstrated its use on transcriptome (RNA-seq) and genome wide location (ChIP-seq) data.
• Recount3 is a database of human and mouse RNA-seq data that was curated and annotated with a Snakemake based workflow.
• VIPER is a snakemake based pipeline for producing graphical reports of RNA-seq data.
• RiboDoc is a ribosomal profiling package that also involves complex steps of quality control and data-intensive statistical analysis managed by Snakemake and Docker.
• Dadasnake integrates amplicon sequencing analysis with DADA2 using Snakemake. The implied workflow is for phylogenetic analysis of specific genes.
• In order to compile timelapses of selective plane illumination microscopy images of cells, Snakemake was used to parallelize the task on high performanc compute clusters.

Other

• HEXME is a dataset of tetrahedral meshes generated from CAD models, using Snakemake. Rules were specified to generate three variants for each input.