Makeflow Feature: JX Representation

There are a number of neat new features in the latest versions of our software that I would like to highlight through some occasional blog posts.  If these sound interesting, please give them a try and send us your feedback.

First, I would like to highlight recent work by Tim Shaffer on JX, a new encoding for Makeflow that makes it easier to express complex workflows programmatically.  

For example, a traditional makeflow rule looks like this:

out.txt: in.txt calib.dat simulate.exe

    simulate.exe -i in.txt -p 10 > out.txt

In the latest version of Makeflow, you can write the same rule in JSON like this:

{

    "command" : "simulate.exe -i in.txt -p 10 > out.txt",

    "inputs" : [ "in.txt", "calib.dat", "simulate.exe" ],

    "outputs": [ "out.txt" ]

}

Now, just using JSON by itself doesn't give you a whole lot.  However, we extended JSON with a few new features like list comprehensions, variables substitutions, and operators.  This gives us a programmable way of generating a lot of rules easily.

For example, this represents 100 rules where the parameter varies from 0-99:

{

   "command" : format("simulate.exe -i in.txt -p %d > out.%d.txt",param,param),

   "inputs" : [ "in.txt", "calib.dat", "simulate.exe" ],

   "outputs": [ format("out.%d.txt",param) ]

} for param in range(100)

For a more detailed example, see these example BWA workflows expressed in three different ways:

• Traditional Makeflow
• JSON Representation
• JX Representation

Thanks to Andrew Litteken for converting and testing many of our example workflows into the new format.