Automatic documentation of workflows

In a distpy command configuration file, you will see something like:

{
"document" : 1,
"name" : "Event detection v1",
"description" : "Low pass to look at the seismic (below 200 Hz) data, take the envelope of the data, apply STA-LTA and then pick using peak_to_peak.",
"command_list" :
[ 
 { "name" : "fft",            "uid" :  1, "in_uid" :  0 },
 { "name" : "thumbnail",      "uid" :  2, "in_uid" :  0, "directory_out" : "orig_png", "clip_level" : 0.1 },
 { "name" : "rms_from_fft",   "uid" :  3, "in_uid" :  1, "low_freq" : 0, "high_freq" : -1 },
 { "name" : "butter",         "uid" :  4, "in_uid" :  0, "freq" : 200, "order" : 5, "type" : "lowpass"},
 { "name" : "thumbnail",      "uid" :  5, "in_uid" :  4, "directory_out" : "filtered_png", "format" : "png", "clip_level" : 0.01 },
 { "name" : "sta_lta",        "uid" :  6, "in_uid" :  4, "sta" : 10, "lta" : 50},
 { "name" : "thumbnail",      "uid" :  7, "in_uid" :  6, "directory_out" : "events_png", "format" : "png", "clip_level" : 0.2 },
 { "name" : "peak_to_peak",   "uid" :  8, "in_uid" :  6, "window_length" : 100},
 { "name" : "write_witsml",   "uid" :  9, "in_uid" :  8, "directory_out" : "p2p", "low_freq" : [0], "high_freq" : [1],  "gather_uids" : [3], "data_style" : "p2p" }
]
}

This is the example we saw discussed in the directed graphs example.

The first 3 items are of interest to us now. "document" : 1, informs distpy that we want to write documentation corresponding to this workflow, rather than bulk processing lots of data chunks.

When this mode is selected, a LaTeX document is generated which can be used as a Technical Appendix, outlining the keywords in the workflow and all their attributes. Additionally a dot language graph is generated corresponding to the flow.

The document and the associated graph in *.gv format appears in the "out_directory" for the processing. There is a placeholder in the LaTeX for a *.png image generated from the *.gv graph. This is done using dot and the command:

dot config.gv -Tpng -o config.png

To turn the document into a PDF you can use your favourite LaTeX compiler, MiKTeX is an example that can be deployed in lots of different hardware environments.

Wiki reference

The Command Dictionary, and its Index are also autogenerated. When the unit_test.py is run to sanity check that all the commands are executable, the files reference.md and sidebar.md are generated locally. These are then used to update the Wiki pages.