Event detection using distpy on Windows

In this tutorial we will look at how event detectors can be constructed using distpy. The example is for Windows, which highlights some design choices in the coding style of the module.

In particular, distpy uses the following basic pattern in all ingesters, controllers and workers

def main():
    # do things

if __name__ == "__main__":
    main()

When writing python for Windows, due to the single-threaded nature of the python kernel this scheme allows the multiprocessing module to generate threads without starting hundreds of instances of the main kernel.

If you use this approach in your python scripts, they will work just as well on Windows and linux and will retain the massively scalable feature of distpy.

If you've already completed the cloud tutorial, there is no difference in the python script when creating an event detector. All of the changes required to complete this case are contained in the JSON file that describes the processing flow.

We will walk through the changes required to turn that example into an event detector. Many types of event can occur, such as cement curing/aging, boiling/bubbling liquid surfaces, actuation of down-hole hardware, falling fluid in pulsating or slugging flows, cable events and of course microseismicity.

Event detectors

Typically events will appear as a hyperbola (microseismic event arriving from the formation) or a chevron (initiated in the cement and travelling up and down at 13,000 ft/s; or initiated on the cable and travelling up and down at 20,000 ft/s). Whatever the cause of the event, the aim in creating an event detector is to highlight, isolate and then measure the event.

Here we will make the following event detector for events on the cable:

1. Highlight by extracting signal travelling between 15,000 and 20,000 ft/s
2. Isolate by applying an STA/LTA transform to the envelope of the data, creating a strong onset signature
3. Measure using the peak-to-peak amplitude of the onset in the transformed data

Rather than counting or locating events, we construct an event detection attribute that highlights a particular data window as likely containing an event. That data window (e.g. 1 second of SEGY) can then be visually inspected via a thumbnail image, and/or that waveform processed in more detail for event determination either by machine-learning or physics-based tools. Another useful feature of using an event detection attribute is that it can be display for use in visually examining hours of data and immediately getting an impression of whether the events are clustered in time or space. Again the attribute can then be subject to further automated processing to mathematically extract the clusters.

An initial signal processing flow for event detection

The processing chain is given by the following JSON, you should copy this into a file in your config directory. For example `myEventDetector.json':

{

"command_list" :
[
 { "name" : "fft",            "uid" :  1, "in_uid" :  0 },
 { "name" : "fft",            "uid" :  2, "in_uid" :  1, "axis" : 0},
 { "name" : "thumbnail",      "uid" :  3, "in_uid" :  0, "directory_out" : "orig_png" },
 { "name" : "rms_from_fft",   "uid" :  4, "in_uid" :  1, "low_freq" : 0, "high_freq" : -1 },
 { "name" : "velocity_map",   "uid" :  5, "in_uid" :  2},
 { "name" : "velocity_mask",  "uid" :  6, "in_uid" :  5, "min_velocity" : 15000, "max_velocity" : 20000, "smooth" : 1.02 },
 { "name" : "multiply",       "uid" :  7, "in_uid" :  2, "gather_uids" : [6] },
 { "name" : "ifft",           "uid" :  8, "in_uid" :  7, "axis" : 0},
 { "name" : "ifft",           "uid" :  9, "in_uid" :  8, "axis" : 1},
 { "name" : "real",           "uid" : 10, "in_uid" :  9},
 { "name" : "butter",         "uid" : 11, "in_uid" : 10, "freq" : 600, "order" : 5, "type" : "highpass"},
 { "name" : "thumbnail",      "uid" : 12, "in_uid" : 11, "directory_out" : "filtered_png", "format" : "png" },
 { "name" : "multiply",       "uid" : 13, "in_uid" : 11, "gather_uids" : [11] },
 { "name" : "sta_lta",        "uid" : 14, "in_uid" : 13, "sta" : 50, "lta" : 200},
 { "name" : "thumbnail",      "uid" : 15, "in_uid" : 14, "directory_out" : "events_png", "format" : "png" },
 { "name" : "peak_to_peak",   "uid" : 16, "in_uid" : 15, "window_length" : 200},
 { "name" : "write_witsml",   "uid" : 17, "in_uid" : 16, "directory_out" : "p2p", "low_freq" : [0], "high_freq" : [1],  "gather_uids" : [4], "data_style" : "p2p" }
]
}

Here we are detecting events that are at high frequency and high phase-velocity (e.g. cement). The flow above makes a thumbnail image of the original strain-rate for visual inspection of the underlying measured signals. The data is subject 2D Fourier Transform, in distpy we separately FFT on each axis using the reduced_mem system. After 2D FFT the phase velocities map to straight-lines (known as an f-k, or frequency-wavenumber, plot). We construct a velocity map corresponding to that f-k space, and use that to generate a mask that can be applied to extract the high phase-velocity portion of the data. After applying the mask, we apply the inverse transform back to the time-distance (tx) domain. At this point we have data filtered to contain only the high phase-velocity signal, we can high-pass filter that signal to extract events with high frequency content. An STA/LTA (short-term average divided by long-term average) transform of the signal's envelope is well-known in microseismic processing to be a good way to highlight events in a data-stream. Finally to make an event attribute over the whole time-window we extract the maximum peak-to-peak (minimum to maximum) signal within the long-term window used in the STA/LTA transform.

To change CASE00.py into an event detector, we need to write a new signal processing flow (as a JSON) and update the strainrate2fbeConfig.json to point to our new configuration.

To change CASE00.py to run on Windows, we change the paths in sgyConfig.json and 'strainrate2fbeConfig.json` so that the lines:

 "in_drive"  : "/archive/projects",
 "out_drive" : "/scratch/myusername",

point to Windows drives, for example:

 "in_drive"  : "c:/archive/projects",
 "out_drive" : "c:/scratch/myusername",

What we learned

In this tutorial we saw how distpy can be used to create event detection algorithms, how to do 2D velocity filtering, and how to convert an existing workflow design for Linux to run on Windows.

We also discuss the coding pattern to make python multiprocessing scripts compatible across Windows and Linux.