Data Acquisition - ad3ller/oskar GitHub Wiki

Integrating data acquisition with the sequencer.

The process of merging experimental control with data acquisition using oskar is very simple. There are two methods:

  1. Add an attribute to the HDF5 group for a given SQUID, which can be used to note single-value measurements (e.g., laser power/ lab temperature).
  2. Record a dataset (e.g., oscilloscope waveforms and camera images).

In both cases, semaphores are used to prevent read/ write conflicts of the single HDF5 data file that each run produces.

Attributes

Often measurements from pressure gauges or power supplies are useful for verifying if interesting "signals" are real, or just arise from drifting experimental conditions. However, lots of precise repeat measurements of the conditions might be overkill, and a record of a single value, or the mean value, for each SQUID is often sufficient for system monitoring. Such a measurement can be recorded into the attributes (metadata) for a SQUID group using the global variable Rec.

Rec is a global variant that is wiped at the start of every SQUID acquisition period. At the end of the acquisition period any values written to the variant will be appended to the SQUID attributes with the prefix REC:. See the example rec_random.vi, which continuously generates a random number every 5 seconds. Attributes with the same name are overwritten, thus the last value to be written during a given SQUID will be the one that is ultimately stored.

Arbitrarily many measurements can be added to the SQUID group's attributes in this way, so long as each has a unique name.

VAR settings are already recorded to a SQUID's attribute record, however, REC can be used to ensure values were successfully set. For example, a piece of hardware that you control using a VAR will probably also have read functionality. By reading back the same setting controlled by the VAR and then storing it as a REC attribute you have a way to check that the hardware is actually listening to your requests.

(This functionality can be built into a VAR control, but it's slightly awkward. It's logical to read from the hardware shortly after writing to it, however, because the REC variant is wiped at the start of the acquisition period (after the VARS have been set), this is too early to add the attribute to the REC variant. The solution is to read the value from the hardware, then wait for the Sequencer.vi to begin acquisition before writing to the Rec variant. See VAR/PAUSE_RAND.vi as an example of a combined VAR / REC vi. In general, it's simpler to just have an independent REC vi that periodically queries the hardware; just be careful to avoid communication conflicts.)

Datasets

Most scientific data will take some form of regularly sized array, such as oscilloscope waveforms, camera images, etc.

For each SQUID there might be many repeat measurements from the same device. In general, to record such data a program is needed to continuously acquire data, then by reading the following global variables:

  • Save (Boolean. Is save selected in Sequencer.vi?)
  • Acquire (Boolean. Is the Sequencer.vi state currently `acquiring data' (not, e.g., setting VARs)?
  • H5 (array of paths to hdf5 file(s)
  • SQUID (integer value as a string. The iterating number that uniquely identifies the current line of the running sequence.)

it's relatively easy to coordinate the saving of data in accordance with the state of the sequencer.

The data acquisition programs should only save when sequencer.vi has set the global variables Save AND Running to True. The global variables H5 and SQUID indicate which file and group it should write the data to. LabVIEW semaphores are used to prevent more than one vi writing to the hdf5 file at the same time. See the examples osc_acquire.vi and cam_acquire.vi.

DAQ

osc_acquire.vi demonstrates how cluster data can be written to a dataset. This data structure is useful if analysis is performed in LabVIEW to determine, e.g., trigger events (time, amplitude, width), or statistical features (mean, max, min, range, FWHM).

The sequencer doesn't know (or care) what data acquisition vi's are open and running. Those that are will save data; those that aren't won't.

Because VAR conditions are stored in each SQUID's attributes, and as data relating to a SQUID is stored within its subgroup, it's trivial to map data to the relevant VAR conditions. This assumes that data is recorded correctly, however, the individual data acquisition vi's are responsible for ensuring this (not the sequencer) and it's possible to get things wrong.

Warning - oskar is designed for experiments where many (>10) repeat measurements are recorded at a rate of 10 Hz to 0.3 Hz, then an experimental parameter is varied and more measurements are similarly made. For lower acquisition rates (e.g., long camera exposure times) you must ensure that data taken near the boundary of two SQUIDS is mapped to the correct one, or safely ignored if it's ambiguous. The global variable EOS (end-of-squid) can be used to synchronise the sequencer to a data acquisition or hardware trigger. See `Tools/skip_squid.vi'. By using EOS with the acquisition time set to be unattainably long, a data acquisition device can effectively control how long the sequencer acquires data for (e.g., until 10 waveforms have been recorded).

Viewing data

A run will produce a file called [RID]_raw.h5, which is saved to the folder(s) `[Base(s)]/[YYYY]/[MM]/[DD]/[RID]/'.

Opening this file using HDFView will reveal a structure akin to:

oskar

The top level group is the RUN and has attributes, RID, SEQ FILE, AUTHOR, and DESCRIPTION. Each group within the RUN represents a SQUID value. This has attributes DATETIME, START, END, ACQUIRE, and all appropriate VAR settings in the form, VAR:NAME = VALUE. REC data is similarly recorded. Each SQUID group contains all of the datasets that are associated with it. Datasets can have their own attributes, which can be used to record, e.g., exposure time, amplifier gain, etc.

Note: the SQUID group names are strings. This is important if you wish to access a given group using, e.g., h5py.

The structured format of HDF5 makes batch analysis very simple. The ability to attach metadata makes it easy to associate measurements with experimental conditions and instrument settings.

The H5Data class in e11_analysis is designed for processing oskar data with python.