IMOS Matlab Toolbox Overview

• Guillaume Galibert (since May 2011)
• Paul McCarthy (from March 2009 to February 2010)
• Lindsay Pender
• Craig Steinberg

Overview

The IMOS Matlab Toolbox aims to provide an automated, easy to use interface for converting raw instrument data into IMOS compatible NetCDF files, ready for handover to AODN. The first stage of this project is underway, and is specifically targeting 'delayed mode' data; in other words, data which is manually retrieved from long-term mooring sites.

Typical usage of the Toolbox would proceed as follows:

1. User downloads data set from an instrument supported by the Toolbox (e.g. SBE37).
2. User imports data set into Toolbox along with a deployment data base containing metadata.
3. User performs pre-processing routines over data set, such as depth calculation for each sensor, time offset to UTC time...
4. User executes semi-automatic QC procedures over data set, such as time and location checks, in/out of water flagging, range checks, spike detection, stationarity checks...
5. User performs manual QC over data via interactive display.
6. Toolbox generates AODN/IMOS compatible NetCDF files containing metadata and data at non QC'd and QC'd levels, along with plots of the data in png format.

While the Toolbox will be designed to execute with minimal user intervention (apart from semi-automatic QC procedures configuration and manual quality control), the Matlab source will be available for users who wish to contribute to the existing code with new features, have more fine grained control over the execution flow, or who wish to integrate the Toolbox into an existing Matlab codebase. The Toolbox will be made available to all.

System requirements

System requirements for using the Toolbox are as follows:

• Windows (either in 32bit or 64bit) or Linux 64bit operating system
• Java Runtime Environment 1.6
• Matlab from R2012b (Optional, a self-contained executable version is available but requires installing the relevant Matlab Component Runtime)
• A local deployment database (Optional)
• Microsoft Access 32bit if you use a 32bit operating system / Matlab, 64bit if you use a 64bit one (Optional, can be bypassed with UCanAccess, especially when using Linux)

Using the toolbox

See how to install, configure and use the toolbox in the corresponding pages.

Deployment database

The provided deployment database is a MS Access database which is local to each ANMN regional node, and which conforms to the deployment database schema developed at AIMS. The deployment database is used to store metadata for every deployment of interest to the site. The toolbox should always be used with a deployment database, otherwise more work is required by the user in filling out metadata for each data set and some important pre-processing/quality control procedures won't be available.

The deployment database contains metadata for all aspects of a deployment, including instrument/sensor/platform model and serial number, and deployment location/depth. This information is used by the Toolbox for the generation of MEST compatible metadata documents, and metadata fields in the NetCDF files. See deployment database manual to have an idea on how to document your local deployment database.

Supported instruments

See the current supported instruments list.

In order for the Toolbox to be useful for a variety of instruments, the input format for a particular instrument sometimes has to be provided in a standard format, or a limited set of formats.

Pre-processing routines

See the current pre-processing routines list.

Quality control procedures

See the current quality control procedures list.

IMOS quantitative scale flag is described here.

Output format

The Toolbox generates NetCDF4 files according to the IMOS NetCDF Conventions and the IMOS NetCDF File Naming Convention.

Global and variables attributes can be automatically documented via configurable template files.

Processed data levels

The following levels of processed data will be saved as separate NetCDF files:

• Level 0: Unmodified (except TIME fom local time to UTC) data without any QC flag
• Level 1: Possibly added and/or modified data with QC flags

These levels (and the output files) map directly to the Processed Data levels defined in the IMOS NetCDF Conventions.