Cookbook - kerfoot/slocum-utils GitHub Wiki
Wiki ▸ Cookbook
Example usage of the various utilities contained in this repo.
Usage and options:
> convertDbds.sh -h
I strongly recommending reading the Slocum Glider Data File Primer for an in-depth discussion of command line processing of native glider binary data files. convertDbds.sh is a shell script wrapper around the vendor provided tools that simplifies the process of converting, merging and filtering the binary data files.
This package also contains the most recent version of the vendor supplied documentation on the dbd file format and processing utilities.
The vendor provided dbd2asc executables convert the native binary encoded *.[demnst]bd files to ascii using the sensor list unique to the file's sampling configuration. The executables use the *nix concept of executing a command and piping the output into the next executable. Details and specific examples of this process can be found in the Slocum Glider Data File Primer.
Native Slocum glider data file contents are specified by the operator using one of the following configuration files, present on the glider:
- dbdlist.dat
- ebdlist.dat
- mbdlist.dat
- nbdlist.dat
- sbdlist.dat
- tbdlist.dat
The sensor list must be either present in the file or stored shore-side. Including this sensor list in the header of every file sent to shore can significantly increase the transmission time. As such, it is often sent only on the first segment where it can be transmitted once and stored locally. Once this file exists, subsequent flight and science binary files containing the same sensors may be decoded using the existing sensor list cac file.
The location of this file must be specified when using dbd2asc or the file conversion will fail. See the configuration for information on how to specify this location via the user's environment or a command line option.
2 environment variables are optionally used by convertDbds.sh:
- DBD_CACHE_DIR: The location of the sensor list cache files (.cac extension) used by dbd2asc.sh to convert binary *.[demnst]bd files to ascii tables. Once a particular sensor list .cac file is present on the shoreside computer, subsequent binary flight and science data files may be decoded using the exiting .cac file. Each sensor list .cac file is uniquely identified by an 8-digit hexadecimal number. The recommended practice is to create a directory to which all sensor list .cac files will be written and/or searched for and to set the location in DBD_CACHE_DIR environment variable. This will result in an archive of all sensor list .cac files for future binary to ascii flight and science data file conversions.
- TWRC_EXE_DIR: The location of the vendor provided executables used to convert, merge, filter and write ascii table files. If not specified, the versions included in this package are used. The versions included in this package are stable and it is recommended that they be used (i.e.: do not set this environment variable unless there is good reason to do so).
convertDbds.sh provides 2 options that may be used to specify these locations on the command line:
- -c DBD_CACHE_DIR: specify the DBD_CACHE_DIR location
- -b TWRC_EXE_DIR: specify the TWRC_EXE_DIR location
Let's say we want to convert and merge the binary flight and (optionally) science binary data files contained in the following directory:
kerfoot@kerfoot-new: ~/sandbox/glider/dbds/binary
$ ls -l
total 67
-rw-r--r-- 1 kerfoot scott 636 Oct 20 2010 ru22-2010-265-5-0.sbd
-rw-r--r-- 1 kerfoot scott 8201 Oct 20 2010 ru22-2010-265-6-0.sbd
-rw-r--r-- 1 kerfoot scott 11926 Oct 20 2010 ru22-2010-265-6-0.tbd
and write the ascii table files to:
~/sandbox/glider/dbds/ascii
We can do this by executing the following command:
> convertDbds.sh ~/sandbox/glider/dbds/binary ~/sandbox/glider/dbds/ascii
TWRC executables directory: /home/kerfoot/sandbox/slocum-utils/dbd/linux-bin
Binary source: /home/kerfoot/sandbox/glider/dbds/binary
ASCII destination: /home/kerfoot/sandbox/glider/dbds/ascii
Temp conversion directory: /tmp/convertDbds.sh.nxPHtsj2kX
Sensor list cache directory: /home/kerfoot/sandbox/slocum-utils/cache
----
Source flight file : /home/kerfoot/sandbox/glider/dbds/binary/ru22-2010-265-5-0.sbd
Converting flight data file ONLY...
Output File Created: /tmp/convertDbds.sh.nxPHtsj2kX/ru22_2010_265_5_0_sbd.dat
Updating sbd aad1ae87.cac permissions (664)
----
Source flight file : /home/kerfoot/sandbox/glider/dbds/binary/ru22-2010-265-6-0.sbd
Source science file: /home/kerfoot/sandbox/glider/dbds/binary/ru22-2010-265-6-0.tbd
Converting & Merging flight and science data files...
Output File Created: /tmp/convertDbds.sh.nxPHtsj2kX/ru22_2010_265_6_0_sbd.dat
Updating sbd aad1ae87.cac permissions (664)
Updating tbd 45b32f1a.cac permissions (664)
------------------------------------------------------------------------------
Moving output files to destination: /home/kerfoot/sandbox/glider/dbds/ascii...Done.
==============================================================================
2/2 files successfully converted.
==============================================================================
The above output to STDOUT can be suppressed using the -q option. The binary files are converted to ascii and the flight and science (if present) files are merged into a single ascii table file, one file per segment:
$ ls -l /home/kerfoot/sandbox/glider/dbds/ascii
total 129
-rw-r--r-- 1 kerfoot scott 1904 Apr 22 23:40 ru22_2010_265_5_0_sbd.dat
-rw-r--r-- 1 kerfoot scott 115235 Apr 22 23:40 ru22_2010_265_6_0_sbd.dat
The default extension for all created ascii table files is .dat. An alternate extension can be specified using the -e EXTENSION option:
$ convertDbds.sh -c ../cache -e mrg ~/sandbox/glider/dbds/binary ~/sandbox/glider/dbds/ascii
TWRC executables directory: /home/kerfoot/sandbox/slocum-utils/dbd/linux-bin
Binary source: /home/kerfoot/sandbox/glider/dbds/binary
ASCII destination: /home/kerfoot/sandbox/glider/dbds/ascii
Temp conversion directory: /tmp/convertDbds.sh.yvDTcbXXho
Sensor list cache directory: /home/kerfoot/sandbox/slocum-utils/cache
----
Source flight file : /home/kerfoot/sandbox/glider/dbds/binary/ru22-2010-265-5-0.sbd
Converting flight data file ONLY...
Output File Created: /tmp/convertDbds.sh.yvDTcbXXho/ru22_2010_265_5_0_sbd.mrg
Updating sbd aad1ae87.cac permissions (664)
----
Source flight file : /home/kerfoot/sandbox/glider/dbds/binary/ru22-2010-265-6-0.sbd
Source science file: /home/kerfoot/sandbox/glider/dbds/binary/ru22-2010-265-6-0.tbd
Converting & Merging flight and science data files...
Output File Created: /tmp/convertDbds.sh.yvDTcbXXho/ru22_2010_265_6_0_sbd.mrg
Updating sbd aad1ae87.cac permissions (664)
Updating tbd 45b32f1a.cac permissions (664)
------------------------------------------------------------------------------
Moving output files to destination: /home/kerfoot/sandbox/glider/dbds/ascii...Done.
==============================================================================
2/2 files successfully converted.
==============================================================================
$ ls -l /home/kerfoot/sandbox/glider/dbds/ascii
total 129
-rw-r--r-- 1 kerfoot scott 1904 Apr 22 23:47 ru22_2010_265_5_0_sbd.mrg
-rw-r--r-- 1 kerfoot scott 115235 Apr 22 23:47 ru22_2010_265_6_0_sbd.mrg
Once the files have been converted from binary to ascii via dbd2asc, the list of available sensors may be filtered to include a subset of the total sensors using dba_sensor_filter and provided the list of desired sensors. This is particularly useful when converting and merging the dbd/ebd flight and science pairs, respectively. For example, let's say we want to convert one or more dbd/ebd pairs but only want time, depth and temperature. We can create a newline-separated text file containing only the sensors of interest and specify this using the -f SENSOR_LIST option with convertDbds.sh:
$ convertDbds.sh -f ../resources/ctd-sensor-list.txt -c ../cache -e mrg ~/sandbox/glider/dbds/binary ~/sandbox/glider/dbds/ascii
TWRC executables directory: /home/kerfoot/sandbox/slocum-utils/dbd/linux-bin
Binary source: /home/kerfoot/sandbox/glider/dbds/binary
ASCII destination: /home/kerfoot/sandbox/glider/dbds/ascii
Sensor filter : /home/kerfoot/sandbox/slocum-utils/resources/ctd-sensor-list.txt
Temp conversion directory: /tmp/convertDbds.sh.2kDqR9H0Nk
Sensor list cache directory: /home/kerfoot/sandbox/slocum-utils/cache
----
Source flight file : /home/kerfoot/sandbox/glider/dbds/binary/ru22-2010-265-5-0.sbd
Converting flight data file ONLY...
Output File Created: /tmp/convertDbds.sh.2kDqR9H0Nk/ru22_2010_265_5_0_sbd.mrg
Updating sbd aad1ae87.cac permissions (664)
----
Source flight file : /home/kerfoot/sandbox/glider/dbds/binary/ru22-2010-265-6-0.sbd
Source science file: /home/kerfoot/sandbox/glider/dbds/binary/ru22-2010-265-6-0.tbd
Converting & Merging flight and science data files...
Output File Created: /tmp/convertDbds.sh.2kDqR9H0Nk/ru22_2010_265_6_0_sbd.mrg
Updating sbd aad1ae87.cac permissions (664)
Updating tbd 45b32f1a.cac permissions (664)
------------------------------------------------------------------------------
Moving output files to destination: /home/kerfoot/sandbox/glider/dbds/ascii...Done.
==============================================================================
2/2 files successfully converted.
==============================================================================
kerfoot@kerfoot-new: ~/sandbox/slocum-utils/dbd
$ ls -l /home/kerfoot/sandbox/glider/dbds/ascii
total 47
-rw-r--r-- 1 kerfoot scott 591 Apr 23 00:13 ru22_2010_265_5_0_sbd.mrg
-rw-r--r-- 1 kerfoot scott 29726 Apr 23 00:13 ru22_2010_265_6_0_sbd.mrg
Comparing the resulting files sizes to those here shows that the file sizes are significantly smaller since they include only a subset of the sensors available in the file.