ObsDB - glos/Documentation GitHub Wiki

The GLOS Observations Database (OBS DB) is a legacy data store that ingests near real-time sensor data from non-federally owned buoys and other fixed platforms in the Great Lakes.

• Data Types

  Data ingested into the OBS DB may include air and water temperature, wind, currents, waves, nutrients, chlorophyll, pH, dissolved oxygen, conductivity, phycocyanin and turbidity.

• Management

  Data from GLOS-funded and other buoys and fixed platforms are pushed to a GLOS ftp directory by the data provider in an agreed XML format. The ftp directory is scanned every ten minutes to identify new data files, which are then parsed to extract the sensor observations. The observations are subjected to basic quality control (QC) tests such as range checks, and then injected into the PostgreSQL OBS DB. Harvesting status is continually updated on the GLOS Buoy and Model Status page (http://data.glos.us/status), which tabulates the time of the most recent ingestion for all buoys.

• QA/QC

  Data providers are expected to conform to the GLOS Quality Management Plan to ensure appropriate procedures are in place to ensure proper calibration and deployment of instruments. GLOS currently performs automated testing of incoming data to detect gross errors.
    The current QC modules perform the following simple validation checks: • water temperature (0-40 deg C) • air temperature (0-50 deg C) • dew point (-30-50 deg C) • relative humidity (0-100%) • significant wave height (0-10m) • wave period (0-15 sec) • maximum wind gust and wind speed (0-50 m/s) • wind direction (0-360 deg) • barometric pressure (700-1200 mmHg) • turbidity (0-1000 NTU) • chlorophyll (-5-120 RFU) • phycocyanin (-5-120 RFU) • specific conductivity (100-700 uS/CM) • pH (6-10) • dissolved oxygen (-1-25 mg/L) • dissolved oxygen saturation (0-150%) • thermal string (0-40 deg C)

• Metadata

  GLOS has worked with the data providers to develop ISO 19119/19139-compliant metadata records for each platform. The records are manually curated and stored in the GLOS GeoNetwork metadata catalog.

• Backups

  The OBS DB is backed up weekly to an external, portable hard drive. Archiving. All non-federally owned near real-time data in the OBS DB are archived to NCEI in netCDF format. Additional details are available in the accompanying NODC Submission Information Form for GLOS initialized 8/24/2015. Data Access: Sensor data in the OBS DB are exposed for discovery, query and access through a JSON service implemented via Java servlet.