MSC OSCAR workflow - wmo-im/wmdr GitHub Wiki

Introduction

OSCAR/Surface is the World Meteorological Organization's official repository of WIGOS metadata for all surface-based observing stations and platforms.

As an NMHS, the Meteorological Service of Canada provides its station metadata to OSCAR on an ongoing basis to ensure station metadata is managed, authoritative and up to date from source.

This document will provide an outline of how we provide WIGOS metadata to OSCAR in an automated fashion.

High level architecture

MSC station metadata repository  ->  produce WIGOS XML  ->  publish to OSCAR

        database                     YAML -> pygeometa          pyoscar

Description

  • script reads MSC databases to send YAML objects to pygeometa and creates WIGOS XML files on disk
  • pyoscar publishes WIGOS XML files to OSCAR

Requirements and tools

Python 3 is required. Conda or venv can be used if an isolated environment is required.

  • OSCAR account and API token
  • pygeometa: Python package to generate geospatial metadata. WIGOS metadata is a supported standard
  • pyoscar: Pythonic API to WMO OSCAR

Workflow

# install pygeometa and pyoscar
pip install pygeometa pyoscar

# generate WIGOS XML
#
# my-generate-wigos-xml-files.py does not exist and is only an example.  The
# requirement is on the user here to develop a script to produce YAML files (if
# using the pygeometa command line tool) or objects in memory (if using the pygeometa
# API)
python my-generate-wigos-xml-files.py  # example, writes WIGOS XML to disk to, say, output-dir

# publish to OSCAR DEPL (default)
for i in output-dir/*.xml; do pyoscar upload -x $i -at <API_TOKEN> -e depl; done

# publish to OSCAR production
for i in output-dir/*.xml; do pyoscar upload -x $i -at <API_TOKEN> -e prod; done

Notes

  • scheduling: currently this is run as a weekly process but can be implemented as per user requirements (e.g. hourly, daily, or event-driven)
  • the above is a 2-step process that writes WIGOS files to disk and then publishes them to OSCAR. Depending on your requirements, environment and development workflow, you can create the entire data workflow in memory and stream accordingly to OSCAR
  • the above can also be modified to run the entire workflow as a single step foreach metadata record

Contacts

⚠️ **GitHub.com Fallback** ⚠️