Jason Fleming [email protected]

The ADCIRC Surge Guidance System (ASGS) provides real time output from the ADCIRC or ADCIRC+SWAN models for analysis and interpretation by downstream applications. An interface specification is an invaluable tool that facilitates reliable coordination between the ASGS and downstream applications while enabling independent development processes.

As a result, this guide was created as a living document to detail the interface contract the ASGS and its downstream applications. This contract consists of the following three components: (1) the run.properties files for each forecast ensemble member; (2) the opendap server configuration; and (3) the format of the results files themselves.

Not all downstream applications will rely on every aspect of the interface described herein. Rather, each downstream application will use only a subset of this specification. However, all instances of the ASGS are required to adhere to all aspects of specification in order to provide assurance of compatibility to all downstream application developers.

A run.properties file is produced for each forecast ensemble run from the ASGS, and placed in the same results directory as the output files for the forecast run.

A sample of the run.properties file is shown below, organized into document subsections according to the nature of various properties. Definitions of these properties will be added later.

The following properties describe the type of input that was used to generate the results.

    storm class : TS
    storm name : KAREN
    stormnumber : 12
    stormname : KAREN
    year : 2013
    storm : 12
    modified : n
    track_raw_dat : bal122013.dat
    track_raw_fst : al122013.fst
    mesh : sl15_2010_HSDRRS_2012_v9
    RunType : Forecast
    ADCIRCgrid : sl15_2010_HSDRRS_2012_v9
    advisory : 10    
    currentcycle : 
    currentdate : 

Definitions

RunStartTime

Date and time in yyyyMMDDHH24 format, generally assumed to be in the GMT time zone unless otherwise indicated. It represents the time of the most recent hotstart file, that is, the date and time at the end of the most recent nowcast or hindcast period. It does not represent the start of the forecast period, or the time that the forecast was issued.

currentcycle

This two digit hour is in a 24 hour format and padded with a leading zero. This hour is equal to the hour from the RunStartTime value.

currentdate

The end date of the most recent nowcast or hindcast period, including the two digit year, month, and day. The value of this property is the same as year, month, and day from the RunStartTime value.

Output

The following properties indicate the presence, name, and format of the output (results) files.

    Water Surface Elevation Stations File Name : fort.61.nc
    Water Surface Elevation Stations Format : netcdf
    Water Surface Elevation File Name : fort.63.nc
    Water Surface Elevation Format : netcdf
    Barometric Pressure Stations File Name : fort.71.nc
    Barometric Pressure Stations Format : netcdf
    Wind Velocity Stations File Name : fort.72.nc
    Wind Velocity Stations Format : netcdf
    Barometric Pressure File Name : fort.73.nc
    Barometric Pressure Format : netcdf
    Wind Velocity File Name : fort.74.nc
    Wind Velocity Format : netcdf
    Maximum Water Surface Elevation File Name : maxele.63.nc
    Maximum Water Surface Elevation Format : netcdf
    Maximum Current Speed File Name : maxvel.63 
    Maximum Current Speed Format : ascii
    Maximum Wind Speed File Name : maxwvel.63.nc
    Maximum Wind Speed Format : netcdf
    Minimum Barometric Pressure File Name : minpr.63.nc
    Minimum Barometric Pressure Format : netcdf

Downstream

The following properties provide information for use by particular downstream applications for uses ranging from load balancing, presentation to different audiences, labeling results, etc.

    ceraServer : cera0
    asgs : nc
    intendedAudience : 
    downloadurl : 
    prodID : SADCsl15_2010_HSDRRS_2012_v9-UNC_vortex-nws19_20131005T1200_20131005T1200_20131007T1200_12<field>_Z.nc.gz

Analysis

The following properties provide key bits of data that describe how the results were generated.

    InitialHotStartTime : 4276800.00000000
    RunStartTime : 2013100512
    RunEndTime : 2013100712
    ColdStartTime : 2013081700
    WindModel : vortex-nws19
    Model : PADCIRC

HPC Resources

The following properties provide data about the High Performance Computing (HPC) resources that were involved in the production of results, which may be used in troubleshooting or performance analysis.

    directory storm : /projects/ncfs/data/asgs15784/10/nhcConsensus
    hostname : hatteras.renci.org
    instance : corpsbackup3
    forecastEnsembleMemberNumber : 0
    cpurequest : 480
    ncpu : 480
    numwriters : 0

THREDDS Catalog Service

To facilitate the publication, distribution, and access of ASGS results to dowstream applications, we are using THREDDS/OPeNDAP as a data access protocol between the ASGS instance that is generating results and any downstream applications that provide further analysis and visualization. Unidata's THREDDS DataServer (TDS) is a web server that provides metadata and data sccess for scientific dadtasets using several remote access protocols. One such protocol is OPeNDAP. As such, the configuration details of this service are critical to reliable compatibility with downstream applications. OPeNDAP can be used directly in applications to retrieve data, as through a service request to a TDS, or as implemented for example in the MATLAB toolbox nctoolbox. Capabilities and details of using TDS and OPeNDAP are available at http://www.unidata.ucar.edu/software/thredds/current/tds/ and http://www.opendap.org/

There are three main steps to the setup of THREDDS for ASGS:

1. Install the THREDDS application on a publically accessible computer and configure a catalog for ASGS the XML configuration file for the server;

2. Populate the THREDDS/ASGS server with model output into a defined and required directory hierarchy; and

3. Generate a catalog file that describes the contents on the THREDDS server.

Each of these steps are briefly described below. However, a detailed and up-to-date description of the setup and configuration of a TDS are fully described at the following URL, because there is no requirement that files posted in the manner described originate from ADCIRC. Any non-rectangular model that conforms to the data format and metadata specifications can be served this way.

https://docs.google.com/document/d/1iHHf4FN1ivGe039l_L0StuHB7d2VveAuzQsz8HVUkHA/pub

THREDDS/OPeNDAP Configuration

We use a Unidata THREDDS Data Server to serve ASGS output and the underlying web server for a THREDDS Data Server (TDS) is tomcat5, not apache. If tomcat5 is not running on the file server, install this first. Instructions are at:

http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.3/tutorial/index.html http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.3/tutorial/Checklist.html

The THREDDS app will be under the <$tomcat5>/webapps location, and its catalog configuration files are in <$tomcat5>/content/thredds. OPeNDAP is a java class under the THREDDS server.

Edit <$tomcat5>/content/thredds/threddsConfig.xml for the local installation.

Once the TDS is running, create a catalog for ASGS output. The xml file <$tomcat5>/content/thredds/catalog.xml is the root catalog for TDS configuration. An example of this root catalog is shown below, where three services are enabled; OPeNDAP, HTTP, and UDDC. The catalogRef tag points to the asgs.xml file, located in the same directory and an example is listed below.

   <?xml version="1.0" encoding="UTF-8"?>
   <catalog name="ASGS THREDDS DATA SERVER"
        xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
        xmlns:xlink="http://www.w3.org/1999/xlink">

     <service name="all" base="" serviceType="compound">
        <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
        <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
        <service name="uddc" serviceType="UDDC" base="/thredds/uddc/"/>
        <!-- test comment -->
        <!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
        <!--service name="wms" serviceType="WMS" base="/thredds/wms/" /-->
        <!--service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/" /-->
     </service>

   <catalogRef xlink:title="ASGS Output" xlink:href="asgs.xml" name=""/>  

   </catalog>

Example asgs.xml catalogRef file:

        <?xml version="1.0" encoding="UTF-8"?>
        <catalog name="ADCIRC/PADCSWAN Results from RENCI"
                xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
                xmlns:xlink="http://www.w3.org/1999/xlink">
   
          <service name="all" base="" serviceType="compound">
            <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
            <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
            <service name="uddc" serviceType="UDDC" base="/thredds/uddc/"/>
            <service name="uddc" serviceType="ISO" base="/thredds/iso/"/>
          </service>
   
          <datasetScan 
	        name="ASGS"
	        ID="ASGSTCResults" 
	        path="ASGS" 
	        location="/projects/ncfs/opendap/data/tc">
        
	        <serviceName>all</serviceName>
     	        <addDatasetSize/>
     	        <metadata inherited="false">
                <documentation type="summary">
		        Results produced by the ASGS system at MyLocation
  	        </documentation>
             </metadata>
             <filter>
                <include wildcard="*"/>
                <exclude wildcard="LOGS" atomic="false" collection="true"/>
             </filter>
             </datasetScan>

        </catalog>

Directory Hierarchy

The results files from the ASGS are posted in a particular directory hierarchy that enables us to have multiple instances of the ASGS on multiple machines producing results for multiple ensemble members on multiple meshes.

Put model output files into the following directory structure on the THREDDS server (ASGS can be configured to do this):

<$SP>/StormName/AdvisoryNumber/AdcircGrid/Machine/Instance/EnsembleName

where $SP is the starting/entry point into the THREDDS catalog. Here is an example of a fully qualified URL to an ADCIRC output file on RENCI’s THREDDS server:

http://opendap.renci.org:1935/thredds/dodsC/tc/nam/2013052300/nc6b/blueridge.renci.org/2/namforecast

There is actually no requirement on these directory names, except that they exist. In particular, the Instance can be used as a tag to denote specific instances of ASGS, such as “NOAATestRuns”. A run.properties file must accompany each ensemble member simulation. This file is automatically generated by ASGS, but can generated by some other process as needed, if the simulations are not being carried out by an ASGS instance.

Catalog

There is, unfortunately, no way to interrogate a thredds server to get a text file that describes the data content of the server. Methods are being developed that will do this for a THREDDS server, but these methods are not ready for production use. In lieu of this, RENCI has developed an automated cataloging script that allows a url-retrieve of the catalog file for each ASGS TDS. This catalog generator can be run periodically in cron to maintain an up-to-date listing of the TDS content.

The current version of the catalog file generator is described in detail at this URL:

https://docs.google.com/document/d/1iHHf4FN1ivGe039l_L0StuHB7d2VveAuzQsz8HVUkHA/pub

Output Files

Adherence to community-defined, accepted and adopted conventions for data formats and metadata content are critical for model interoperbility, workflow development, and end-user applications. To facilitate wide accessibility to ADCIRC output through THREDDS Data Servers, ADCIRC/ASGS outputs netCDF files (optionally specified in the fort.15 file). These files must also have metadata (global, dimension, and variable attributes) that conform to NetCDF Climate and Forecast (CF) Metadata Conventions with extensions for non-rectangular grid models (CF-UGRID).

NOTE: Make sure that ADCIRC and ADCPREP are compiled with netCDF4 support with compression.

It is critical for ADCIRC netCDF4 files to maintain strict CF-UGRID compliance. This extension to CF conventions allows applications to discover, access, and process data in uniform ways and without having to make assumptions about the content of the data files. This is the default for ASGS-generated output.

There is no technical requirement that the files be generated by ADCIRC. As long as files are posted that are netCDF CF-UGRID compliant, any application that can handle ADCIRC files can also handle other model files, including SLOSH when properly formatted.

Information on CF and CF-UGRID can be found at:

http://cf-pcmdi.llnl.gov/documents/cf-conventions

https://github.com/ugrid-conventions/ugrid-conventions

Climate and Forecast Standards

Description of the relevant parts of the CF standards that must be implemented in the ADCIRC and ADCIRC+SWAN NetCDF4 output files for compatibility with downstream applications.

http://cf-pcmdi.llnl.gov/documents/cf-conventions

NetCDF Metadata

Any other key pieces of NetCDF metadata that are used by downstream applications and must be provided for compatibility.

This must include a time stamp indicating the cold start date.

'** JASON: I don't see where this is in the metadata. I see that there is a "cs:" in the fort.15 header, and as a consequence its in the global attribute "description", but I don't see it as an attribute. I think this is a good idea, though, to put it as a global attribute in each file. **'

Appendix A: This Document

This document was prepared from the text file ASGSInterfaceGuide.txt using software called asciidoc (http://www.methods.co.nz/asciidoc/). The document can be formatted as an html page with the command

asciidoc --backend html5 -a toc2 ASGSInterfaceGuide.txt

or formatted as a pdf with the command

a2x --format=pdf -a toc2 ASGSInterfaceGuide.txt