Detailed Analysis of HOMEDIR - TerrenceMcGuinness-NOAA/global-workflow GitHub Wiki

Detailed Analysis of HOMEDIR in the Weather Forecast System

The HOMEDIR: '/scratch1/NCEPDEV/global/${USER}' setting defines a critical path in the global weather forecast modeling system. This directory serves multiple specialized functions:

Core Usage Purposes

  1. Experiment Root Directory

    • Serves as the base location for experiment directories (EXPDIR)
    • Each forecast experiment (identified by PSLOT) typically creates a subdirectory here
    • Contains the workflow configuration files (XML definitions for Rocoto)
  2. Persistent Model Output Storage

    • Houses COM directories that store model outputs needing to persist between cycles
    • Stores post-processed GRIB2 files and other forecast products
    • Maintains ensemble member outputs for statistical processing
  3. Restart File Repository

    • Contains model restart files required for cycling data assimilation
    • Stores initial condition files modified by the user
    • Holds atmospheric and oceanic state information between forecast cycles
  4. Configuration Management

    • Stores custom namelists and parameter files
    • Contains user modifications to model physics options
    • Houses experiment-specific variable settings
  5. Log Collection

    • Central repository for job logs that need persistence
    • Maintains error reports across multiple cycles
    • Stores performance metrics and timing information

Technical Characteristics

  1. File System Properties

    • Located on scratch1 filesystem, optimized for high I/O performance
    • Not backed up but more persistent than temporary directories
    • Higher quota allocation than standard home directories
  2. Workflow Integration

    • Referenced by the Rocoto workflow manager for job dependencies
    • Used as a base path for experiment directory structures
    • Linked to NOSCRUB variable for long-term data retention
  3. User Management Responsibilities

    • User is responsible for managing quota usage
    • Requires periodic cleanup of old experiments
    • Not automatically purged like STMP and PTMP directories
  4. Path Relationships

    • Distinguished from STMP and PTMP which handle transient data
    • More persistent than job-specific temporary directories
    • Less permanent than archive storage (HPSS) defined in ATARDIR

This directory is fundamental to the workflow's data management strategy, providing the balance between performance, persistence, and user-specific isolation needed for operational meteorological modeling.