Proposal: EMC-Bot for Automated Badge Management in Global Workflow CI/CD

Executive Summary

This proposal outlines a solution to implement a dedicated GitLab project called "emc-bot" to manage CI/CD status badges for the NOAA Global Workflow system. This approach will help streamline the continuous integration pipeline by offloading badge management tasks from the main workflow, increasing scalability as the system expands to support additional computational environments, and providing a centralized mechanism for badge management across NOAA's critical weather modeling infrastructure.

Background

The Global Workflow system is a critical component of NOAA's weather modeling infrastructure that requires robust CI/CD practices. As the system scales to support multiple computational environments, managing status badges for each environment becomes increasingly complex, cluttering the main pipeline configuration with non-essential tasks.

Solution Architecture

1. The EMC-Bot Project

Create a dedicated GitLab project called "emc-bot" specifically designed to manage CI/CD concerns that should be decoupled from the main workflow:

emc-bot/
├── .gitlab-ci.yml       # Main pipeline configuration
├── badge_updater/       # Badge updating scripts
│   └── update_badges.sh # Script to handle badge updates
└── README.md            # Documentation

2. Badge Updater Pipeline Configuration

EMC-Bot: Badge Management Solution for NOAA's Global Workflow CI/CD

Implementation Approach

3. Webhook Integration

To connect your main Global Workflow project with the EMC-Bot, implement a webhook-based trigger system:

graph LR
    A[Global Workflow Pipeline] --> |Completes| B[Webhook Trigger]
    B --> C[EMC-Bot Project]
    C --> |Updates| D[GitHub Gists]
    D --> |Displays| E[Status Badges on GitHub]

Setup Steps:

1. In the EMC-Bot project:

   • Create a pipeline trigger in Settings → CI/CD → Pipeline triggers
   • Name it "Global Workflow Pipeline Completion"
   • Save the generated trigger token securely

2. In the Global Workflow project:

   • Navigate to Settings → Webhooks
   • Add a new webhook with:
     • URL: https://vlab.noaa.gov/gitlab-licensed/NWS/Operations/NCEP/EMC/emc-bot/trigger/pipeline
     • Secret token: [Generated trigger token from step 1]
     • Trigger events: Select "Pipeline events"
     • Filter by branch (optional): develop

4. Badge Creation Process

For each HPC system that requires monitoring:

1. Create a GitHub Gist to hold the badge SVG
2. Record the Gist ID in the EMC-Bot configuration
3. Update the badge-updater-pipeline.yml file with the appropriate curl commands
4. In GitHub READMEs, reference the badges using URLs like: https://gist.githubusercontent.com/userid/gistid/raw/filename.svg

5. Remove Badge Updating from Main Pipeline

Clean up your main pipeline configuration by removing all badge-related code:

# Note: Badge updating has been moved to the EMC-Bot project
# See: https://vlab.noaa.gov/gitlab-licensed/NWS/Operations/NCEP/EMC/emc-bot

Benefits for NOAA's Global Workflow

Scientific Impact

1. Focus on Core Mission: By decoupling badge management from the main workflow, scientists and developers can focus on improving weather prediction models rather than CI/CD infrastructure.

2. Enhanced Visibility: Machine-specific badges provide clear status indicators for each computational environment, essential for global weather modeling that depends on multiple HPC systems.

3. Global Collaboration: Improved status visibility facilitates collaboration with international partners by clearly communicating the operational status of various computational environments.

Technical Benefits

1. Scalability: As NOAA integrates additional HPC resources, new badges can be added to the EMC-Bot without modifying the main workflow.

2. Separation of Concerns: Isolates CI/CD status reporting from the core global workflow pipeline.

3. Reduced Pipeline Complexity: Keeps the main workflow pipeline focused on building and testing the global weather modeling system.

4. Centralized Management: All badge-related code is maintained in a single location.

Future Expansion

The EMC-Bot project can be expanded beyond badge management to handle other CI/CD concerns:

1. Notification System: Implement alerts for pipeline failures across different computational environments
2. Testing Statistics: Generate reports on test coverage and success rates
3. Resource Usage: Track and report on CI/CD resource consumption across HPC systems
4. Deployment Tracking: Monitor production deployments of the global workflow system

Specific Implementation Steps

1. Create EMC-Bot project and configure basic pipeline structure
2. Implement badge updating functionality and create initial GitHub Gists
3. Set up webhook triggers and test integration with the main workflow
4. Remove badge updating from main workflow and document the new process

Conclusion

The EMC-Bot approach provides a robust solution for managing CI/CD badges for NOAA's Global Workflow system. By centralizing badge management in a dedicated project, the solution enhances the main workflow's focus on its critical mission: improving global weather prediction capabilities that are essential for understanding Earth's weather systems and their long-term consequences.

This approach aligns with best practices in CI/CD architecture and will scale effectively as NOAA continues to expand its computational resources across multiple HPC environments, ultimately supporting the organization's mission to observe, understand, and predict Earth's environment to protect life and property and conserve natural resources.

---

This proposal provides a comprehensive blueprint for implementing the EMC-Bot solution to manage CI/CD badges for the Global Workflow system. The approach is designed to be scalable, maintainable, and aligned with the critical mission of NOAA's weather modeling infrastructure.