_{Analysis and design produced with OMD Code Assist (GitHub Copilot, Claude Opus 5, VS Code) against the live GitLab CI configuration and the eib-mcp-rag MCP/RAG knowledge base — scoped, directed, and gated by Terry.McGuinness. Every finding was verified against repository source before publication; line references are to develop as of 2026-09-30.}

Analysis and design — September 30, 2026

Tracking issue: NOAA-EMC/global-workflow#5285

SDD phase: phase101_ctest_pr_label_state_transitions — ready; implementation not started

When a maintainer dispatches Trigger GitLab Pipelines with pipeline_type = CTests, the GitHub PR label advances CI-<Machine>-Ready → CI-<Machine>-Building and then stops.

The PR never shows CI-<Machine>-Running, and never reaches CI-<Machine>-Passed or CI-<Machine>-Failed. To a PR author the pipeline appears permanently stuck mid-build, even after every CTest has completed green.

The identical label sequence works correctly for the PR Cases modality.

The pipeline carries two parallel notions of "what kind of run is this," and they never reconcile:

.github/workflows/trigger-gitlab-pipelines.yml posts variables[PIPELINE_TYPE] and never GFS_CI_RUN_TYPE. The only bridge between them is this fragment in .base_config.before_script:

if [[ "${PIPELINE_TYPE}" == "ctests" ]]; then
  GFS_CI_RUN_TYPE="ctests"
  WORKSPACE_ID="${GFS_CI_RUN_TYPE}_${PR_NUMBER}_${CI_COMMIT_SHORT_SHA}_${CI_PIPELINE_ID}"
fi

That is a plain shell assignment inside before_script, so its visibility is phase-dependent:

Every label transition after Building is guarded by GFS_CI_RUN_TYPE == "pr_cases" — an equality test that is false under the CTest modality. The one transition that does work (Building) is guarded by GFS_CI_RUN_TYPE != "nightly", an inequality that is incidentally true for ctests.

That single == vs != operator difference is the entire visible symptom.

.gitlab-ci.yml, .build_template.script (~line 223):

elif [[ ${GFS_CI_RUN_TYPE} == "pr_cases" && ${PR_NUMBER} != 0 ]]; then
  ${GH} pr edit ${PR_NUMBER} --repo ${GW_REPO_URL} \
    --add-label "CI-${machine^}-Running" --remove-label "CI-${machine^}-Building"
fi

In script: scope GFS_CI_RUN_TYPE is ctests → guard false → transition skipped. Compare the Building transition 59 lines earlier (~line 164), reached through if [[ ${GFS_CI_RUN_TYPE} != "nightly" && ${PR_NUMBER} != 0 ]] — true for ctests.

dev/ci/gitlab-ci-hosts.yml defines finalize_success-{hera,gaeac6,orion,hercules,ursa,derecho}, each gated:

rules:
  - if: $PIPELINE_TYPE == "pr_cases" && ($CI_PIPELINE_SOURCE == "trigger" || ...)

No job carries $PIPELINE_TYPE == "ctests". Under the CTest modality the finalize stage is empty, so Passed has no code path whatsoever.

Note that rules: are evaluated at pipeline creation, where GFS_CI_RUN_TYPE is still pr_cases — so this is a PIPELINE_TYPE problem, not a GFS_CI_RUN_TYPE problem.

dev/ci/gitlab-ci-cases.yml, .finalize_success_template (~line 77):

elif [[ "${GFS_CI_RUN_TYPE}" == "pr_cases" && "${PR_NUMBER}" != 0 ]]; then
  "${GH}" pr edit "${PR_NUMBER}" --repo "${GW_REPO_URL}" \
    --add-label "CI-${Machine}-Passed" --remove-label "CI-${Machine}-Running" || true
fi

D2 and D3 are independent: adding CTest finalize jobs without relaxing this guard yields jobs that run and label nothing.

.failure_cleanup_template.after_script (~line 251) reads GFS_CI_RUN_TYPE, which in after_script scope is the unmodified pr_cases. Both .setup_ctests_template and .run_ctests_template extend this template, so a failed CTest job does currently get Failed — for the wrong reason. Two consequences:

1. The emitted comment is titled **BUILD FAILED** and points at ${GW_HOMEgfs}/sorc, even when the failure was in run_ctests. It reads sorc/logs/error.logs, which does not exist for a CTest failure, yielding (no error.logs file found).
2. Any fix that makes GFS_CI_RUN_TYPE consistent across shells will flip this guard to false and delete the one working CTest failure path.

GIT_CLONE_PATH is expanded by GitLab at pipeline-creation time from the unmutated WORKSPACE_ID:

GIT_CLONE_PATH = ${CI_BUILDS_DIR}/pr_cases_${PR_NUMBER}_${SHA}_${PIPELINE_ID}/global-workflow

while before_script recomputes GW_RUN_PATH = ${CI_BUILDS_DIR}/ctests_${PR_NUMBER}_${SHA}_${PIPELINE_ID}. The clone and the run root therefore live in different directories for CTest pipelines. Currently benign (CTests execute from ${GW_HOMEgfs}/dev/ctests/build), but any finalize/cleanup logic keyed on GW_RUN_PATH would target an empty tree.

graph TD
    A["GitHub trigger sets<br/>PIPELINE_TYPE=ctests"] --> B["GFS_CI_RUN_TYPE stays 'pr_cases'<br/>at pipeline level"]
    B --> C["before_script mutates<br/>GFS_CI_RUN_TYPE='ctests'"]
    C --> D["script: sees 'ctests'"]
    C --> E["after_script: sees 'pr_cases'"]
    B --> F["rules: see 'pr_cases'"]

    D --> G["D1: Running guard<br/>== pr_cases is FALSE<br/>label stuck at Building"]
    D --> H["D3: finalize body guard<br/>== pr_cases is FALSE<br/>no Passed label"]
    F --> I["D2: no finalize job<br/>matches ctests<br/>stage is empty"]
    E --> J["D4: failure path fires<br/>under wrong identity<br/>wrong comment text"]
    B --> K["D5: GIT_CLONE_PATH vs<br/>GW_RUN_PATH diverge"]

    G --> L["Issue 5285:<br/>PR appears stuck on Building"]
    H --> L
    I --> L

Replace the before_script mutation with workflow:rules:variables:, evaluated once at pipeline creation and therefore visible to rules:, before_script, script, and after_script:

workflow:
  name: "PR ${PR_NUMBER} - ${RUN_ON_MACHINES} (${GFS_CI_RUN_TYPE})"
  rules:
    # Scheduled nightly sets GFS_CI_RUN_TYPE explicitly; never override it.
    - if: $GFS_CI_RUN_TYPE == "nightly"
    - if: $PIPELINE_TYPE == "ctests"
      variables:
        GFS_CI_RUN_TYPE: "ctests"
    - when: always

Because WORKSPACE_ID and GIT_CLONE_PATH both interpolate GFS_CI_RUN_TYPE, this also resolves D5 with no additional change — the clone root and run root reconverge.

Every surviving guard collapses to a single question — does this pipeline label a PR?

labels_a_pr  ⇔  PR_NUMBER != 0  ∧  GFS_CI_RUN_TYPE != "nightly"

Both pr_cases and ctests satisfy it; nightly and PR_NUMBER == 0 do not. Encode it once in dev/ci/scripts/utils/ci_utils.sh as update_pr_label(), then repoint all six call sites:

This removes the ==/!= asymmetry as a class of defect rather than patching D1 alone.

Why the helper takes the machine name as an argument. MACHINE_ID is not a CI-layer variable — it is set by hostname pattern matching in ush/detect_machine.sh, reached only via dev/ush/gw_setup.sh or sorc/link_workflow.sh. It is therefore unset in after_script, which never sources either. Separately, .build_template already references source ".../config.${MACHINE_ID}" (~line 151) before anything sources gw_setup.sh — a pre-existing latent bug; every other template correctly uses config.${machine}.

.failure_cleanup_template hardcodes build semantics. Derive the stage from CI_JOB_NAME:

case "${CI_JOB_NAME}" in
  build-*)         _stage_label="BUILD FAILED"        ; _log="sorc/logs/error.logs" ;;
  setup_ctests-*)  _stage_label="CTEST SETUP FAILED"  ; _log="" ;;
  run_ctests-*)    _stage_label="CTEST FAILED"        ; _log="dev/ctests/build/Testing/Temporary/LastTest.log" ;;
  *)               _stage_label="CI JOB FAILED"       ; _log="" ;;
esac

CTest failures then surface LastTest.log (already captured as a job artifact by .run_ctests_template) instead of a nonexistent build log.

.finalize_fail_template is presently dead code — no host file instantiates it; pr_cases failure labelling comes from run_check_gitlab_ci.sh and .failure_cleanup_template. This work brings it to life for CTests, where the after_script path is being narrowed. Add finalize_success-ctests-* and finalize_fail-ctests-* for all six hosts, gated on $PIPELINE_TYPE == "ctests" and needs: the corresponding run_ctests-<machine>.

stateDiagram-v2
    [*] --> Ready: trigger dispatch
    Ready --> Building: build starts
    Building --> Running: build succeeds
    Running --> Passed: finalize_success
    Running --> Failed: finalize_fail / failure_cleanup
    Building --> Failed: build fails
    Passed --> [*]
    Failed --> [*]

Identical for pr_cases and ctests; bypassed entirely for nightly and PR_NUMBER == 0.

Considered: a distinct namespace (CI-<M>-CTest-Passed) to disambiguate concurrent modalities.

Rejected. It doubles the label set across six hosts (24 → 48 labels), requires changes to wcoss2.yaml's contains(...) merge gates, and solves a collision that cannot occur in practice — the trigger is workflow_dispatch-only and a maintainer runs one modality at a time per PR.

Documented caveat: if both modalities are run against the same PR, the later run's terminal label overwrites the earlier one's.

D4 is a regression landmine. Fixing the variable scoping (design item 1) flips the after_script guard false and deletes the only working CTest failure path. Design items 1–3 must ship as a single atomic commit — never item 1 alone.

This page is the specification for SDD phase phase101_ctest_pr_label_state_transitions, tracked under sdd_framework/workflows/ in the eib-mcp-rag-server repo. No pipeline YAML has been modified yet — the design sections above describe proposed changes, not merged code.

The phase defines 7 implementation steps, each with its own test gate, plus the risk register and rollback plan summarized above. A tracking session opens when implementation begins, and each step is recorded as its gate passes.

1. Should setup_ctests failures post a PR comment at all? They are usually infrastructure (CMake configure) rather than PR-code faults. Current design says yes, labelled as such.
2. optional: true on the CTest finalize needs: mirrors the pr_cases pattern, but the CTest matrix is a single parallel job rather than a per-case fan-out. Verify on_failure still triggers when the upstream job is skipped rather than failed.
3. Does wcoss2.yaml's merge gate need awareness of CTest-origin labels? It checks CI-Wcoss2-Passed only, and WCOSS2 is not in the CTest host matrix — believed unaffected.
4. Fix the pre-existing config.${MACHINE_ID} reference in .build_template here or separately? Unrelated to #5285 but sits three lines from touched code. Recommendation: file separately to keep the #5285 diff reviewable.

• Home — see Background information of Cases used in the CTest Framework for the CTest case catalog
• docs/source/contributor_guide/ci_cd_pipeline.rst — in-repo contributor documentation
• docs/source/contributor_guide/testing.rst — CTest framework reference