Date started: 2026-09-09
Analyst: George.Vandenberghe (original build system)
Investigating: Terry.McGuinness (with AI assistant: GitHub Copilot, model Claude Fable 5.1, VS Code agent mode)
Working copy: /scratch3/NCEPDEV/global/Terry.McGuinness/sorci
Original (read-only reference): /scratch3/NCEPDEV/global/gwv/heradat/u16/sorci

George's self-constructed GFS v16-era build (NEMS/FV3/WW3, fv3gfs.fd checked out at dbf413f9, WW3 submodule 41e7c9ca) builds successfully with WW3=N but fails with WW3=Y. Driver script is build.gwv in the sorci directory.

• rsync -aHS --no-owner --no-group of the full sorci tree (105 GB, 17,407 files, 43 symlinks, 0 errors). Log: /scratch3/NCEPDEV/global/Terry.McGuinness/rsync_sorci.log
• fv3gfs.fd/NEMS/src/conf/ was mode 700 in George's tree (unreadable), so it was excluded and recreated empty. NEMS regenerates configure.nems* / modules.nems* there during gmake ... configure, so this is harmless.

build.gwv:

• source /home/George.Vandenberghe/w3/simstacks/simu4/build/setupa changed to the real shared path /scratch3/NCEPDEV/global/gwv/heradat/simstacks/simu4/build/setupa
• Removed the interactive read a pause so the script can run unattended
• Script is currently set to WW3=Y 32BIT=Y (the failing case)

No other files have been modified.

build.gwv
  -> module load intel/2022.1.2 impi/2022.1.2 cmake intelpython
  -> source setupa            (sets NETP, ESMFMKFILE, CMAKE_PREFIX_PATH)
  -> exports NCEPLIBS *_ROOT / *_LIB / *_INC vars pointing at $NETP
  -> fv3gfs.fd/tests/compile.sh <PATHTR> hera.intel "WW3=Y 32BIT=Y" 1 YES NO
       -> cd NEMS ; gmake COMPONENTS="WW3,FMS,FV3" ... distclean
       -> gmake COMPONENTS="WW3,FMS,FV3" ... build -j 1 VERBOSE=1
            -> NEMS/src/incmake/component_WW3.mk
                 -> cd WW3/model/esmf ; make WW3_COMP=hera ww3_nems
                      -> targets: env setup gout switch
                      -> gout builds ww3_grid/outf/outp/prep/gint/prnc/ounf/ounp
                         then rewrites switch NOGRB -> "NCEP2 NCO" and builds ww3_grib
                      -> w3_make ww3_multi_esmf ; w3_make ww3_multi

Note: despite the user-level description "CMake build", this NEMS-era tree drives WW3 through GNU make + WW3's own w3_make/comp/link scripts, not CMake. compile_cmake.sh exists in tests/ but is NOT what build.gwv calls.

• George built on Hera (hfe06), modules: intel/2022.1.2 impi/2022.1.2 cmake/3.28.1 intelpython/2023.2.0.
• Library stack: $NETP = /scratch3/NCEPDEV/global/gwv/heradat/simstacks/simu4/netcdf140.492.460.mapl241.fms2301.crtm240
  • esmf.mk hardcodes ESMF_F90COMPILER=mpiifort (classic Intel)
  • Both ESMF_8_0_1 and ESMF_8_8_0 exist; build.gwv overrides setupa to use 8_0_1
• Ursa cannot build this: no intel/2022.1.2, no impi, no mpiifort. Only oneAPI 2024 is available. Investigation must continue on Hera.
• George's readable env file: /home/George.Vandenberghe/envsett (superset of what build.gwv exports; notably it DOES set PNG_LIB and Z_LIB, and W3NCO_LIB4).

The WW3=Y path is the only one that runs WW3's gout target, which builds ww3_grib with NCEP2 NCO switches. WW3/model/bin/link.tmpl then links:

${G2_LIB4} ${W3NCO_LIB4} ${BACIO_LIB4} ${JASPER_LIB} ${PNG_LIB} ${Z_LIB}

Comparing against what build.gwv exports:

Expected symptom if hypothesis is right: ww3_grib link fails with undefined references to w3tagb_, w3tage_, png_*, inflate*/deflate*, or jas_*.

All referenced library files were verified to exist in $NETP: lib/libw3nco_4.a, lib64/libg2_4.a, lib64/libpng.a, lib/libz.a, lib/libbacio_4.a, lib64/libjasper.a. libw3nco_4.a exports w3tagb_.

Secondary things to watch for in the log:

• WW3/WW3_INSTALL/nuopc.mk in the copied tree is stale and points at /lfs/h1/ops/para/packages/gfs.v16.3.32/... (WCOSS2 paths). distclean should remove and regenerate it; confirm it does.
• component_WW3.mk passes WW3_COMP=$(FULL_MACHINE_ID) = hera. Both WW3/model/esmf/Makefile and bin/cmplr.env accept hera, so this should be fine.

1. Open this workspace on a Hera front end with folders:

   • /scratch3/NCEPDEV/global/Terry.McGuinness/global-workflow_forked
   • /scratch3/NCEPDEV/global/Terry.McGuinness/sorci

2. Confirm toolchain:

   module load intel/2022.1.2 impi/2022.1.2 cmake intelpython
   which mpiifort mpiicc

3. Reproduce with a captured log (WW3=Y is already set in build.gwv):

   cd /scratch3/NCEPDEV/global/Terry.McGuinness/sorci
   ./build.gwv > build_ww3y.log 2>&1
   grep -n -i -E "error|undefined|not found|\*\*\*" build_ww3y.log | head -50

   under fv3gfs.fd/WW3/model/tmp/ and obj/ -- check those if the top-level log is not specific.

4. If the hypothesis holds, the minimal fix is to add to build.gwv (next to the other library exports):

   export W3NCO_LIB4=${w3nco_ROOT}/lib/libw3nco_4.a
   export PNG_LIB=${PNG_ROOT}/lib64/libpng.a
   export Z_LIB=${ZLIB_ROOT}/lib/libz.a

   then rerun step 3.

5. If the failure is something else, capture the exact failing command and error, update this file, and continue from there.

6. Once WW3=Y builds, verify fv3gfs.fd/NEMS/exe/NEMS.x contains WW3 symbols:

   nm fv3gfs.fd/NEMS/exe/NEMS.x | grep -i -c wmesmfmd

   (George's WW3=N executable returns ~2 hits; a real WW3 build should return many.)

• Modify only files under /scratch3/NCEPDEV/global/Terry.McGuinness/sorci.
• Do not touch George's original tree or his library stack.
• Do not change anything system-wide.

module load intel/2022.1.2 impi/2022.1.2 cmake intelpython works on Hera; mpiifort/mpiicc resolve to /apps/oneapi/mpi/2021.5.1/bin/.

The excluded NEMS/src/conf/ was NOT harmless. Four git-tracked files were missing (make.rules, configure.nems.NUOPC.in, module-setup.sh.inc, module-setup.csh.inc) and GNUmakefile:16 hard-includes conf/test-results.mk, so distclean aborted before touching WW3. Fix: git -C fv3gfs.fd/NEMS checkout -- src/conf.

module-setup.sh.inc runs module purge on Hera, then loads modulefiles/hera.intel/fv3.lua, which only sets CMAKE_* variables. Result: mpiifort: Command not found when FMS compiles. George's unreadable conf/ probably held a modified module-setup.sh.inc without the purge. Fix (our copy only): added load("intel/2022.1.2"), load("impi/2022.1.2"), load("cmake"), load("intelpython") to fv3gfs.fd/modulefiles/hera.intel/fv3.lua.

WW3/model/esmf/Makefile line 17:

NETCDF_CONFIG := $(shell which nc-config)

bin/comp.tmpl (line 84) and bin/link.tmpl (line 118) then run $NETCDF_CONFIG --fc, --includedir, --flibs for every serial NetCDF program in the gout target (ww3_prnc, ww3_ounf, ww3_ounp, ...).

George's stack is netCDF-C 4.9.2 (nc-config --version). nc-config removed the Fortran options (--fc, --flibs) in netCDF-C 4.4; they now print unknown option: --fc to stdout. WW3's comp captured that text as the compiler name, giving the observed error in WW3/model/tmp/ww3_prnc.err:

bin/comp: line 101: unknown: command not found
      *** file ww3_prnc.o not found ***
*** Missing object files ***
make[2]: *** [makefile:101: .../exe/ww3_prnc] Error 3
gmake[1]: *** [Makefile:175: gout] Error 1

ww3_grid, ww3_outf, ww3_outp, ww3_prep, ww3_gint linked fine because they do not use NetCDF. WW3=N never runs the gout target, so it never hits this. nf-config (netCDF-Fortran 4.6.0) in the same $NETP/bin supports all three options and returns --fc = mpiifort.

1. fv3gfs.fd/WW3/model/esmf/Makefile: NETCDF_CONFIG ?= $(shell which nf-config) (was := with nc-config, which also ignored any environment override).
2. build.gwv: export NETCDF_CONFIG=$NETP/bin/nf-config.
3. build.gwv: added W3NCO_LIB4, PNG_LIB, Z_LIB exports required by link.tmpl for ww3_grib (next step in gout after the NetCDF tools; pre-empts the original section-5 hypothesis).

./build.gwv with WW3=Y 32BIT=Y completed with exit=0 (log: build_ww3y.log). All WW3 programs built: ww3_grid ww3_outf ww3_outp ww3_prep ww3_gint ww3_prnc ww3_ounf ww3_ounp ww3_grib ww3_multi_esmf ww3_multi, plus WW3_INSTALL/nuopc.mk.

Verification of fv3gfs.fd/NEMS/exe/global_fv3gfs.x (80 MB):

1. WW3/model/esmf/Makefile line 17: NETCDF_CONFIG ?= $(shell which nf-config)
2. build.gwv: export NETCDF_CONFIG=$NETP/bin/nf-config
3. build.gwv: export W3NCO_LIB4=... PNG_LIB=... Z_LIB=... (for ww3_grib)

Items 1+2 are the actual root cause. Item 3 was not observed to fail in our run because we set it pre-emptively, but link.tmpl line 111 references all six variables and three were unset in the original build.gwv, so it is very likely the next failure George would have hit.

Blockers A and B above were artifacts of our copy (unreadable conf/ dir) and are not needed in George's tree.

Agent: GitHub Copilot running Claude Fable 5.1 (VS Code). The agent selected and issued all tool calls below, performed the log analysis, and applied the fixes; Terry.McGuinness directed the investigation and reviewed results.

Server: eib-mcp-gateway (v7.21.0, tenant gw, branch develop). First attempts on the eib-mcp-gateway-ms instance all returned Cannot read properties of undefined (reading 'invoke'); the server was restarted and the calls were repeated.

Summary: 11 calls, 1 high-value (search_documentation #10), 2 low-value, 8 no value. The one high-value hit came from the ingested WW3 wiki (Tier 3 RAG), not from the code graph.

• Fix cross-repo search_issues: repository: ufs-weather-model returns "Validation Failed"; WW3 should also be a valid target repo.
• Index sorc/build_*.sh: analyze_code_structure reports sorc/build_ufs.sh not found although it is in develop. The graph should cover the whole sorc/ build layer since build failures are a common task.
• Deduplicate symbols in analyze_code_structure output.
• get_system_configs config_type=modules should return modulefiles/module_base.hera.lua / module_gwsetup.hera.lua (build modules), not env/HERA.env (runtime env).
• Lower the search_architecture confidence threshold or return the nearest community instead of an empty result; two natural-language build queries returned nothing.
• Ingest WW3 model/bin/comp.tmpl, link.tmpl, esmf/Makefile and nf-config vs nc-config guidance into the RAG; the WW3 FAQ hit was the closest thing and it still recommended nc-config, which is exactly what broke here with netCDF-C >= 4.4.
• Server robustness: the first instance failed every call with an internal JS error (reading 'invoke') rather than a descriptive message; a health check on connect would save a round of wasted calls.