Troubleshooting By Topic

Below we have debugging and troubleshooting steps below, topics will be added as needed.

Suggested Debugging of an ASGS Installation Step

Based on comments provided directly in #1119.

Given,

prompt> ./init-asgs.sh -b -x "--list-steps"
           setup-env - Updates current environment with variables needed for subsequent steps. It only affects the environment within the asgs-brew.pl environment.
             openmpi - Downloads and builds OpenMPI on all platforms for ASGS. Note: gfortran is required, so any compiler option causes this step to be skipped.
                hdf5 - Downloads and builds the version of HDF5 that has been tested to work on all platforms for ASGS.
             netcdf4 - Downloads and builds the versions of NetCDF and NetCFD-Fortran that have been tested to work on all platforms for ASGS.
              wgrib2 - Downloads and builds wgrib2 on all platforms for ASGS. Note: gfortran is required, so any compiler option passed is overridden.
       cpra-postproc - Runs the makefile and builds associated utilities in the output/cpra_postproc directory
              output - Runs the makefile and builds associated utilities in the output/ directory.
                util - Runs the makefile and builds all associated utilities in the util/ directory.
          input-mesh - Runs the makefile and builds all associated util/input/mesh in the input-mesh/ directory.
     input-nodalattr - Runs the makefile and builds associated utilities in the util/input/nodalattr directory.
                perl - Install local Perl version used for ASGS.
        perl-modules - Install Perl modules used for ASGS.
        image-magick - Install local ImageMagick tools and Perl module Image::Magick.
             python3 - install python 3 locally and install required modules
              ffmpeg - Install ffmpeg and required libraries (nasm)
             gnuplot - Install gnuplot (commandline only)
               units - Install GNU Units utility
                 nco - Install The netCDF Operators (NCO) Toolkit
                pigz - Install pigz, unpigz - parallel gzip
              adcirc - Builds ADCIRC and SWAN if $HOME/adcirc-cg exists.

You can straight up skip breaking steps for debugging purposes. For example,

./init-asgs.sh -x "--skip-steps input-mesh"

You can skip multiple steps, via:

./init-asgs.sh -x "--skip-steps input-mesh,image-magick,..."

This is how I debug. Basically I skip whatever is breaking, then the given command build; then you can run ./asgsh and manually debug the "build" commands present in ./cloud/general/asgs-brew.pl inside of the proper environment.

Also, look at the created ./update-asgs script - it contains the full asgs-brew.pl, which lets you then run it instead of ./init-asgs.sh,

./update-asgs "flags passed to asgs-brew.pl command ..."

An extreme command to immediately get to the shell, with nothing built but the environment set up would be:

prompt> ./init-asgs.sh -b -x "--update-shell"
...
prompt> ./asgsh

Detecting Model Failure

Using NCO (netCDF Operators), specifically ncap2 (netCDF arithmetic processor) and ncks (netCDF kitchen sink) as a concise method of creating diagnostics.

Determine the number of nodes

np=$(ncks --trd -m -M fort.63.nc | grep -E -i ": node, size =" | cut -f 7 -d ' ' | tr -d "," | uniq)

Count the number of nodes that would be considered dry on cold start based on topobathy depth:

ncap2 -O -v -s "land_msk[node]=0.0;land_msk.set_miss(-99999.0);where(depth < 0.0) land_msk=-99999.0; elsewhere land_msk=depth; print(land_msk.number_miss(),\"%d\n\");" fort.63.nc foo.nc

Counting the number of datasets in the file:

ndset=$(ncks --trd -M fort.63.nc | grep -E -i "^Root record dimension 0:" | cut -f 10- -d ' ')

Counting the number of missing values in the first dataset to see if all values are missing:

num_missing=$(ncap2 -O -C -v -s "print(zeta(0:0:1,0:$(($np - 1)):1).number_miss(),\"%d\");" fort.63.nc diagnostics.nc)

Count the number of missing values in each dataset in a file:

for snap in $(seq 0 $(($ndset - 1))) ; do 
   ncap2 -O -C -v -s "print(zeta($snap:$snap:1,0:$(($np - 1)):1).number_miss(),\"%d\");" fort.63.nc diagnostics.nc
done

Count the number of missing values in each dataset in a file and store the number of missing values in an array:

declare -a num_missing
for snap in $(seq 0 $(($ndset - 1))) ; do 
   num_missing+=( $(ncap2 -O -C -v -s "print(zeta($snap:$snap:1,0:$(($np - 1)):1).number_miss(),\"%d\");" fort.63.nc diagnostics.nc) )
done

Determine the max water level within a lat/lon bounding box for one dataset:

ncap2 -O -v -s "bbox_msk[node]=0.0;where((y >= 20.0 && y <= 30.1) && (x >= -95.0 && x <= -65.1)) bbox_msk=zeta(0:0:1,0:$(($np - 1)):1); elsewhere bbox_msk=zeta@_FillValue; zeta_msk_max=bbox_msk.max(); print(zeta_msk_max,\"%f\"); " fort.63.nc foo.nc

Compute the min, max, average, and standard deviation of water level within a lat/lon bounding box excluding the -99999.0 _FillValue along with the number of dry nodes for one dataset :

ncap2 -O -v -s "bbox_msk[node]=0.0;bbox_msk.set_miss(-99999.0);where((y >= 20.0 && y <= 30.1) && (x >= -95.0 && x <= -65.1)) bbox_msk=zeta(0:0:1,0:$(($np - 1)):1); elsewhere bbox_msk=zeta@_FillValue; print(bbox_msk.max());print(bbox_msk.min());print(bbox_msk.avg());bbox_msk_stdev=(bbox_msk-bbox_msk.avg()).rmssdn();print(bbox_msk_stdev);print(bbox_msk.number_miss());" fort.63.nc foo.nc

Compute the min, max, average, and standard deviation of water level within a lat/lon bounding box excluding the -99999.0 _FillValue, along with the number of dry nodes, for one dataset:

for snap in $(seq 0 $(($ndset - 1))) ; do 
   ncap2 -O -v -s "bbox_msk[node]=0.0;bbox_msk.set_miss(-99999.0);where((y >= 20.0 && y <= 30.1) && (x >= -95.0 && x <= -65.1)) bbox_msk=zeta($snap:$snap:1,0:$(($np - 1)):1); elsewhere bbox_msk=zeta@_FillValue; print(bbox_msk.min(),\"%f \");print(bbox_msk.max(),\"%f \");print(bbox_msk.avg(),\"%f \");print((bbox_msk-bbox_msk.avg()).rmssdn(),\"%f \");print(bbox_msk.number_miss(),\"%d\n\");" fort.63.nc foo.nc
done

sample output from the above:

-0.968176 1.927551 0.473537 0.137439 912724
-0.968176 1.927551 0.503579 0.136248 907240
-0.968176 1.927551 0.528694 0.134726 901804
-0.968176 1.927551 0.548272 0.132958 897026
-0.968176 1.927551 0.561440 0.131201 893015
-0.968176 1.927551 0.568402 0.130402 889561
-0.968176 1.927551 0.569775 0.130022 887166
-0.968176 1.927551 0.565483 0.129697 885613
...

Compute the min, max, average, and standard deviation of water level excluding the -99999.0 _FillValue along with the number of dry nodes over the full domain for one dataset:

for snap in $(seq 0 $(($ndset - 1))) ; do 
   ncap2 -O -v -s "data[node]=0.0;data.set_miss(-99999.0);data=zeta($snap:$snap:1,0:$(($np - 1)):1);print(data.min(),\"%f \");print(data.max(),\"%f \");print(data.avg(),\"%f \");print((data-data.avg()).rmssdn(),\"%f \");print(data.number_miss(),\"%d\n\");" fort.63.nc foo.nc
done

Compute the min, max, average, and standard deviation of water level excluding the -99999.0 _FillValue along with the number of dry nodes over the full domain for all datasets, writing the results to a file called statistics_${file}.txt:

file=fort.63.nc
if [ -e statistics_${file}.txt ](/StormSurgeLive/asgs/wiki/--e-statistics_${file}.txt-); then 
   rm statistics_${file}.txt
fi
for snap in $(seq 0 $(($ndset - 1))) ; do 
   ncap2 -O -v -s "data[node]=0.0;data.set_miss(-99999.0);data=zeta($snap:$snap:1,0:$(($np - 1)):1);print(data.min(),\"%f \");print(data.max(),\"%f \");print(data.avg(),\"%f \");print((data-data.avg()).rmssdn(),\"%f \");print(data.number_miss(),\"%d\n\");" $file foo.nc >> statistics_${file}.txt
done