ASGS Setup Instructions on unity - StormSurgeLive/asgs GitHub Wiki
NOTE: These installation instructions are specific to unity. Specifics will vary server to server and may change over time as well. If you find anything that is incomplete or out of date, please update this guide!
NOTE ALSO: unity currently does not have a THREDDS server, so that piece of the setup is not covered in this guide. Refer to the ASGS Setup Instructions on hatteras for more details on this piece, as well as some miscellaneous tips that will also be useful here.
Setup Instructions:
Consider contacting Jason Fleming ([email protected]), Rick Luettich ([email protected]), Zach Cobell ([email protected]) or other ADCIRC repository contributors for access to https://github.com/adcirc/adcirc if you don’t already have it. Make sure to include your GitHub username. If you think you may need access, it's best to get the ball rolling now.
If you want to build an official release of ADCIRC, you will need access to this repo.
If you want to build a custom version of ADCIRC, in theory you will not need access. However, if ASGS fails to build the custom ADCIRC version of your choice you may want access to help work around this.
At some point you may get a warning like "Authenticity of host GitHub.com can't be established". You can ignore that.
Navigate to a directory on unity in which you’d like to set up your ASGS instance.
Run “git clone [email protected]:StormSurgeLive/asgs.git”. This will pull down the latest asgs code from GitHub. Any references to "your main asgs folder" later in the guide are talking about the folder you clone during this step.
There are also a number of detailed READMEs mixed in with the code in the ASGS repository. Don’t forget to check these out!
You may need to do some git setup if you haven’t used it on hatteras yet. Git should prompt you if so. Just follow the instructions.
Connect to a compute node via the “srun -t 1-0:0 --mem=4G -c 1 -p uri-cpu --pty bash” command.
4 GB should be enough, but if you run out of memory during the install just request more via the “--mem” flag
Make your own copy of Dave Ullman's fork of ADCIRC v54.01, which lives at /work/pi_iginis_uri_edu/adcirc-cg. Remember your copy's path for later.
This guide assumes you are looking to run ADCIRC with Dave's custom ADCIRC version, which has support for electronic pumps. If you would like to run an official release of ADCIRC, ignore the steps pertaining to "Dave's custom ADCIRC version" or "the adcirc-cg folder" and just build the version of your choice when you eventually run the "build adcirc" command.
Copy /work/pi_iginis_uri_edu/joshua_port_uri_edu/cmplrflags/cmplrflags.mk to [your adcirc-cg folder]/work.
This file should be similar to the existing one, except with --heap-arrays 8192 added to FFLAGS1 and with "NETCDFHOME :=modules/modulefiles/netcdf/2022.2.1_intel" and "FLIBS :=$(FLIBS) -L$(NETCDFHOME)/lib -lnetcdff -lnetcdf" added under "ifeq ($(NETCDF),enable)".
Copy /work/pi_iginis_uri_edu/joshua_port_uri_edu/unity to [your main asgs folder]/platforms.
Copy /work/pi_iginis_uri_edu/joshua_port_uri_edu/qscript.template.jwp to your asgs folder.
Run the following command to load some necessary modules for the ASGS install: "module load mpich-intel/4.0 icc/oneapi/2022.0.2 flex/2.6.4 netcdf/2022.2.1_intel"
As of July 2023, the mpich-intel/4.0 module gives a warning that it's broken. However, the install still seems to work as normal.
If this command fails, run “module avail” to see what your options are. Module names will likely change over time as versions change.
If you get stuck, reach out to Kevin Bryan ([email protected]) to ask which modules you should use.
Navigate to the main asgs folder. Run "./init-asgs.sh" and go through the setup instructions.
The defaults should be correct for the most part: unity, intel, skip git checkout.
init-asgs.sh could take up to a couple of hours to finish. Be patient.
Run asgsh (still within the main asgs folder) to enter the ASGS Shell.
Run “build adcirc” and follow the instructions to build Dave Ullman's custom version ADCIRC v54.01 directly. You will just need the path of your copy of the adcirc-cg folder from earlier in the guide.
If you hit errors when building ADCIRC via ASGS, try running "module avail netcdf" to see which NetCDF modules currently exist on unity. If the available Intel NetCDF module's version is different than what's in the cmplrflags.mk file you copied earlier, you'll need to update the "2022.2.1_intel" to whatever the current module version is.
If this doesn't do the trick, try running "export MACHINE=x86_64" and "export OS=linux-gnu" before trying to build ADCIRC. I did this at some point while trying to resolve errors, but I'm not sure it was necessary or not.
If you still can't get the custom version to build, you can work around this by building an official ADCIRC release via "build adcirc" (e.g. v55.01) and building the custom version manually. To manually build ADCIRC:
Load Kevin's list of recommended modules: "module load uri/main intel/2021b HDF5/1.12.1-iimpi-2021b netCDF-C++4/4.3.1-iimpi-2021b netCDF-Fortran/4.5.3-iimpi-2021b".
If this command fails, run “module avail” to see what your options are. Module names will likely change over time as versions change.
If you get stuck, reach out to Kevin to ask which modules you should use.
Make every executable file in adcirc-cg/work using "make clean NETCDF=enable NETCDF4=enable compiler=intel [name of executable file]".
Executable files show up in green when you run "ls" from the terminal.
If the build still won't work, you can make a copy /home/joshua_port_uri_edu/adcirc-cg and try to run with that.
Once the build finishes, run “load adcirc [build name from the last step]” at the prompt
If you don’t remember the build name, run “list adcirc”
Copy /work/pi_iginis_uri_edu/joshua_port_uri_edu/config_files/asgs_config_jwp.sh to your asgs directory. You will likely need to make changes to this file, but it is difficult to give complete guidance on this piece given the amount of configuration options. So, I recommend you lean on the wiki and config files stored on GitHub (https://github.com/StormSurgeLive/asgs-configs/) for reference. Also refer to the ASGS Setup Instructions on hatteras wiki page, as it contains some tips on some key parameters in this file.
Be sure to set ADCIRCDIR=[path of your adcirc-cg/work folder] in your config file if you manually built ADCIRC.
Enter “define config [name of your configuration file]” at the prompt
You may want to make multiple config files for different kinds of runs. Use this command to switch between them.
Create a scratch directory in which you’d like ASGS to conduct its ADCIRC runs (e.g. /work/pi_iginis_uri_edu/[your username]/scratch). No need to leave ASGSH to do this. This folder will hold all ADCIRC input and output files as well as ASGS-specific files and metadata.
Enter “define scratchdir [scratch directory]” to point ASGS at the directory you just created.
To confirm things are working as intended, run “verify”. If you encounter any errors, attempt to repair your configuration by leaning on this guide, the wiki, other ASGS users you’re in contact with, the issues list on the ASGS GitHub page (https://github.com/StormSurgeLive/asgs/issues) – maybe it’s a bug!, etc. and then retry the verify command.
Congratulations, you should now be all configured! There are just a few steps you should take before running. First, run “exit” to leave ASGSH and then “exit” again to leave interactive mode.
Run “screen”. This is necessary before you can detach the ASGS process from your console window so that you can close down your console without closing ASGS.
Feel free to request more or less time based on how long you need to continuously run ASGS by updating the bit after the -t flag.
Run asgsh.
Load the list of recommended modules from before: "module load uri/main intel/2021b HDF5/1.12.1-iimpi-2021b netCDF-C++4/4.3.1-iimpi-2021b netCDF-Fortran/4.5.3-iimpi-2021b" or, if applicable, whatever you had to update that list to.
Run "ulimit -s unlimited" to remove limits on stack size. Without this, ADCIRC runs will fail on unity.
Type “run” at the prompt to kick off ASGS.
If you’d ever like to run a different ASGS build, you can do that by running “load adcirc [build name]” before you attempt to run.
To detach the process from the console, type “CTRL+A CTRL+D”. To reattach, run “screen -r”.
If multiple detached processes exist at once, you can run “screen -ls” to specify the one to reattach.