ASGS Setup Instructions on hatteras - StormSurgeLive/asgs GitHub Wiki
NOTE: These installation instructions are specific to hatteras and were compiled in August 2022. Specifics will vary server to server and may change over time as well. If you find anything that is incomplete or out of date, please update this guide!
You will need this in several steps when you try to build ADCIRC. Get the ball rolling now.
If you plan to have ASGS post forecast results to the THREDDS server and also want email notifications to send when forecasts are posted, you should also ask Jason for an AWS SMTP username and password. See the section on "OPENDAPNOTIFY" below for more specifics. If you're not sure you can always sort this out later.
There are also a number of detailed READMEs mixed in with the code in the ASGS repository. Don’t forget to check these out!
You may need to do some git setup if you haven’t used it on hatteras yet. Git should prompt you if so. Just follow the instructions.
Now we’ll follow a similar process to step 2) to set up SSH keys for renci_tds. First navigate to ~/.ssh. Generate a public/private key pair using “ssh-keygen”. The default name (id_rsa) is just fine.
Copy the contents of id_rsa.pub to ~/.ssh/authorized_keys. If the file doesn’t exist, create it.
Note that renci_tds, the hatteras THREDDS server, is also on ht1.renci.org, meaning that it has the same file system as when you connect to ht1.renci.org.
Open ~/.ssh/config. If it doesn’t exist, create it. Copy the contents of [main asgs folder]/ssh_config into the file and then update the variables for renci_tds as appropriate.
As of the writing of these instructions, you should update the username from ncfs to your own username and you should also change the hostname from ht4.renci.org to ht1.renci.org.
IdentityFile should be the private key you created earlier.
Connect to a compute node via the “sinteractive” command.
You’ll likely want to request a bit of extra time via the “-t HH:MM:SS” flag, since sinteractive processes only last for two hours by default on hatteras and this could definitely take longer than that (e.g. “sinteractive -t 24:00:00”)
If it turns out you’re running out of memory during the install, add the “-m [requested memory in MiB]” flag to the sinteractive call (e.g. “sinteractive -m 4096” to allocate 4 GiB).
Load the Intel compiler and mvapich modules only.
This command might look like: “module load compiler/2022.0.2 compiler-rt/2022.0.2 mvapich2/2.3.7-intel”
Run “module avail” to see what your options are. Module names will likely change over time as versions change.
Navigate to the main asgs folder. Run init-asgs.sh and go through the setup instructions.
The defaults should be correct for the most part: hatteras, intel, skip git checkout.
Wait for init-asgs.sh to finish. This will take quite a while: potentially a couple of hours.
Run asgsh (still within the main asgs folder) to enter the ASGS Shell.
Run “build adcirc”.
Follow the instructions to select your desired version of ADCIRC.
Once the build finishes, run “load adcirc [build name from the last step]” at the prompt
If you don’t remember the build name, run “list adcirc”
Create a configuration file in your asgs directory. This should be a shell script (*.sh) and can be named whatever you’d like. It is difficult to give complete guidance on this piece given the amount of configuration options, so I recommend you lean on the wiki, config files from other ASGS users on hatteras, and config files stored on GitHub (https://github.com/StormSurgeLive/asgs-configs/) for reference. With that said, here are some key parameters to understand that will hopefully help you avoid some roadblocks:
BACKGROUNDMET=GFS instructs ASGS to run while being forced by GFS winds. NAM is another supported wind product.
FORECASTCYCLE dictates when BACKGROUNDMET forecasts will run (as opposed to just nowcasts to keep the model “hot”). You could set them to run at “00”, for example, for midnight forecasts, or “00,06,12,18”, for forecasts every six hours.
TROPICALCYCLONE=on is your other main forecast option. It dictates that NHC forecasts for the specified STORM and YEAR will be used as inputs to GAHM (NWS = 20) to force the model. Only one of BACKGROUNDMET and TROPICALCYCLE can be on at a time.
OPENDAPNOTIFY are the email addresses that should be sent a notification when files are posted.
To get emails working via OPENDAPNOTIFY, reach out to Jason Fleming (see step 1) for an AWS SMTP username and password.
If you already have an ~/asgs-global.conf file, make sure it has an "[email]" block in it like the one in the asgs-global.conf.sample file in your main asgs folder. If not, copy that entire block into your asgs-global.conf file.
If you don't have an ~/asgs-global.conf file, copy asgs-global.conf.sample to your home directory (i.e. ~) and rename it to "asgs-global.conf".
Open asgs-global.conf and replace smtp_username and smtp_password with the ones you got from Jason. Save and close the file.
SCENARIOPACKAGESIZE dictates how many forecast scenarios run. It can be any integer from 1 to 10. These specific scenarios are then specified with ENSTORM=[scenario name]. Look at an example to emulate the full syntax.
If the keyword “veer” is part of the scenario name, ASGS will look for another parameter, PERCENT, to dictate how it should adjust the storm track relative to the width of the track cone. PERCENT=100 implies a right edge track, while -100 implies a left edge track.
POSTPROCESS=(script1 script2 … ) allows you to run code after a forecast completes. This can include custom code that you write.
ASGS will look for post-processing scripts in [main asgs folder]/output
postAdditionalFiles=(file1 file2 … ) allows you to post additional output files to THREDDS than the normal set of output files from ADCIRC.
postExcludeFiles=(file1 file2 … ) lets you prevent files that would normally be posted to THREDDS from being posted
QSCRIPTTEMPLATE defines the name of your qscript file, which you should store in your main asgs folder. The qscript file allows you to set parameters related to the ADCIRC job. A template qscript file, qscript.template, is included in the main asgs folder.
If you're running using a large mesh, you will probably need to increase the memory available to the execution by adding the line "#SBATCH —mem-per-node=[Desired memory, e.g. "2G"]. You can test with various amounts to gauge how much your jobs need.
Enter “define config [name of your configuration file]” at the prompt
Create a scratch directory in which you’d like ASGS to conduct its ADCIRC runs (e.g. /projects/ees/dhs-crc/[username]/asgs). No need to leave ASGSH to do this. This folder will hold all ADCIRC input and output files as well as ASGS-specific files and metadata.
Enter “define scratchdir [scratch directory]” to point ASGS at the directory you just created.
To confirm things are working as intended, run “verify”. If you encounter any errors, attempt to repair your configuration (leaning on this guide, the wiki, other ASGS users you’re in contact with, the issues list on the ASGS GitHub page (https://github.com/StormSurgeLive/asgs/issues) – maybe it’s a bug!, etc.) and then retry the verify command.
Congratulations, you should now be all configured! There are just a few steps you should take before running. First, run “exit” to leave ASGSH and then “exit” again to leave sinteractive mode.
Run “screen”. This is necessary before you can detach the ASGS process from your console window so that you can close down your console without closing ASGS.
Run “sinteractive -t HH:MM:SS” with some reasonably long amount of time specified. This should be the duration of time you’d like ASGS to run uninterrupted in the background.
Run asgsh.
Type “run” at the prompt.
If you’d ever like to run a different ASGS build, you can do that by running “load adcirc [build name]” before you attempt to run.
To detach the process from the console, type “CTRL+A CTRL+D”. To reattach, run “screen -r”.
If multiple detached processes exist at once, you can run “screen -ls” to specify the one to reattach.
Miscellaneous Tips:
If you’re stuck and have struck out in your troubleshooting, have questions about how ASGS works that you haven’t been able to answer, etc. try posting on the ASGS Discussions page on GitHub (https://github.com/StormSurgeLive/asgs/discussions). There is also an ASGS-Ops Slack channel that Jason, Brett, and other ASGS users monitor. If the Discussions area on GitHub just isn’t working out, it may be useful to ask Jason, Brett, or another ASGS operator to add you to the Slack channel.
I strongly recommend backing up your ASGS configuration file (for example on StormSurgeLive/asgs-configs) and any other custom code you’re using with your ASGS instance (e.g. post-processing tasks). You don’t want to lose those by accident.
If you ever need to update ASGS, you can do that by pulling the latest ASGS code from GitHub via “git pull origin master” and then running update-asgs (in the main asgs folder). update-asgs requires parameters in order to know what to update. A common parameter is "--update-shell”, which instructs update-asgs to update asgsh. ASGSH should be updated at a minimum every time you pull new code from GitHub.
If you run into errors after updating, Jason recommends just setting up a new ASGS instance rather than wasting time troubleshooting. Once you have SSH configured, your config file made, and you understand the install process, this should be relatively painless. Never fear!
If you set up the GitHub SSH Key per their documentation and it won't let you commit to your repository without a password, try running "git remote -v". If your addresses start with https:// you'll connect via HTTPS instead of SSH, which is bad, because you set up your keys for SSH. To fix this, run "git remote set-url origin [email protected]:[end of github URL]". For example, "https://github.com/StormSurgeLive/richamp-support" would become "[email protected]:StormSurgeLive/richamp-support". “git remote -v” should now show the updated value and if everything else is set up right you should be able commit without a password going forward.
This is why the instructions have you set up SSH keys before cloning the ASGS repo.
Jason and Brett welcome contributions to ASGS. If you encounter a bug you know how to fix, you can fork the repo, make the change, and submit a pull request.