access_TotalviewCylc - ACCESS-NRI/accessdev-Trac-archive GitHub Wiki
Using the Totalview debugger from cylc
The method previously described here for raijin no longer works on gadi because PBS now only allows setting the DISPLAY in interactive jobs.
Instead use reverse connections.
Using Totalview on gadi (using reverse connections)
Job configuration
In your suite, or job, you could avoid using rose mpi-run, and instead use an explicit launcher. Most calling scripts (in UM, OPS, VAR, SURF) have a variable called e.g. RECON_LAUNCHER, or OPS_LAUNCHER, which is the alternative to using rose mpi-launch.
Make sure that any variables that are used by rose-mpi-launch to do something, are instead defined otherwise, e.g. setting ulimits.
For intel-mpi set the launcher variable to e.g. tvconnect $(which mpiexec.hydra) --tv --debug -n $NPROC
or whichever are your $ROSE_LAUNCHER_PREOPTS normally. the --debug may not be necessary?
tvconnect
creates a way for a reverse connection to act to connect to totalview once the job begins.
Note for intel-mpi, the mpirun wrapper may not pass --tv
properly, so explicit specification of mpiexec.hydra is needed.
For openmpi, --tv
or --debug
is not needed (they are synonyms in openmpi), and the mpirun wrapper should work. So just tvconnect $(which mpirun) -n $NPROC
Ensure the totalview module is loaded in your PBS job also. And give your job a longer walltime.
Running Totalview
On gadi, load the totalview module, and launch totalview. Check under the file menu that it is looking for reverse connections.
Once your job begins on gadi, totalview should give you a prompt to connect to the job. Once you have done so, hit "go" (green play button). It should then ask you what you want to do about starting a parallel job. Say Yes, and it will stop, (or No for it to continue running). Then you can search for source files and put in break points, then hit play again. Or just let it run until it fails.
Useful pages
- https://wikis.uni-paderborn.de/pc2doc/Noctua-Software-TotalView (someone else's wiki on using Totalview)
- Totalview help on reverse connections
- https://opus.nci.org.au/display/Help/Totalview (NCI help on Totalview)
Using DDT
A similar approach with reverse connections also works with the DDT debugger. In the suite set
ROSE_LAUNCHER = ddt
ROSE_LAUNCHER_PREOPTS = --connect mpirun -n $NPROC
and load the arm-forge module.
Start ddt on gadi and wait for the reverse connection message.