gilgamesh - ntpl/ntpl GitHub Wiki
running jobs on other machines will usually be similar to gilgamesh.
##job dependancies
https://marylou.byu.edu/documentation/pbs/advanced
http://www.nics.tennessee.edu/node/421
http://beige.ucs.indiana.edu/I590/node45.html
http://web.mit.edu/longjobs/www/faq.html
##job arrays
https://wiki.hpcc.msu.edu/display/hpccdocs/Advanced+Scripting+Using+PBS+Environment+Variables
http://www.clusterresources.com/torquedocs21/2.1jobsubmission.shtml
##kill all user jobs
http://docs.notur.no/uit/stallo_documentation/faq/how-can-i-kill-all-my-jobs
#lammps compile
Important steps:
- make sure to issue a:
module load openmpi-psm-gcc
before building LAMMPS
- do a:
make no-kspace
-
use my makefile (see below) which simply uses mpiCC to build LAMMPS. The "module load openmpi-psm-gcc" command will make sure the proper mpiCC gets used.
-
issue the qsub command with a "cput" parameter along with the usual "nodes:ppn" directives. e.g. "qsub -l cput=32:00:00 -l nodes=2:ppn=16 pbs.sh" will run a job with a cputime limit of 32 hours.
###job script
---------------- pbs.sh ---------------------
#!/bin/bash
cd $PBS_O_WORKDIR
module load openmpi-psm-gcc
mpirun -np `cat $PBS_NODEFILE | wc -l` ./lmp_generic < lammpsInputScript.txt > output
-------------------------------------------------
###makefile
-------------------- src/MAKE/Makefile.generic--------------
# generic mpiCC-based build user should make sure mpiCC points to the correct place
SHELL = /bin/sh
#.IGNORE:
# System-specific settings
CC = mpiCC
CCFLAGS = -g -O -DFFT_FFTW -DGZIP -DMPICH_IGNORE_CXX_SEEK
DEPFLAGS = -M
LINK = mpiCC
LINKFLAGS = -g -O
#USRLIB = -lfftw -lmpich
SYSLIB = -lpthread
ARCHIVE = ar
ARFLAGS = -rc
SIZE = size
# Link target
$(EXE): $(OBJ)
$(LINK) $(LINKFLAGS) $(OBJ) $(USRLIB) $(SYSLIB) -o $(EXE)
$(SIZE) $(EXE)
# Library target
lib: $(OBJ)
$(ARCHIVE) $(ARFLAGS) $(EXE) $(OBJ)
# Compilation rules
%.o:%.cpp
$(CC) $(CCFLAGS) -c $<
%.d:%.cpp
$(CC) $(CCFLAGS) $(DEPFLAGS) $< > $@
# Individual dependencies
DEPENDS = $(OBJ:.o=.d)
include $(DEPENDS)
-----------------------------------
#gulp compile
/opt/mcgaughey/
module unload openmpi/intel_qlc
module load openmpi-psm-gcc
my .bashrc is based off the one john has on the gilgamesh website, do you kno that site?
http://gilgamesh.cheme.cmu.edu/doc/gilgamesh.html
[jason@gilgamesh Src]$ which mpirun
/opt/openmpi-1.4.2/bin/mpirun
[jason@gilgamesh Src]$ which g95
/opt/mcgaugheygroup/g95/g95-install/bin/g95
[jason@gilgamesh Src]$ which gfortran
/usr/bin/gfortran
[jason@gilgamesh Src]$ which ifort
/opt/intel/Compiler/11.1/072/bin/intel64/ifort
[14:32:03] Jason Larkin: so, we want to switch the module to
module load intel/intel64
module load openmpi/intel_qlc
[14:32:48] Jason Larkin: see the change it made to mpirun
[jason@gilgamesh Src]$ which mpirun
/usr/mpi/intel/openmpi-1.4-qlc/bin/mpirun
[jason@gilgamesh Src]$ which g95
/opt/mcgaugheygroup/g95/g95-install/bin/g95
[jason@gilgamesh Src]$ which gfortran
/usr/bin/gfortran
[jason@gilgamesh Src]$ which ifort
/opt/intel/Compiler/11.1/072/bin/intel64/ifort
[14:32:52] Jason Larkin: now we can try and compile
[14:36:45] Jason Larkin: ok with standard getmachine you get folloing error:
[jason@gilgamesh Src]$ make
/bin/sh getmachine gulp
make[1]: Entering directory `/opt/mcgaugheygroup/gulp-4.0.5/Src/Linux'
g95 -O -I.. -DFLUSH -c ../datatypes.F90
g95 -O -I.. -DFLUSH -c ../modules.f90
g95 -O -I.. -DFLUSH -c ../iomod.f90
g95 -O -I.. -DFLUSH -c ../itcom.f90
g95 -O -I.. -DFLUSH -c ../reallocate.f90
/tmp/ccQD1io3.s: Assembler messages:
/tmp/ccQD1io3.s:11: Error: suffix or operands invalid for `push'
/tmp/ccQD1io3.s:13: Error: suffix or operands invalid for `push'
[14:36:55] Jason Larkin: which is realted to 32 vs 64 bit
http://www.techblogistech.com/2011/08/error-suffix-or-operands-invalid-for-push-or-pop/
[14:38:23] Jason Larkin: I have a compile at
/home/jason/GULP/gulp.4.0.5/gulp-4.0.5/Src
[14:38:31] Jason Larkin: so we'll copy that getmachine over
[14:38:57] Jason Larkin: [jason@gilgamesh Src]$ cp /home/jason/GULP/gulp.4.0.5/gulp-4.0.5/Src/getmachine /opt/mcgaugheygroup/gulp-4.0.5/Src/
[jason@gilgamesh Src]$ make
/bin/sh getmachine gulp
make[1]: Entering directory `/opt/mcgaugheygroup/gulp-4.0.5/Src/Linux'
ifort -O -I.. -DFLUSH -c ../datatypes.F90
[14:39:23] Jason Larkin: ya, that getmachine is setup to compile usign ifort
[15:09:54] Jason Larkin: lets call it "gulp_ifort"
[15:10:05] Jason Larkin: i have another there called "gulp_jason" this is the version compiled on my laptop
[15:10:36] Jason Larkin: give it a shot
[15:10:44] Jason Larkin: /opt/mcgaugheygroup/gulp-4.0.5/Src/gulp_ifort