gilgamesh - ntpl/ntpl GitHub Wiki

running jobs on other machines will usually be similar to gilgamesh.

##job dependancies

https://marylou.byu.edu/documentation/pbs/advanced

http://www.nics.tennessee.edu/node/421

http://beige.ucs.indiana.edu/I590/node45.html

http://web.mit.edu/longjobs/www/faq.html

##job arrays

https://wiki.hpcc.msu.edu/display/hpccdocs/Advanced+Scripting+Using+PBS+Environment+Variables

http://www.clusterresources.com/torquedocs21/2.1jobsubmission.shtml

##kill all user jobs

http://docs.notur.no/uit/stallo_documentation/faq/how-can-i-kill-all-my-jobs

#lammps compile

Important steps:

  1. make sure to issue a:

module load openmpi-psm-gcc

before building LAMMPS

  1. do a:

make no-kspace

  1. use my makefile (see below) which simply uses mpiCC to build LAMMPS. The "module load openmpi-psm-gcc" command will make sure the proper mpiCC gets used.

  2. issue the qsub command with a "cput" parameter along with the usual "nodes:ppn" directives. e.g. "qsub -l cput=32:00:00 -l nodes=2:ppn=16 pbs.sh" will run a job with a cputime limit of 32 hours.

###job script

---------------- pbs.sh ---------------------
#!/bin/bash
cd $PBS_O_WORKDIR
module load openmpi-psm-gcc
mpirun -np `cat $PBS_NODEFILE | wc -l` ./lmp_generic < lammpsInputScript.txt > output
-------------------------------------------------

###makefile

-------------------- src/MAKE/Makefile.generic--------------
# generic mpiCC-based build user should make sure mpiCC points to the correct place

SHELL = /bin/sh
#.IGNORE:

# System-specific settings

CC =    mpiCC
CCFLAGS =       -g -O -DFFT_FFTW -DGZIP -DMPICH_IGNORE_CXX_SEEK
DEPFLAGS =      -M
LINK =          mpiCC
LINKFLAGS =     -g -O
#USRLIB =       -lfftw -lmpich
SYSLIB =        -lpthread
ARCHIVE =       ar
ARFLAGS =       -rc
SIZE =          size

# Link target

$(EXE): $(OBJ)
        $(LINK) $(LINKFLAGS) $(OBJ) $(USRLIB) $(SYSLIB) -o $(EXE)
        $(SIZE) $(EXE)

# Library target

lib:    $(OBJ)
        $(ARCHIVE) $(ARFLAGS) $(EXE) $(OBJ)

# Compilation rules

%.o:%.cpp
        $(CC) $(CCFLAGS) -c $<

%.d:%.cpp
        $(CC) $(CCFLAGS) $(DEPFLAGS) $< > $@

# Individual dependencies

DEPENDS = $(OBJ:.o=.d)
include $(DEPENDS)
-----------------------------------

#gulp compile

/opt/mcgaughey/

module unload openmpi/intel_qlc
module load openmpi-psm-gcc

my .bashrc is based off the one john has on the gilgamesh website, do you kno that site?

http://gilgamesh.cheme.cmu.edu/doc/gilgamesh.html

[jason@gilgamesh Src]$ which mpirun
/opt/openmpi-1.4.2/bin/mpirun
[jason@gilgamesh Src]$ which g95
/opt/mcgaugheygroup/g95/g95-install/bin/g95
[jason@gilgamesh Src]$ which gfortran
/usr/bin/gfortran
[jason@gilgamesh Src]$ which ifort
/opt/intel/Compiler/11.1/072/bin/intel64/ifort
[14:32:03] Jason Larkin: so, we want to switch the module to

module load intel/intel64
module load openmpi/intel_qlc
[14:32:48] Jason Larkin: see the change it made to mpirun

[jason@gilgamesh Src]$ which mpirun
/usr/mpi/intel/openmpi-1.4-qlc/bin/mpirun
[jason@gilgamesh Src]$ which g95
/opt/mcgaugheygroup/g95/g95-install/bin/g95
[jason@gilgamesh Src]$ which gfortran
/usr/bin/gfortran
[jason@gilgamesh Src]$ which ifort
/opt/intel/Compiler/11.1/072/bin/intel64/ifort
[14:32:52] Jason Larkin: now we can try and compile
[14:36:45] Jason Larkin: ok with standard getmachine you get folloing error:

[jason@gilgamesh Src]$ make
/bin/sh getmachine gulp
make[1]: Entering directory `/opt/mcgaugheygroup/gulp-4.0.5/Src/Linux'
g95  -O  -I..   -DFLUSH  -c ../datatypes.F90
g95  -O  -I..   -DFLUSH  -c ../modules.f90
g95  -O  -I..   -DFLUSH  -c ../iomod.f90
g95  -O  -I..   -DFLUSH  -c ../itcom.f90
g95  -O  -I..   -DFLUSH  -c ../reallocate.f90
/tmp/ccQD1io3.s: Assembler messages:
/tmp/ccQD1io3.s:11: Error: suffix or operands invalid for `push'
/tmp/ccQD1io3.s:13: Error: suffix or operands invalid for `push'
[14:36:55] Jason Larkin: which is realted to 32 vs 64 bit

http://www.techblogistech.com/2011/08/error-suffix-or-operands-invalid-for-push-or-pop/

[14:38:23] Jason Larkin: I have a compile at

/home/jason/GULP/gulp.4.0.5/gulp-4.0.5/Src
[14:38:31] Jason Larkin: so we'll copy that getmachine over
[14:38:57] Jason Larkin: [jason@gilgamesh Src]$ cp /home/jason/GULP/gulp.4.0.5/gulp-4.0.5/Src/getmachine /opt/mcgaugheygroup/gulp-4.0.5/Src/

[jason@gilgamesh Src]$ make
/bin/sh getmachine gulp
make[1]: Entering directory `/opt/mcgaugheygroup/gulp-4.0.5/Src/Linux'
ifort  -O  -I..   -DFLUSH  -c ../datatypes.F90
[14:39:23] Jason Larkin: ya, that getmachine is setup to compile usign ifort

[15:09:54] Jason Larkin: lets call it "gulp_ifort"
[15:10:05] Jason Larkin: i have another there called "gulp_jason" this is the version compiled on my laptop
[15:10:36] Jason Larkin: give it a shot
[15:10:44] Jason Larkin: /opt/mcgaugheygroup/gulp-4.0.5/Src/gulp_ifort