Overview

The following section lists the different python versions used at runtime when running a job.

The main job executable bash script

All jobs run a bash script called submit.sh. Here, a SCRAM architecture is set (currently hardcoded):

https://github.com/dmwm/WMCore/blob/master/etc/submit.sh#L51

This SCRAM architecture is used to find the latest python version in the COMP repository inside CVMFS. Since this is currently fixed, we always get the following version at present:

/cvmfs/cms.cern.ch/COMP/slc6_amd64_gcc493/external/python/2.7.13/bin/python

Note we use slc6, but WMAgent supports RHEL6 and RHEL7 workflows. This python version somehow works with CC7 containers too, which is RHEL7 workflows can still run:

[khurtado@earth ~]$ cmssw-cc7
Singularity> /cvmfs/cms.cern.ch/COMP/slc6_amd64_gcc493/external/python/2.7.13/bin/python
Python 2.7.5 (default, Apr  2 2020, 13:16:51)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

You've got color, history, and pretty printing.
(If your ~/.inputrc doesn't suck, you've also
got completion and vi-mode keybindings.)

>>> platform.dist()
('centos', '7.8.2003', 'Core')
>>>

In this script, the job object is unpacked via the Unpacker.py runtime script (running with that python version)

• This creates job directory, unpacks the sandbox and setups environment so job can be called using the Startup runtime script, which will also use the same python version. E.g.:

/cvmfs/cms.cern.ch/COMP/slc6_amd64_gcc493/external/python/2.7.13/bin/python Startup.py

The CMSSW executor prescripts

Startup.py above bootstraps the job and executes it

• The execution method will look and use the proper executor to run here (e.g.: the CMSSW executor)

For a CMSSW executor, prescripts are added here and looked here, being SetupCMSSWPset the only one added by this method at present.

The SetupCMSSWPset.py runtime script is called via another script: ScriptInvoke. The python version used here set in Scram.py.

https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMRuntime/Tools/Scram.py#L384-L394

One could easily add a block above to run e.g.: python3.6 when available. The python version coming from Scram.py is stored in sys.executable and called HERE

The command invoked looks like this:

/cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_11_patch1/external/slc7_amd64_gcc700/bin/python2.7 -m WMCore.WMRuntime.ScriptInvoke WMTaskSpace.cmsRun1 SetupCMSSWPset

cmsRun executable

To run the actual payload, a bash script with some arguments is generated and executed. E.g.:

INFO:root:Executing CMSSW. args: ['/bin/bash', '/tmpscratch/users/khurtado/work/job/WMTaskSpace/cmsRun1/cmsRun1-main.sh', '', u'slc7_amd64_gcc700', 'scramv1', 'CMSSW', 'CMSSW_10_6_11_patch1', 'FrameworkJobReport.xml', 'cmsRun', 'PSet.py', '', '', '']

Here, the arguments passed represent the following:

#  cmsRun arguments
 SCRAM_SETUP=$1
SCRAM_ARCHIT=$2
SCRAM_COMMAND=$3
SCRAM_PROJECT=$4
CMSSW_VERSION=$5
JOB_REPORT=$6
EXECUTABLE=$7
CONFIGURATION=$8
USER_TARBALL=$9

Where EXECUTABLE is usually: cmsRun, which will run on python2. To run on python3, cmsRunPython3 can be called instead (note this will depend on whether the CMSSW framework release has that available or not)

The following PR shows where to replace that: https://github.com/dmwm/WMCore/pull/9599/files

Questions: How do we verify cmsRunPyhon3 is available? Can any Pset run on cmsRun or cmsRunPython3 (when available)?