setup wmcore unittest - dmwm/WMCore GitHub Wiki

Brief Summary (for Linux)

  • WMCore unittests requires setting up a RDBMS (MariaDB or Oracle) and CouchDB, in addition to many python libraries. The easiest way to get the full stack is through the deployment of the WMAgent RPM from the cmsrep repository.
  • This tutorial assumes you have the proper grid certificates available (for external service communication)
  • This tutorial assumes you also have the git client installed.
  • It also assumes you have Condor installed on the same node, even though it's not mandatory to have it. Some unit tests require Condor to run properly, e.g. BossAir_t module
  • Deployment of wmagentpy3-dev rpm is only used for setting up external sources (including nose). Actual WMCore code will be checked out from git and linked with symbolic link.
  • deploy_unittest_py3.sh will do all the work for you. There is no need to deploy anything in advance.

Initial node setup and/or creation

You can choose between two different resources/hardware:

  1. create an openstack VM on CERN OpenStack, or
  2. deploying docker on your local (or a remote machine). See section 1.1 for further details.

If you create an openstack VM, mind the following recommended settings:

  • flavor: m2.medium
  • boot source: from the latest CC7 Base - x86_64 image.

Set up for WMCore Unittest

1. Using the WMAgent docker container

Prerequisite: Download docker https://www.docker.com/products/overview/

Run the docker container which contains a CC7 distribution, all our software, and some extras. First you need to pull the WMAgent image from the CERN Gitlab Registry:

## you might have to first login to the registry (your CLI password is available under User Profile --> CLI Secret
docker login gitlab-registry.cern.ch

## then you can pull the latest image
docker pull gitlab-registry.cern.ch/cmsdocks/dmwm:wmcorepy3_tests

Now that you have the image in your local machine, you can run it with the command below. But notice that you will need to mount your grid user credentials (usercert/userkey pem files) and your WMCore remote clone, for a seamless integration with your development environment.

export LOCAL_DIR=blah
cd $LOCAL_DIR
git clone https://github.com/dmwm/WMCore.git

## note that the command below assumes that your grid certificate is available under $LOCAL_DIR/certs
docker run  -v $LOCAL_DIR/certs:/home/dmwm/certs -v $LOCAL_DIR/WMCore:/home/dmwm/wmcore_unittest/WMCore -it  --entrypoint /bin/bash gitlab-registry.cern.ch/cmsdocks/dmwm:wmcorepy3_tests

This will download the container image from Docker Hub and start it. You don't have to "get" the image yourself if you don't want to, but without the docker pull command it will not be kept up to date.

The first -v option maps a directory on the local host (~/DMWM/certs) to a certs/ directory in the container which is presumed to contain the servicekey.pem and servicecert.pem from step 6 below.

While the container comes with a version of our code (current at the time of creation), the 2nd -v option above mounts code from your host machine into the container making it easy to use your editor.

Setup the environment, start couch and MySQL:

source env_unittest_py3.sh 
$manage start-services

Carry on. You will find some useful scripts in ContainerScripts/ including one to run a slice of the unit tests. There is also an update git script that is used to update the code inside the container to the very latest master (this assumes you omitted the 2nd -v option above).

See https://github.com/dmwm/WMCore/wiki/Docker-and-WMCore for a discussion on how these containers were created.

PS.: If your docker container run out of space, you might have to run the following command in order to make some space by deleting old containers: docker system prune -a -f

2. Alternative: using an OpenStack VM to deploy services

First we need to download the script to set up unittests (in your current working directory).

$ curl https://raw.githubusercontent.com/dmwm/WMCore/master/test/deploy/deploy_unittest_py3.sh > deploy_unittest_py3.sh

The command below will deploy the latest wmagentpy3-dev version available in cmsrep comp repository. It will also update the source code to master HEAD. You can also change the WMA_TAG inside the script to the WMAgent version you want to deploy.

$ sh deploy_unittest_py3.sh -r dmwm -b master

Alternatively, you can specify a wmagentpy3-dev version (-v), the git organization (-r) and the branch to pull the source code from (-b)

$ sh deploy_unittest_py3.sh -v 1.0.16.pre2 -r dmwm -b master

If everything goes well, both MariaDB and CouchDB should be up and running. You can check it with:

source env_unittest_py3.sh 
$manage status

PS.: In case the WMAgent deployment fails in the installation step (sw), you might need to manually install one RPM dependency: bzip2 tk perl-ExtUtils-Embed compat-libstdc++-33 libXmu libXcursor libXrandr libXft mesa-libGLU libXi libXinerama libXft-devel libXpm perl-Test-Harness libX11-devel libXpm-devel libXext-devel mesa-libGLU-devel perl-Switch perl-Env perl-Thread-Queue

Just a FYI, the "unittestdeploy" directory will contain the RPM content while "wmcore_unittest" directory contains the content as checked out from git.

Executing WMCore Unittest

3.1 Run a simple unittest

This example test spec which doesn't use db.

$ cd $TEST_DIR/WMCore/test/python/WMCore_t/WMSpec_t

$ python3 WMStep_t.py

3.2 Run a simple unittest with nose

For more info about nose, see http://nose.readthedocs.org/en/latest/usage.html

$ cd $TEST_DIR/WMCore/test/python/WMCore_t

$ nosetests WMSpec_t

3.3 Run a single test with nose

Like above, but say you just want to run the one test you are trying to fix:

$ cd $TEST_DIR/WMCore/test/python/WMCore_t/WMSpec_t/StdSpecs_t

$ nosetests TaskChain_t.py:TaskChainTests.testMultithreadedTasksTaskChain

where before the : you have the file name, after the : the class and function names.

3.4 Running several unittests

You can also select all the unit tests under the patch to run.

$ cd $TEST_DIR/WMCore/

$ python3 setup.py test --buildBotMode=true --reallyDeleteMyDatabaseAfterEveryTest=true --testCertainPath=test/python/WMCore_t/WMSpec_t/StdSpecs_t/

3.5 Running tests like Jenkins

code is the directory where the checkout was done. SLICES is the number of slices you want create for all the unittests (jenkins use 10). SLICE is the slice number you want to run, usually it's the same SLICE that failed in Jenkins.

$ python3 $TEST_DIR/WMCore/setup.py test --buildBotMode=true --reallyDeleteMyDatabaseAfterEveryTest=true --testCertainPath=$TEST_DIR/WMCore/test/python/ --testTotalSlices=$SLICES --testCurrentSlice=$SLICE

Miscellaneous

4. Recovering from a corrupt database

If you screw up the underlying MySQL/MariaDB database, you may need to drop and recreate it:

$ mysql -u unittestagent -p --sock $INSTALL_DIR/current/install/mysql/logs/mysql.sock

and at the DB prompt type
drop database wmcore_unittest; create database wmcore_unittest;

5. Authenticate yourself when running unittests

Some unittests check the inetraction between WMCore code and external existing services. In order to run such tests, it is necessary that the user authenticates himself. There are two ways of authenticating. You can either use a proxy file, or you can use certificates files. Some unittests will work with either, while some require to have certificate files (such as DQMUpload_t).

BEWARE: The files you will deal with in the following commands can be used to impersonate you if they fall into the wrong hands! handle them with caution.

Start by setting up the environment in the container or VM with:

[@dockercontainer ~]$ source env_unittest_py3.sh`
[@dockercontainer ~]$ $manage start-services  # this is not compulsory, but if you need it, you can do it now

Make sure that host/user certificates are ok (i.e. if have been loaded when sourcing env_unittest_py3.sh), checking that the following env variables point to existing files:

[@dockercontainer ~]$ ls $X509_HOST_CERT 
[@dockercontainer ~]$ ls $X509_HOST_KEY  
[@dockercontainer ~]$ ls $X509_USER_CERT 
[@dockercontainer ~]$ ls $X509_USER_KEY

If not, you can manually source the new locations of your certificates:

  • I assume that you run the docker container with docker run -v ${CERT_DIR}:/home/dmwm/certs-mount, where ${CERT_DIR} is the directory that you keep you certificates in your laptop. The same steps applies in a VM, you just have to adjust the paths.
  • get a .p12 Grid certificate file from cern account service, save it to ${CERT_DIR} on your laptop. -[@dockercontainer ~]$ openssl pkcs12 -in ~/certs-mount/2021_myCertificate.p12 -clcerts -nokeys -out ~/certs/usercert.pem: (enter your grid certificate pwd)
  • [@dockercontainer ~]$ openssl pkcs12 -in ~/certs-mount/2021_myCertificate.p12 -nocerts -out ~/certs/userkey.pem : unlock the grid certificate inserting your grid certificate pwd, enter two times a new pwd for protecting your private key userkey.pem.
  • [@dockercontainer ~]$ chmod 400 ~/certs/userkey.pem
  • [@dockercontainer ~]$ openssl rsa -in ~/certs/userkey.pem -out ~/certs/userkey_nopwd.pem: enter the pwd of your private key userkey.pem to get a passwordless private key. This allows you to run unittests without inserting the password protecting the certificate key every time that it is used by a unittest. Beware: this file can be used to impersonate you!
  • [@dockercontainer ~]$ chmod 400 ~/certs/userkey_nopwd.pem
  • [@dockercontainer ~]$ export X509_HOST_CERT=/home/dmwm/certs/usercert.pem
  • [@dockercontainer ~]$ export X509_HOST_KEY=/home/dmwm/certs/userkey_nopwd.pem
  • [@dockercontainer ~]$ export X509_USER_CERT=/home/dmwm/certs/usercert.pem
  • [@dockercontainer ~]$ export X509_USER_KEY=/home/dmwm/certs/userkey_nopwd.pem
  • annotation: I export the grid certificate into a directory that lives only in the docker filesystem, not in a dicrectory that is mounted into the host. this way i am sure that when i exit the container the --rm parameter above will remove the passwordless private key from my system

Instead of dealing with certificates, a faster way of authenticating yourself is using a proxy file, but this does not work for every unittest. I suggest trying this out first, then if the unittest fail try using the certificates. Using a proxy file means providing into the docker container at the path /tmp/x509up_u1000 (similar statements apply in the case of a VM) the proxy file you get in lxplus when you run voms-proxy-init -voms cms with (the proxy expires after ~1day, so you may need to repeat this procedure even if you do not exit the docker container)

  • [@laptop ~]$ ssh -XY lxplus.cern.ch
  • [@lxplus ~]$ voms-proxy-init -voms cms
  • [@laptop ~]$ scp lxplus.cern.ch:/tmp/x509up_u${UID} ${CERT_DIR} use the same ${CERT_DIR} as above, set ${UID} to your lxplus user id. Anyway, voms-proxy should tell you the path where it created the proxy file.
  • [@dockercontainer ~]$ cp ~/certs-mount/x509up_u99307 /tmp/x509up_u1000
  • This is it. This line will load your proxy file without requiring any env variable.

6. Checking code quality

We use pylint to automatically analyze our code for errors and bad coding practice. We keep our own standards file which allows us to use camel case for variables, etc. To run a report, execute a command like this:

$ pylint --rcfile=$TEST_DIR/WMCore/standards/.pylintrc some_python_file_or_directory/

Most of the codes are self explanatory, but there is more help available for some at http://pylint-messages.wikidot.com/all-codes

7. Shutdown mysql and couch server

$ $manage stop-services

8. Update the current code base.

Once unittest is set you don't have go through the first step for new update. Assume everything was set up before, go to the directory where the initial setup was made. (-u for update, -r for git repository name, -b for branch name)

$ sh deploy_unittest_py3.sh -u -r dmwm -b master

9. In case your unittests need to access central services:

In order to contact central services (DBS, PhEDEx, etc), we need to also create a proxy and copy this file into (example):

$ cp myproxy.pem $TEST_DIR../certs/servicecert.pem

$ cp myproxy.pem $TEST_DIR../certs/servicekey.pem

Alternatively you can use your certificate and key directly (however, the key needs to be unencrypted so above option is recommended although you need to renew the proxy often).

To create the cert and unencrypted key (with -nodes option) from p12 file.

$ openssl pkcs12 -in myCert.p12 -clcerts -out $TEST_DIR../certs/servicecert.pem -nokeys

$ openssl pkcs12 -in myCert.p12 -out $TEST_DIR../certs/servicekey.pem -nocerts -nodes

and change the cert, key file permission

$ chmod 600 $TEST_DIR../certs/servicecert.pem

$ chmod 400 $TEST_DIR../certs/servicekey.pem

10. Using git bisect to find which commit breaks a particular test

First, take a look at https://www.kernel.org/pub/software/scm/git/docs/git-bisect.html which is full documentation for git bisect. Strangely, git bisect has to be run from the root directory of the git tree, so you will see all my commands have (cd blah prepended to them.

Verify a patch is broken in master: $ git checkout master $ nosetests MonteCarlo_t:MonteCarloTestCase.testLHEProductionWorkload Ok, it's broken. Let's try to find a place it's still working. Because the testing infrastructure may have changed, it doesn't make sense to go as far back as possible, so let's try going back in 1-year steps: $ git checkout `git rev-list -n 1 --before="2015-01-01 00:00" master Here the test is OK, so let's tell git bisect this:

$ git checkout master
$ (cd /storage/local/data1/ewv/wmcore_unittest/WMCore; git bisect start)
$ (cd /storage/local/data1/ewv/wmcore_unittest/WMCore; git bisect bad)
$ git checkout `git rev-list -n 1 --before="2015-01-01 00:00" master`
Note: checking out '267b8164a534ea3bf7b649ac122f0bc4f5e86961'.
$ (cd /storage/local/data1/ewv/wmcore_unittest/WMCore; git bisect good)
Bisecting: 400 revisions left to test after this (roughly 9 steps)
error: You have local changes to 'src/couchapps/AlertsCollector/vendor/couchapp/_attachments/jquery.js'; cannot switch branches.

Ok, now the checkout failed to switch because of local changes. This is probably because git is changing end-of-line characters on me. Change your config settings with git config core.autocrlf false, do a force checkout of the last commit mentioned in git bisect log, and carry on.

Set up for Python2 WMCore Unittest with docker (DEPRECATED)

This section explains how to get the Docker image for a Python2 WMCore stack, which is very similar to the instructions_above.

Pull and run the Python2-based WMCore container with:

docker pull gitlab-registry.cern.ch/cmsdocks/dmwm:wmcore_tests
docker run -v $LOCAL_DIR/certs:/home/dmwm/certs -v $LOCAL_DIR/WMCore:/home/dmwm/wmcore_unittest/WMCore -it  --entrypoint /bin/bash gitlab-registry.cern.ch/cmsdocks/dmwm:wmcore_tests

Next, you need to source the python3 WMCore environment with:

source env_unittest.sh

now you can start services (CouchDB and MySQL):

$manage start-services

and you are ready to run the unit tests - with python2 - for a single module, e.g.:

cd $TEST_DIR/WMCore/
nosetests test/python/Utils_t/FileTools_t.py 
....
----------------------------------------------------------------------
Ran 4 tests in 0.061s

OK

alternatively, you can run those unit tests as they are executed in Jenkins. So, running a full slice out of 10, e.g.:

cd $TEST_DIR/WMCore/
python setup.py test --buildBotMode=true --reallyDeleteMyDatabaseAfterEveryTest=true --testCertainPath=test/python --testTotalSlices=10 --testCurrentSlice=2

Set up for WMCore Unittest with Oracle (outdated)

It follows basically the same procedure as above. However, we need to make a few local changes to 3 files before we run the deployment.

  1. create a new oracle account NewOracleAccount

  2. WMAgent_unittest.secrets file: replace MYSQL by the ORACLE credentials, for instance:

ORACLE_USER=(write your oracle user name)
ORACLE_PASS=(write your oracle password)
ORACLE_TNS=(write your tns name, e.g. INT2R_NOLB)
  1. env_unittest_py3.sh file: remove the DBSOCK line; update the DATABASE (you need to provide the actual values, it does not get automatically filled from the secrets file) and the DIALECT (Oracle) variables. Diff would be something like:
amaltaro@alan-wmacloud:/data/amaltaro/WMAgent $ diff -rup env_unittest_py3.sh.ori env_unittest_py3.sh | colordiff
-export DBSOCK=$INSTALL_DIR/current/install/mysql/logs/mysql.sock
 
-export DATABASE=mysql://unittestagent:passwd@localhost/wmcore_unittest
+export DATABASE=oracle://ORACLE_USER:ORACLE_PASS@ORACLE_TNS
-export DIALECT=MySQL
+export DIALECT=Oracle
  1. deploy_unittest_py3.sh: before you run this script, make sure to first comment out the mysql database creation line; als comment out the line downloading the env_unittest_py3.sh and WMAgent_unittest.secrets files, since you do not want them to get overwritten. Diff would be something like:
amaltaro@alan-wmacloud:/data/amaltaro/WMAgent $ diff -rup deploy_unittest_py3.sh.ori deploy_unittest_py3.sh | colordiff
-    curl -s https://raw.githubusercontent.com/dmwm/WMCore/master/test/deploy/env_unittest_py3.sh > env_unittest_py3.sh
-    curl -s https://raw.githubusercontent.com/dmwm/WMCore/master/test/deploy/WMAgent_unittest.secrets > WMAgent_unittest.secrets
+    #curl -s https://raw.githubusercontent.com/dmwm/WMCore/master/test/deploy/env_unittest_py3.sh > env_unittest_py3.sh
+    #curl -s https://raw.githubusercontent.com/dmwm/WMCore/master/test/deploy/WMAgent_unittest.secrets > WMAgent_unittest.secrets
 
-    mysql -u unittestagent --password=passwd --sock $INSTALL_DIR/current/install/mysql/logs/mysql.sock --exec "create database wmcore_unittest"
+#    echo "--- when prompted type 'passwd' (without ')"
+#    mysql -u unittestagent -p --sock $INSTALL_DIR/current/install/mysql/logs/mysql.sock --exec "create database wmcore_unittest"

Troubleshooting

  • No space left on device

If you get following error in Mac

[dmwm@486dbd7e0a4e ~]$ $manage start-services

Starting Services...
starting couch...
/home/dmwm/unittestdeploy/wmagent/1.1.10.pre7/sw/slc7_amd64_gcc630/external/couchdb15/1.6.1-comp/bin/couchdb: line 131: cannot create temp file for here-document: No space left on device

It can be resolved with the following command (as suggested in this link):

rm  ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
  • docker.sock permission denied

If you fail to run docker login or docker pull with the following error:

dial unix /var/run/docker.sock: connect: permission denied

it can be resolved by changing the file permission such as:

sudo chmod 666 /var/run/docker.sock

Useful scripts

Convert the unittest identifier that Jenkins provide, such as WMCore_t.JobSplitting_t.RunBased_t.RunBasedTest:testMultipleRunsCombine, to a path that can be passed to nosetests, such as test/python/WMCore_t/JobSplitting_t/RunBased_t.py:RunBasedTest.testMultipleRunsCombine with the script

#!/usr/bin/python3

# add ~/.local/bin to $PATH
# cd  ~/.local/bin
# ln -s path/to/convert_unittest_to_path.py convert_unittest_to_path.py

import sys
import os

ut = sys.argv[1]
ut = ut.split(".")
ut[-2] = ut[-2] + ".py"
tmp = ut[-1].replace(":", ".")
ut.pop(-1)
ut[-1] = ut[-1] + ":" + tmp
ut = "/".join(ut)
ut = os.path.join("test/python", ut)

print(ut)