Developing against WMCore - dmwm/WMCore GitHub Wiki

Setting up your git repositories

We use a workflow very similar to the one explained here: https://github.com/sevntu-checkstyle/sevntu.checkstyle/wiki/Development-workflow-with-Git:-Fork,-Branching,-Commits,-and-Pull-Request

  1. Create a GitHub account and ask to be made a part of the DMWM Contributors team
  2. Fork the WMCore repository (visit https://github.com/dmwm/WMCore and click fork on the upper right side)
  3. Clone the WMCore repository: git clone https://github.com/USERNAME/WMCore.git (You can find the URL in the text box on your page.
  4. Configure the DMWM version of the code as your first remote:
    • cd WMCore
    • git remote add upstream https://github.com/dmwm/WMCore.git (you don't need to call it upstream, but this is convention) (you can also add any user there if you want to follow what others are doing)

Starting a new development

It's always best to start development against the latest master tag if you can. Follow these steps

  1. Update your local and personal GitHub code to the latest master. You can do this any time, not just when you start development
    • git checkout master
    • git fetch upstream - this gets all the references from the DMWM branch but doesn't change anything about what you have locally
    • git merge upstream/master - this merges the changes in DMWM into your local repository
    • git push origin master - this completes the triangle so that DMWM, your GitHub, and your local all have the same master If you want to directly update your master branch according to the latest central WMCore developments, you can alternatively run:
    • git pull upstream master
    • git push origin master
  2. Begin a new development branch
    • git checkout master - you probably are already here, but in case not...
    • git checkout -b my_feature - this creates a new branch where you will fix something
  3. Hack away to your heart's content
    • git add for each file you change
    • git commit -m "This patch fixes all the bugs and is awesome"
    • git push origin my_feature
    • Go immediately to https://github.com/dmwm/WMCore and you should see a button to make a new pull request (PR). Click it and tell us why we should include this code.
    • Some will say each PR should be one commit. I don't agree with this. Each commit should be self contained, but a PR should sometimes have more than one. For instance, if you are making logical changes and then cleaning up code, make them separate. Then we can understand the logical changes you are making and the quickly verify "Yup, all they did in the 2nd commit was change a bunch of spaces"
    • Jenkins will check your code and the unit tests. If nothing needs to be changed, go back and start your next development. Otherwise, continue development on your branch with:
      • git checkout my_feature
      • Hack away to your heart's content
      • git add for each file you change
      • git commit -m "This patch REALLY fixes all the bugs and is awesome"
      • git push origin my_feature
      • No need to make another PR, GitHub already knows you updated your branch. Jenkins will run again.
    • Once you've got everything fixed, and if you had to go through multiple rounds of commits, see the section below on squashing commits. Use those instructions to produce a PR of, at most, a few well contained, legible commits.

That's the basic workflow. Remember to keep your master up to date and to never develop on your master branch, only on branches you've created from master with git checkout -b. You can also develop against release branches like 1.0.9_wmagent, but only if you know what you are doing. These branches are used for patches (hot fixes).

What to do when things go wrong

You will notice that some of these recipes show a "force push" which, if you read about git, you will see mentions "don't do this" or at least "don't do this to shared repositories". The key here is that your GH repo is not really shared. You should not, under any circumstances, force push to a repository like DMWM.

Rebasing

As long as you start from a recent "master" branch and complete your work quickly, you may never have to do this. But if your code can't be merged with the current master branch, it will tell you on the PR. In that case you have to rebase your code onto a new master branch.

  1. Follow the instructions for updating your code to the latest master
  2. git checkout my_feature - check out your feature branch (based on some earlier master)
  3. git rebase master - rebase your code onto the master branch
  4. You will almost certainly encounter conflicts doing this. They are clearly marked in the conflicting files. Edit those files to resolve the conflict
  5. git rebase --continue - continue the rebase until the next conflict arises and repeat
  6. git push -f origin my_feature - this overwrites what had been the my_feature branch on your GitHub repository with the new one

Squashing commits

Sometime you will be asked to "squash" your commits. For instance, if you have one commit and then find a bug in that code, you can create another commit that fixes it. It makes sense to squash those two commits into one so that anyone looking at them sees the finished product. You do that with the commands

  1. git checkout my_feature
  2. git rebase -i HEAD~3 which says "I want to do something to rewrite the history of the last 3 commits"

Take a look at http://www.git-scm.com/book/en/v2/Git-Tools-Rewriting-History to see exactly how to use the resulting tool, but keep in mind you can combine (squash), erase, reorder, or even break apart commits in this way. It's a very useful way to turn a chaotic set of commits into something orderly that a reviewer can follow.

Cherry-picking

The last useful tool is git cherry-pick COMMIT_ID. This can be used to copy commits from one branch to another (maybe a patch release and a regular release). Or maybe you accidentally committed commits from two features into one branch, so you can split them apart this way. In any case, this command will take the "patch" from COMMIT_ID and apply it to your current branch and commit it immediately to your current branch.

Useful tools

The Pro Git book: http://www.git-scm.com/book/en/v2 is a great resource.

On a Mac, Eric finds the following tools useful:

  • SourceTree - this gives a visual representation of the git repository and any remotes you have configured. It can also be used to give single button clicks to most all of the commands in the standard workflow
  • PyCharm - This is a full featured IDE for python. Eric uses it mostly as a glorified editor, but one that knows how to reformat code to PEP-8 standards, spot variable typos, etc.
  • MacFusion - mount any SSH-accessible directory as if it was local so that I can edit, with PyCharm, remote files

Integrate WMCore development container with your IDE

VSCode

It is possible to integrate cmsdocks/dmwm:wmcorepy3_tests with VSCode, so that you can for example run a unittest inside the python debugger from vscode gui. The integration is maintained on gitlab.cern.ch/cms-dmwm-test/dmwm-wmcore-docker-dev-personalized (which should be visible after logging in with CERN SSO). It is possible to fully integrate the py3 container only, for example the debugger provided with the py2 container is not recognized by vscode.

I will briefly describe here the required steps, leaving all the details to the README.md of the aforementioned repository.

  1. First you should get a new docker image. You can either download the repo from gitlab and build the image on you machine, or you can get the image with
git clone https://gitlab.cern.ch/cms-dmwm-test/dmwm-wmcore-docker-dev-personalized.git # or ssh URI
docker build -t gitlab-registry.cern.ch/cms-dmwm-test/dmwm-wmcore-docker-dev-personalized/dmwm/wmcorepy3_tests_vscode .
# OR
docker login gitlab-registry.cern.ch # i suggest using a `read_registry` access token, obtained from https://gitlab.cern.ch/-/profile/personal_access_tokens for this step
docker pull gitlab-registry.cern.ch/cms-dmwm-test/dmwm-wmcore-docker-dev-personalized/dmwm/wmcorepy3_tests_vscode
  1. Run the container with (replace ${CERT_DIR} with the directory that you keep you certificates in your laptop)
docker run \
-v ${CERT_DIR}:/home/dmwm/certs-mount \ 
# -v ${CERT_DIR}:/home/dmwm/certs-mount:z # for fedora
-v ${PATH_TO}/github.com/WMCore:/home/dmwm/wmcore_unittest/WMCore \
#  -v ${PATH_TO}/github.com/WMCore:/home/dmwm/wmcore_unittest/WMCore:z \ # for fedora
-it --rm \
--name cms_dmwm_wmcore_py3_vscode \
gitlab-registry.cern.ch/cms-dmwm-test/dmwm-wmcore-docker-dev-personalized/dmwm/wmcorepy3_tests_vscode bash

  1. configure VSCode accordingly

    • Open VSCode, even without specifying any directory on the host
    • press the green button in the lower left corner, then press "attach to running container"
    • go the explorer tab, select a directory inside the running container, i suggest /home/dmwm/wmcore_unittest/WMCore
    • press F1, then remote containers, open attach container config file, then select the configuration file related to the current container. Copy inside the content from containername.json. If it does not start automatically downloading the new extensions, press the green button in the lower left corner, then "close remote connection", then click again the green button, then "attach to running container", select the wmcore docker running container. In any case, if it does not download the extentions, you only need "Pylance", or the current official python extension for VSCode provided bny microsoft. After the extensions have been downloaded, just make sure that the configuration file (which vscode uses to remember how to connect to running docker containers) contains "postAttachCommand": "source /home/dmwm/env_unittest_py3.sh". This line is paramount!
    • copy the content of settings.json into /home/dmwm/wmcore_unittest/WMCore/.vscode/setting.json.

  2. For some unittests you need either/both to start the services or/and to setup your certificates every time you start the docker container. Setup the environment in the container with:

    • @dockercontainer $ source env_unittest_py3.sh: this should already have been done automatically when the container is launched
    • @dockercontainer $ $manage start-services: this is not compulsory, but if you need it, you can do it now
    • If you need to run unittest that require authentication, follow these instructions

What to expect: vscode now should

  • identify the correct version of the python runtime: 3.8.2 under /home/dmwm/unittestdeploy/wmagent/current/...
  • be able to run a unittest *_t.py file as a standalone program in the debugger
  • find all the unittests
  • run a single unittest in the debugger

Close VSCode and exit the container without any worry. When you repeat the procedure the second time, you can skip many of the aforementioned steps: Step 1. is necessary only after the docker image is upgraded, Step 2. is necessary to start the docker container, Step 3. should not be necessary, Step 4. is necessary depending on your needs

Troubleshoot

If vscode does not find unittests, that you can troubleshoot by manually running the following command in a shell attached to the container, after sourcing env_unittest_py.sh:

python3 -m unittest discover --verbose --start-directory test/python --pattern *_t.py

if you encounter the error

ImportError: 'Daemon_t' module incorrectly imported from '/home/dmwm/wmcore_unittest/WMCore/test/python/WMCore_t/Agent_t/Daemon_t'. Expected '/home/dmwm/wmcore_unittest/WMCore/test/python/WMCore_t/Agent_t'. Is this module globally installed?

then you have to remove the file test/python/WMCore_t/Agent_t/Daemon_t/__init__.py.

VSCode should discover all the unittests by itself, but you can also force it to search for them.

Disclaimer

Disclaimer: tested on debian and fedora. If it doesn't work on your system, please feel free to point that out and we can find a solution!