setup wmcore virtual environment - dmwm/WMCore GitHub Wiki

Brief summary for python versions:

While writing the current document we are in the process of migrating to python3 and the information regarding python2.7 may very easily go stale really soon, but I will try to cover both setups as much as I can.

Brief summary on eventual deployment target points:

The so described method bellow is to have your WMCore/WMAgent development virtual environment setup and running, where in addition to the major dependency packages you can also setup all the utilities, IDEs and tools that you find useful in making your development process as comfortable and productive as you wish. Given that this is a virtual environment one may set it up wherever found to be convenient, but the two most probable places would be either the remote CERN infrastructure or the developer's personal PC. There are pros and cons for both approaches:

  • Remote deployment:
    • Con: Because there are still dependencies that exist in WMCore to packages from the OS itself, which may or may not be resolved on the lxplus machines, there is a possibility one to end up using a VM instance from his personal openstack project or a Docker container, so that he has the rights to install the missing packages himself.
    • Con: May require wasting some resources from your personal openstack project at CERN
    • Pro: Your deployment environment lives behind the CERN firewall, which makes a lot of things simpler
    • Pro: You may reach some integration services' REST APIs directly from python and make some single line tests/queries on the go, which speeds things a lot.
    • Pro: You may try to run some of your services directly from the source path where you are doing your development without the need for a full service deployment, just by changing your python source path
    • Pro: You have the ability to patch an already deployed service as it already runs in your VM/Container directly from your development virtual environment (supposed the service is running on the same VM/Container you are deploying your virtual environment at or your virtual environment is hosted on a shared filesystem so that it is accessible from the VM or Condor container where you run the service).
  • Local deployment:
    • Pro: You have the comfort of being able to work locally and fully offline
    • Pro: You do not need to use VMs or Containers for the setup
    • Con: Some of the CERN services are out of (direct) reach for you since they live behind the CERN firewall and sometimes you may need to create a tunnel or a socks proxy in order to access them.
    • Con: you cannot patch a live service directly from your development environment. You will have to either make a PR and redeploy or patch the running service using the PR as a patch or in case you want to avoid an early PR you must create a git --diff patch and scp it to the final destination.

Install the python virtual environment:

  • python2:
$ git clone https://github.com/dmwm/WMCore.git
$ pip2 install -U virtualenv
$ mkdir WMCore.venv2
$ /usr/bin/python2 -m venv WMCore.venv2
  • python3:
$ git clone https://github.com/dmwm/WMCore.git
$ mkdir WMCore.venv3
$ /usr/bin/python3 -m venv WMCore.venv3

And just for keeping track on how the size of the virtual environment grows in the process:

  • python2:
$ du -hs WMCore.venv2/
6.7M	WMCore.venv2/
  • python3:
$ du -hs WMCore.venv3/
6.7M	WMCore.venv3/

Installing all WMCore dependencies:

NOTE: The following instruction is a repetition of several trail and error attempts. So there is a possibility I have some OS package dependencies resolved in some of the previous tests and being well forgotten by the time I write this wiki. So any one who finds such, is very welcome to edit the document here directly. Thank you in advance.

Installing the OS packages dependencies:

Some of the python packages from the WMCore dependency list even though installed thorough pip are still relying on some OS tools and services, that's why those need to be resolved in advance before one starts the python deployment. A good example is pycurl which uses the system curl library and gcc to recompile things linking the shared libraries we have it configured with (e.g. openSSL).

$ sudo yum install gcc python-devel mariadb mariadb-devel openssl openssl-devel

*NOTE: * One needs to find the equivalent package names for the hosting OS he is about to use for this deployment.

(e.g. for Debian 10.7):

$ sudo apt-get update
$ sudo apt-get install gcc libpython2-dev python2-dev libpython3-dev python3-dev libmysqlclient-dev libmariadb-dev libcurl4-openssl-dev libssl-dev

Installing the python packages dependencies:

  • python2:
$ cd WMCore
$ source ../WMCore.venv2/bin/activate
(WMCore.venv2) [user@host:WMCore]$ pip install -r requirements.txt
  • You will most probably face the following error:
    ERROR: Command errored out with exit status 1:
     command: /home/user/Projects/WMCoreDev.d/WMCore.venv2/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-CyTllw/cx-oracle/setup.py'"'"'; __file__='"'"'/tmp/pip-install-CyTllw/cx-oracle/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-HZLyn6
         cwd: /tmp/pip-install-CyTllw/cx-oracle/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-CyTllw/cx-oracle/setup.py", line 170, in <module>
        raise DistutilsSetupError("cannot locate an Oracle software " \
    distutils.errors.DistutilsSetupError: cannot locate an Oracle software installation
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This is because Oracle for WMAgent is a centrally provided service (by CERN) and since in private deployments or at any FNAL machine we would never use Oracle but we would rather have mariadb as a backend instead, the suggested solution would be:

(WMCore.venv2) [user@host:WMCore]$ cp requirements.txt  requirementsNoOracle.txt
(WMCore.venv2) [user@host:WMCore]$ sed -i -e 's/cx-Oracle.*//g' requirementsNoOracle.txt
(WMCore.venv2) [user@host:WMCore]$ pip cache purge
(WMCore.venv2) [user@host:WMCore]$ pip install -r requirementsNoOracle.txt
  • In case you run into the following error (happens mostly for Debian due to packages versions mismatch:
    creating build/temp.linux-x86_64-2.7
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Dversion_info=(1,2,5,'final',1) -D__version__=1.2.5 -I/usr/include/mariadb -I/usr/include/mariadb/mysql -I/usr/include/python2.7 -c _mysql.c -o build/temp.linux-x86_64-2.7/_mysql.o
    In file included from _mysql.c:44:
    /usr/include/mariadb/my_config.h:3:2: warning: #warning This file should not be included by clients, include only <mysql.h> [-Wcpp]
     #warning This file should not be included by clients, include only <mysql.h>
      ^~~~~~~
    _mysql.c: In function ‘_mysql_ConnectionObject_ping’:
    _mysql.c:2005:41: error: ‘MYSQL’ {aka ‘struct st_mysql’} has no member named ‘reconnect’
      if ( reconnect != -1 ) self->connection.reconnect = reconnect;
                                             ^
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/user/Projects/WMCoreDev.d/WMCore.venv2/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-F1BUsK/mysql-python/setup.py'"'"'; __file__='"'"'/tmp/pip-install-F1BUsK/mysql-python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-pWwZYT/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/Projects/WMCoreDev.d/WMCore.venv2/include/site/python2.7/MySQL-python Check the logs for full command output.

The solution should be:

(WMCore.venv2) [user@host:WMCore]$ sed '/st_mysql_options options;/a unsigned int reconnect;' /usr/include/mysql/mysql.h -i.bkp
(WMCore.venv2) [user@host:WMCore]$ pip install -r requirementsNoOracle.txt
  • The virtual environment size at the end of this step:
(WMCore.venv2) [user@host:WMCore]$ du -hs ../WMCore.venv2/
213M	../WMCore.venv2/
  • python3:
$ cd WMCore
$ source ../WMCore.venv3/bin/activate
(WMCore.venv3) [user@host:WMCore]$ pip install -r requirements_py3.txt
  • In case you run into the following error or anything related to missing wheel package:
  ----------------------------------------
  Failed building wheel for pycurl
  Running setup.py clean for pycurl
  Running setup.py bdist_wheel for rucio-clients ... error
  Complete output from command /home/user/Projects/WMCoreDev.d/WMCore.venv3/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-45k4i4hp/rucio-clients/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-beei9j2l --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
  error: invalid command 'bdist_wheel'

The solution should be:

(WMCore.venv3) [user@host:WMCore]$ pip install wheel
(WMCore.venv3) [user@host:WMCore]$ pip install -r requirements_py3.txt
  • In case one runs into this error:
rust 0.1.1 has requirement mock<=1.0.1, but you'll have mock 4.0.3 which is incompatible.
...
        =============================DEBUG ASSISTANCE=============================
        If you are seeing a compilation error please try the following steps to
        successfully install cryptography:
        1) Upgrade to the latest pip and try again. This will fix errors for most
           users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
        2) Read https://cryptography.io/en/latest/installation.html for specific
           instructions for your platform.
        3) Check our frequently asked questions for more information:
           https://cryptography.io/en/latest/faq.html
        4) Ensure you have a recent Rust toolchain installed:
           https://cryptography.io/en/latest/installation.html#rust
        5) If you are experiencing issues with Rust for *this release only* you may
           set the environment variable `CRYPTOGRAPHY_DONT_BUILD_RUST=1`.
        =============================DEBUG ASSISTANCE=============================
    
    error: can't find Rust compiler
    If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
     To update pip, run:
         pip install --upgrade pip
     and then retry package installation.
    
    If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
    
    This package requires Rust >=1.41.0.
     ----------------------------------------
  This package requires Rust >=1.41.0.
  ----------------------------------------
  Failed building wheel for cryptography

The solution should be:

(WMCore.venv3) [user@host:WMCore]$ pip install --upgrade pip
(WMCore.venv3) [user@host:WMCore]$ pip install -r requirements_py3.txt

  • Any of the equivalent errors as in the python2 environment should have the identical solution for python3 too.

  • The virtual environment size at the end of this step:

(WMCore.venv3) [user@host:WMCore]$ du -hs ../WMCore.venv3/
375M	../WMCore.venv3/

Notes about pycrul and SSL backends:

Since we still need to use SSL for authentication purposes we have to have pycrl compiled with the same authentication backend library as libcurl (openssl in most of the cases), so it is a must at the end of the deployment procedure to check if this is the case with the mainstream package coming with pip install pycrl. The error that one should expect at run time should be similar to:

(WMCore.venv2) [user@host:WMCore]$ curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.53.1 zlib/1.2.7 libidn/1.28 libssh2/1.8.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets 

(WMCore.venv2) [user@host:WMCore]$ python
Python 2.7.5 (default, Nov 16 2020, 22:23:17) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pycurl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: pycurl: libcurl link-time ssl backend (nss) is different from compile-time ssl backend (none/other)

And the exact reason how it happens is highlighted here: http://pycurl.io/docs/latest/install.html#ssl

In case one runs into such a libcurl - pycurl backend mismatch, there are two possible situations and ways to react on this (independent of the python version):

  • Use pycrl with openssl:
(WMCore.venv2) [user@host:WMCore]$ sudo yum install python-devel openssl openssl-devel
(WMCore.venv2) [user@host:WMCore]$ export PYCURL_SSL_LIBRARY=openssl
(WMCore.venv2) [user@host:WMCore]$ export CPLUS_INCLUDE_PATH=/usr/include/python2.7/
(WMCore.venv2) [user@host:WMCore]$ export C_INCLUDE_PATH=/usr/include/python2.7/
(WMCore.venv2) [user@host:WMCore]$ export LDFLAGS=-L/usr/local/opt/openssl/lib
(WMCore.venv2) [user@host:WMCore]$ export CPPFLAGS=-I/usr/include/openssl/
(WMCore.venv2) [user@host:WMCore]$ unset LDFLAGS
(WMCore.venv2) [user@host:WMCore]$ pip install --compile --no-cache-dir pycurl
  • Or in case you need to use pycurl with nss:
(WMCore.venv2) [user@host:WMCore]$ pip uninstall pycurl
(WMCore.venv2) [user@host:WMCore]$ export PYCURL_SSL_LIBRARY=nss
(WMCore.venv2) [user@host:WMCore]$ pip install --compile --install-option="--with-nss" --no-cache-dir pycurl

Once the mismatch have been resolved, you should be able to see the same backend library for both curl and pycurl:

(WMCore.venv2) [user@host:WMCore]$ curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.53.1 zlib/1.2.7 libidn/1.28 libssh2/1.8.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets 


(WMCore.venv2) [user@host:WMCore]$ python
Python 2.7.5 (default, Nov 16 2020, 22:23:17) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pycurl
>>> pycurl.version
'PycURL/7.43.0.5 libcurl/7.29.0 NSS/3.53.1 zlib/1.2.7 libidn/1.28 libssh2/1.8.0'

Adding Ipython into your virtual environment directly:

In case you have ipython in the OS/VM/Container where you are setting up your development environment, you may run into the absurd situation of having a python3 environment but once you try to start ipython you find out the interpreter is actually a python2. In this case, and also in general, it is strongly recommended to have the ipython package setup inside the virtual environment so that any interpreter or packages or library version mismatches with the ones coming from outside the virtual environment may be avoided.

Here is the actual mistake visualized:

  • python2:
(WMCore.venv2) [user@host:WMCore]$ which ipython
/home/user/.local/bin//ipython
  • python3:
(WMCore.venv3) [user@host:WMCore]$ which ipython
/home/user/.local/bin//ipython

(WMCore.venv3) [user@host:WMCore]$ ipython
/home/user/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py:726: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
  warn("Attempting to work in a virtualenv. If you encounter problems, please "
Python 2.7.16 (default, Oct 10 2019, 22:02:15) 
Type "copyright", "credits" or "license" for more information.

IPython 5.10.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]:

The solution in both cases python{2,3} should be:

(WMCore.venv3) [user@host:WMCore]$ pip install ipython
(WMCore.venv3) [user@host:WMCore]$ deactivate

$ source ../WMCore.venv3/bin/activate

(WMCore.venv3) [user@host:WMCore]$ which ipython
/home/user/Projects/WMCoreDev.d/WMCore.venv3/bin/ipython

(WMCore.venv3) [user@host:WMCore]$ ipython
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]:
  • The virtual environment size at the end of this step:
(WMCore.venv3) [user@host:WMCore]$ du -hs ../WMCore.venv3/
397M	../WMCore.venv3/

Rucio and WMCore virtual environment:

In the next step we are about to run a WMCore service from inside the virtual environment. In the current one we are about to test the setup the Rucio Client. We presume the development environment have been deployed on a VM. We already have the Rucio Client package installed there with pip and we also have the Rucio Client configuration file inside the virtual environment. Now we need to change the configuration at the proper place:

(WMCore.venv3) [user@host:WMCore]$ cat << EOF >../WMCore.venv3/etc/rucio.cfg
[common]
[client]
rucio_host = http://cmsrucio-int.cern.ch
auth_host = https://cmsrucio-auth-int.cern.ch
auth_type = x509
ca_cert = /etc/grid-security/certificates/
client_cert = \$X509_USER_CERT
client_key = \$X509_USER_KEY
client_x509_proxy = \$X509_USER_PROXY
request_retries = 3
EOF

Just for the current test we need to export the paths to the user key and cert files. In the next step those variables will enter the virtual environment activate script so those exports will never be repeated again.

(WMCore.venv3) [user@host:WMCore]$ export X509_USER_CERT=/data/auth/dmwm-service-cert.pem
(WMCore.venv3) [user@host:WMCore]$ export X509_USER_KEY=/data/auth/dmwm-service-key.pem

And test the final configuration:

(WMCore.venv3) [user@host:WMCore]$ rucio -a wmcore_transferor whoami
status     : ACTIVE
account    : wmcore_transferor
account_type : SERVICE
created_at : 2020-08-18T16:04:03
updated_at : 2020-08-18T16:04:03
suspended_at : None
deleted_at : None
email      : *****@cern.ch

In case you would prefer to use your credentials instead of the wmcore ones then change the authentication method in the rucio .cfg file:

(WMCore.venv3) [user@host:WMCore]$ sed -i -e 's/auth_type = x509/auth_type = x509_proxy/g' ../WMCore.venv3/etc/rucio.cfg

And generate your proxy yourself as usual with:

(WMCore.venv3) [user@host:WMCore]$ unset X509_USER_CERT X509_USER_KEY
(WMCore.venv3) [user@host:WMCore]$ export X509_USER_PROX=/tmp/x509up_u`id -u`
(WMCore.venv3) [user@host:WMCore]$ voms-proxy-init --voms cms
(WMCore.venv3) [user@host:WMCore]$ rucio -a `whoami` whoami
status     : ACTIVE
account    : ****
account_type : USER
created_at : 2020-04-28T23:38:10
updated_at : 2020-04-28T23:38:10
suspended_at : None
deleted_at : None
email      : ****@cern.ch

Running services from the virtual environment - directly from your source code repository:

In case you have chosen to setup this environment remotely behind the CERN firewall, in something like a VM or a Docker Container or a shared filesystem reachable by a VM or Docker image running the central services, you may try to run a WMCore service directly from your local code repository. This in theory should be achievable by just changing your python path to point to the desired place (your working tree where you have synced WMCore code from github). In practice this is tested and proven to work very well only for micro services.

Here is what you need to do in order to make it happen:

(WMCore.venv3) [user@host:WMCore]$ cat << EOF |tee -a ../WMCore.venv3/bin/activate
export PYTHONPATH=</FIXME/PATH/TO/LOCAL/WORKINGTREE>/WMCore/src/python/:\$PYTHONPATH
export export RUCIO_HOME=</FIXME/PATH/TO/LOCAL/WORKINGTREE>/WMCore.venv3/
export X509_USER_CERT=/data/auth/dmwm-service-cert.pem
export X509_USER_KEY=/data/auth/dmwm-service-key.pem
unset X509_USER_PROXY
EOF

Since we are about to run without the central services deployment environment and without the configurations coming from there, we need to construct an init.py script so that a minimal set of msconfig parameters could be provided to the service.

(WMCore.venv3) [user@host:WMCore]$ mkdir -p ../test/ms-rulecleaner-standalone
(WMCore.venv3) [user@host:WMCore]$ cat << EOF > ../test/ms-rulecleaner-standalone/init.py

import logging
from pprint import pformat
from WMCore.MicroService.Unified.MSRuleCleaner import MSRuleCleaner

FORMAT = "%(asctime)s:%(levelname)s:%(module)s:%(funcName)s(): %(message)s"
logging.basicConfig(stream=sys.stdout, format=FORMAT, level=logging.DEBUG)
logger = logging.getLogger()

baseUrl = "https://FIXME.cern.ch/"

# Service config
msConfig = {"enableRealMode": False,
            "verbose": True,
            "interval": 1 *60,
            "services": ['ruleCleaner'],
            "rucioWmaAcct": "wma_test",
            "rucioMStrAccount": "wmcore_transferor",
            "useRucio": True,
            "rucioAccount": "wma_test",
            "wmstatsUrl": "%s/%s" % (baseUrl, "wmstatsserver"),
            "logDBUrl": "%s/%s" % (baseUrl, "couchdb/wmstats_logdb"),
            "logDBReporter": 'reqmgr2ms_ruleCleaner',
            "archiveDelayHours": 8,
            "reqmgr2Url": "%s/%s" % (baseUrl, "reqmgr2"),
            "msOutputUrl": "%s/%s" % (baseUrl, "ms-output"),
            "reqmgrCacheUrl": "%s/%s" % (baseUrl, "couchdb/reqmgr_workload_cache"),
            "dbsUrl": 'https://cmsweb-testbed.cern.ch/dbs/int/global/DBSReader',
            "couchDBUrl": 'https://cmsweb-testbed.cern.ch/couchdb',
            "rucioUrl": 'http://cmsrucio-int.cern.ch',
            "rucioAuthUrl": 'https://cmsrucio-auth-int.cern.ch'}

reqStatus = ['announced', 'rejected', 'aborted-completed']

msRuleCleaner = MSRuleCleaner(msConfig)
msRuleCleaner.resetCounters()
result = msRuleCleaner.execute(reqStatus)
logger.info('Execute result: %s', pformat(result))

EOF

And once we are ready with the init script we have two options:

  • Either runinng it once from outside and just observing/debuging the results:
(WMCore.venv3) [user@host:WMCore]$ cd ../test/ms-rulecleaner-standalone/
(WMCore.venv3) [user@host:ms-rulecleaner-standalone]$ ipython init.py
  • Or running it from within ipython and explore/debug:
(WMCore.venv3) [user@host:WMCore]$ cd ../test/ms-rulecleaner-standalone/
(WMCore.venv3) [user@host:ms-rulecleaner-standalone]$ ipython
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: %load_ext autoreload

In [2]: %autoreload 2

In [3]: %run init.py

p.s. We need %aoutoreload 2 so that we get assured that every time a change in the source code happens the relevant module will be reloaded at runtime while re-executing with %run init.py inside ipython.

From here on the Code-Test-Run cycles repeats on your pace.

Patching a live WMCore service directly from the local working tree:

In case you chose to deploy your virtual environment and develop directly on a machine or container which is having the WMCore central services running (or on a shared filesystem accessible from the central services instance) you can patch a running service directly from your working area and just restart it:

(WMCore.venv3) [user@host:WMCore]$ git diff --no-color src/python/WMCore/Services/pycurl_manager.py | sudo bin/patchComponent.sh reqmgr2ms
Patching component: reqmgr2ms
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/src/python/WMCore/Services/pycurl_manager.py b/src/python/WMCore/Services/pycurl_manager.py
|index 75bb01d..d544bae 100644
|--- a/src/python/WMCore/Services/pycurl_manager.py
|+++ b/src/python/WMCore/Services/pycurl_manager.py
--------------------------
patching file WMCore/Services/pycurl_manager.py
Using Plan A...
Hunk #1 succeeded at 99.
done

(WMCore.venv3) [user@host:WMCore]$  (A=/data/cfg/admin; cd /data; $A/InstallDev -s start:reqmgr2ms)

Or in case you have a commit ready to be added to a PR but not yet uploaded/pushed upstream.

(WMCore.venv3) [user@host:WMCore]$ git show e6bfaf41f3f370201fb1fe656b0c5b82409e04a2 --no-color | sudo bin/patchComponent.sh reqmgr2ms
Patching component: reqmgr2ms
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|commit e6bfaf41f3f370201fb1fe656b0c5b82409e04a2
|Author: Todor Ivanov <[email protected]>
|Date:   Sat Mar 13 04:38:21 2021 +0100
|
|    Fix difference between str/bytes in python2 and python3.
|
|diff --git a/src/python/WMCore/Services/pycurl_manager.py b/src/python/WMCore/Services/pycurl_manager.py
|index d544bae..35607be 100644
|--- a/src/python/WMCore/Services/pycurl_manager.py
|+++ b/src/python/WMCore/Services/pycurl_manager.py
--------------------------
patching file WMCore/Services/pycurl_manager.py
Using Plan A...
Hunk #1 succeeded at 60.
Hunk #2 succeeded at 68.
Hunk #3 succeeded at 82.
Hunk #4 succeeded at 100.
done

(WMCore.venv3) [user@host:WMCore]$  (A=/data/cfg/admin; cd /data; $A/InstallDev -s start:reqmgr2ms)

Running WMCore with python debugger:

Including pylint and pep coding style guiding tools to the virtual environment:

⚠️ **GitHub.com Fallback** ⚠️