DMWM Packaging with PyPi - dmwm/WMCore GitHub Wiki

This document outlines the transition of DMWM software to the Python Package Index, https://pypi.org/

Developers should create their own personal PyPI accounts for maintaining DMWM software packages in PyPI. DMWM uses PyPI collaborator roles to make all projects owned by the user cms-oc-dmwm, with maintainer access going to individual developers. More info about roles can be found here: https://pypi.org/help/#collaborator-roles

DMWM PyPi project owner account: cms-oc-dmwm

Projects: https://pypi.org/user/cms-oc-dmwm/

This should be used when a developer wants access to manage DMWM PyPI projects.

• Email cms-oc-dmwm-pypi E-Group and request to be added as a maintainer of a project
• Once the role is configured, you will receive a notification email about the new role.

This should be used if a developer has created a new PyPI project and wants to combine it with other DMWM PyPI projects managed by the cms-oc-dmwm account.

• Initial upload of packages to PyPI should be performed by the developer under their account
• Once you have uploaded, give the cms-oc-dmwm account the "Owner" role under the "Collaboration" settings
• The cms-oc-dmwm-pypi E-Group members will be notified of the change in role via email
• We will change your role to "Maintainer", allowing you to release new versions, but not delete the project, etc.

General docs for packaging projects: https://packaging.python.org/tutorials/packaging-projects/

Test packages location: https://test.pypi.org/user/cms-oc-dmwm/

Production packages location: https://www.pypi.org/user/cms-oc-dmwm/

This was all documented using lxplus. Instructions might vary depending on your OS and python version.

Create a virtualenv for building packages. Need setuptools and twine.

$ virtualenv pypibuild
$ source pypibuild/bin/activate
(pypibuild) $ pip install twine

Make a ~/.pypirc file with required contents for uploading to PyPI.

[distutils]
index-servers =
  pypi

[pypi]
repository: https://upload.pypi.org/legacy/
username: <your username>

$ git clone https://github.com/dmwm/WMCore.git
$ cd WMCore
$ git fetch && git fetch --tags

$ git checkout <tag>

$ sh tools/build_pypi_packages.sh <wmagent|wmcore|all>

(pypibuild) [username@lxplus789 WMCore]$ sh tools/build_pypi_packages.sh wmcore
Building wmcore package
/afs/cern.ch/user/u/username/pypibuild/lib/python2.7/site-packages/setuptools/dist.py:476: UserWarning: Normalizing '1.3.3.pre2' to '1.3.3rc2'
  normalized_version,
running sdist
<snip>
Writing wmagent-1.3.3rc2/setup.cfg
Creating tar archive
removing 'wmagent-1.3.3rc2' (and everything under it)
Enter your password: 
Uploading distributions to https://upload.pypi.org/legacy/
Uploading wmagent-1.3.3rc2.tar.gz
View at:
https://pypi.org/project/wmagent/1.3.3rc1/

Check the cms-oc-dmwm PyPI page to find the new uploaded version: https://pypi.org/project/wmcore/#history

[username@lxplus789 WMCore]$ virtualenv wmcore-pypi
New python executable in /afs/cern.ch/user/u/username/WMCore/wmcore-pypi/bin/python
Please make sure you remove any previous custom paths from your /afs/cern.ch/user/u/username/.pydistutils.cfg file.
Installing setuptools, pip, wheel...done.

[username@lxplus789 WMCore]$ source wmcore-pypi/bin/activate
(wmcore-pypi) [username@lxplus789 WMCore]$ pip install wmcore
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting wmcore
  Downloading wmcore-1.3.5.tar.gz (2.2 MB)
<snip>
Successfully built wmcore
Installing collected packages: wmcore
Successfully installed wmcore-1.3.5

If you want a specific version:

pip install wmcore==1.3.6.pre3

This is the list of dependencies that are required before one can install wmagent via pip. Python dependencies are automatically pulled when the wmagent package is installed.

• MariaDB
• CouchDB
• Oracle Client
• Condor
• libcurl-devel
• jemalloc
• libuuid
• yui (might not be necessary)
• pystack (might not be necessary)
• gcc

If any of these packages are installed in a non-standard location, you will need to set some environment variables as well.

• Add mariadb bin and gcc bin directory to PATH
• Add gcc lib directory to LD_LIBRARY_PATH
• Set ORACLE_HOME to Oracle client location

Create a wmagent virtualenv

$ virtualenv wmagent
$ source wmagent/bin/activate
(wmagent) $ pip install wmagent

Instructions on how to build and upload DBS Client and Pycurl Client to PyPi.

Create a .pypirc file in your home directory with the following contents:

[distutils]
index-servers =
  pypi

[pypi]
repository: https://upload.pypi.org/legacy/
username: <your account>

The following changes should be made when a new version of DBS is released. Some version numbers will have to be updated manually for now. This will (hopefully) be automated in the future.

1. dbs-client version in setup_dbs_client.py, L10
2. pycurl-client version in setup_pycurl_client.py, L10
3. pycurl-client version in requirements.dbs_client.py, L1

Clone the DBS repo and checkout the latest tag:

$ git clone https://github.com/dmwm/DBS.git
$ git fetch && git fetch --tags
$ git checkout <tag>

Run the build_pypi_packages.sh script build and upload the packages. The script takes the following options, but you will usually want to build "all".

  all              Build all DBS packages
  dbs-client       Build dbs-client package
  pycurl-client    Build pycurl-client package

$ sh Tools/build_pypi_packages.sh all

Enter the cms-oc-dmwm account password when prompted. This can also be set in the .pypirc file if you don't mind storing passwords in clear text.

Check the details of the new build here: https://pypi.org/user/cms-oc-dmwm/

Source a CMSSW environment.

[user@lxplusXXX ~]$ export SCRAM_ARCH=<arch>
[user@lxplusXXX ~]$ cmsrel CMSSW_X_Y_Z
[user@lxplusXXX ~]$ cd CMSSW_X_Y_Z/src
[user@lxplusXXX src]$ cmsenv

To use DBS client, one needs to install it via pip and then activate a CMSSW envrionment. Using DBS client with user certificates and proxies requires a curl shared library built with special configuration options. This curl library is pulled in when setting the CMSSW environment. Using virtual python environments (virtualenv) in conjunction with the CMSSW environment gets messy real quick, so it is best to just install DBS client into your home directory with the --user option.

[user@lxplusXXX ~]$ pip install --user dbs-client
Collecting dbs-client
<snip>
Successfully installed dbs-client-3.7.8 pycurl-7.19.0.3 pycurl-client-3.7.8 python-cjson-1.2.1

Create a proxy

[user@lxplusXXX ~]$ voms-proxy-init -rfc -voms cms

Test access with a simple python script

[user@lxplusXXX ~]$ cat dbs-test.py
from dbs.apis.dbsClient import DbsApi
dbs = DbsApi('https://cmsweb.cern.ch/dbs/prod/global/DBSReader')
ds = dbs.listDatasets(dataset = '/MinBias/Summer*/*')
print(ds)

[user@lxplusXXX ~]$ python dbs-test.py 
[{'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa1E31-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa8E29-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_7TeV_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHChcal-v1/GEN-SIM'}]