DevOps tutorial – How We Bring Automation to the Mainframe - IBA-mainframe-dev/Global-Repository-for-Mainframe-Developers GitHub Wiki
Mainframe needs an update.
While mainframe continues to provide a lot of value to many different organizations, some of its elements remain stuck in the past and are still performed sub-optimally.
For example — mainframe software development is still very inefficient. It’s complicated, it’s high-effort, and it includes many time-consuming steps that reduce productivity.
Thankfully, it doesn’t have to be this way.
Mainframe software development can be updated with a more modern approach, and we wrote this article to share how we implement DevOps tools, processes, and best practices to automate every stage of mainframe software development.
We will show you:
- Our vision for a mainframe DevOps workflow, including which processes and components we use. We will explain how to use them individually or in combination to either implement our vision as-is within your local environment, or to test, customize, and combine them with your existing workflows.
- How modern Continuous Integration/Continuous Delivery (CI/CD) tools can be applied to mainframe to create automated analysis, build, test, run, and deployment of z/OS applications within a CI/CD pipeline.
- How we introduce a universal and flexible pipeline solution for a wide range of z/OS-based applications to provide a full range of CI/CD processes for mainframe teams and processes that currently utilize minimal or no automation.
Overall, we will present our approach with as much flexibility as possible, to make sure you can implement some of it no matter how specific or unique your context might be.
And to give you a comprehensive approach to mainframe DevOps, we will try to cover as many of the stages within the common DevOps “infinity sight” framework as possible.
Our solution is built around the “One Push Button” concept. In this concept, developers will only have to take a single action to start the pipeline, and the pipeline will then perform all other steps automatically, and move new code to production without any additional manual intervention.
This approach is fully customizable, and uses the following components and tools. (We have underlined the freeware tools.)
- ZIGI ISPF Git interface + Git for z/OS
- REXX scripts
- Automated SMP/E installation and management
- JCL jobs for various automation
- Jenkins - pipeline creation and execution
- TestLink integration with Jenkins pipeline
- JIRA - issues, statuses and document management
- SonarQube Enterprise - code analysis and quality gate
All of the tools used in our example are open-source, and any stage of the automation process can be called from Jenkins. We’ve found two options to integrate Jenkins with mainframe:
- IBM z/OS Connector Jenkins plugin
- Automated Shell scripts working through FTP
We use Git for z/OS and the ZIGI ISPF interface to create a “One Push Button” pipeline in mainframe, and to rapidly deliver code changes and fixes to production. Together, they create a convenient format to work with Git right on the mainframe in the 3270 session.
To implement this solution, you first must install Git for z/OS on your system.
Here’s how.
Installation
Here’s a quick guide to install Git onto z/OS. (You can find detailed information about the product and how to install it at: https://my.rocketsoftware.com/)
Prerequisite software
- z/OS 2.3 or higher: Required for Git 2.26.2 for z/OS.
- z/OS Miniconda: Git 2.26.2 for z/OS (and all Rocket z/OS tool and language ports) are installed via conda, so you need conda already installed on your system. z/OS Miniconda contains everything you need to install and run conda.
To download and install z/OS Miniconda from the Rocket Community Portal:
- Sign in to the Rocket Community Portal. https://my.rocketsoftware.com/RocketCommunity
- Click Downloads from the navigation menu. A list of available Rocket products for your account appears.
- Select z/OpenSource.
- Type 'Miniconda' in the search box to filter the list of available downloads.
- Click the filtered ID number. A dialog window appears with the files that are required for setup.
- Click each file to download them to your system.
- Using a FTP or SFTP client transfer the setup files to z/OS:
- miniconda-zos-<X.Y>-.run
- appdev_manifest-<X.Y>.txt
Note: If you have a multi-user environment you shouldn’t give your end users write access to the Miniconda installation directory. Instead, create a dedicated z/OS user to install and configure Miniconda. This Miniconda administrator won’t require any special permission nor superuser access.
To install Miniconda on z/OS, complete the following steps:
- Login to your z/OS system using ssh client (e.g. PuTTy) and change directory to the path where you saved Miniconda installation files.
- Set tags for these files:
chmod +x miniconda-zos-<X.Y>-<yyyy-mm-dd>.run
chtag -tc 819 appdev_manifest-<X.Y>.txt
- Set following environment variables to be able to run the install script:
export _BPXK_AUTOCVT=ON
export _CEE_RUNOPTS='FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)'
- Execute the self-extracting installer at the prompt:
./miniconda-zos-<X.Y>-<yyyy-mm-dd>.run --path <miniconda_installation_path>
- Confirm installation directory and conda channel server configuration in the interactive installation
Installing Git via conda:
conda install -c <channel> git
Here is the Git for z/OS sources, installation instructions and documentation: https://www.rocketsoftware.com/platforms/ibm-z/git-for-zos
Here is a separate article about ZIGI ISPF interface installation and usage: https://github.com/IBA-mainframe-dev/Global-Repository-for-Mainframe-Developers/blob/master/zOS%20System%20operating/Mainframe%20automation%20solutions%20and%20best%20practices/Zigi/README.md
In this article, we want to give an example of a Jenkins Declarative Pipeline code, which implements the DevOps practices on the mainframe. This Jenkins pipeline example is not the only correct approach, since all cases are individual and the silver bullet does not exist - our goal is to help define the approaches and the tool stack to provide a clear example and a starting point from which to start your mainframe DevOps journey.
NOTE: This pipeline is an example and is in the process of improvement and development. If you have any suggestions or comments, you can open an issue or create a pull request.
Pipeline consists of 13 stages. Each stage is responsible for a specific scope of tasks.
By clicking on the link below you can read a separate article on setting up and Jenkins pipeline creation, in which we tried to describe the entire process, the necessary stages and how they work:
When we say “Build”, we mean starting with source files produced by developers, and ending with things like installation packages that are ready for deployment.
To build code on the mainframe, you must run various jobs. To do so with DevOps, you need to perform a remote call to a job on the mainframe.
There are a few ways you can do this.
You can also use Shell script to run JCL jobs on the mainframe via FTP.
Execution format:
>./sendChangedSrc.sh [HLQ] [restore]
Where
-
[HLQ]- datasets qualifiers, required parameter -
[restore]- constant value which must be specified to restore changed modules from master branch in case of errors
The script will return 1 if HLQ was not specified, if the changed modules list is empty, or if the file transfer was unsuccessful.
Here is an example of this script: https://github.com/IBA-mainframe-dev/Global-Repository-for-Mainframe-Developers/wiki/Shell-script-for-transferring-changed-sources-in-git-to-the-mainframe-via-ftp
You can also group regularly repeated z/OS actions into one JCL job that can be executed on the mainframe from the pipeline via FTP. We often use this approach to prepare our environment to test code on another mainframe.
Below, we have prepared JCL scripts to automate the actions we need to perform on the mainframe. (These steps are part of our fully automated pipeline.)
Here are available automated options that we use in our pipeline example:
- Shell script for running JCL jobs on the mainframe via ftp
- Shell script for datasets transfering from UNIX to the mainframe via ftp
This plugin uses an FTP connection to IBM z/OS LPAR. Most often, it is used to run JCL jobs on z/OS, but you can also configure your SCLM project on z/OS and then check for the changes via Jenkins.
Here’s the official z/OS Connector plugin Jenkins hub: https://plugins.jenkins.io/zos-connector/
In our example, we didn't use this plugin due to limitations in its use. Because shell scripts for similar tasks are a more versatile and easily extensible solution. However, you can also use this plugin instead of the shell scripts we use.
Here’s configuration steps and other information about the z/OS Connector: https://github.com/IBA-mainframe-dev/Global-Repository-for-Mainframe-Developers/wiki/Jenkins-plug-in-solution-(IBM-z-OS-Connector)-for-running-JCL-jobs-and-manage-SCLM-on-mainframes
You can also automate some repetitive parts of code development for mainframe by writing REXX scripts, and using them in JCLs are part of the pipeline. In our case, we save time and eliminate careless mistakes by automating the process of building the PTF dataset.
Here’s our REXX script template: https://github.com/IBA-mainframe-dev/Global-Repository-for-Mainframe-Developers/wiki/REXX-script-for-buillding-PTF (IN PROGRESS, COMING SOON)
Code testing is an important step in the development cycle. You must make sure that your old functionality works without changes, and that your new functionality works as intended.
To do so, you must write test descriptions as test cases. You then need to store them somewhere that’s easy to monitor, that makes it easy to generate reports, and that collects all required functionality in one place.
There are many test management systems (TMS), including TestLink, TestRail, and BugZilla. We prefer TestLink for test automation because it has a good API that lets you integrate with a wide range of other software.
TestLink is a freeware TMS that drives software quality assurance. With it, you can manage different test-related artifacts — like cases, suites, plans, and builds — and you can run execution reports and statistics through a web browed user interface.
TestLink gives you:
- User roles and management.
- Grouping of test cases in test specifications, making test plans.
- Requirements with versioning and revisioning.
- Customization of the user interface using Smarty templates.
- Integration with LDAP.
We install TestLink on a separate virtual Ubuntu server.
You can download TestLink from these links. You can learn how to install TestLink within its official documentation, and within its GitHub .readme file.
Testlink source files: https://sourceforge.net/projects/testlink/files/latest/download?source=files
Testlink GitHub: https://github.com/TestLinkOpenSourceTRMS/testlink-code
Note: TestLink’s installation scripts are not fully automated and require some manual tweaking. Because of this you have to be careful when installing TestLink, especially in the output logs and messages. After you install TestLink, its GUI may contain bugs and artifacts, some of which are caused by bugs in TestLink’s source code. If you encounter problems during installation, or if the installed program is behaving incorrectly, then contact us and we will be happy to help you. Just open a GitHub issue on the relevant topic, or write to us, and we’ll address your problems as soon as possible.
GRMD GitHub Issues: https://github.com/IBA-mainframe-dev/Global-Repository-for-Mainframe-Developers/issues
Email for direct communication: [email protected]
We use the Jenkins plugin to link TestLink to our DevOps pipeline. This lets you associate automated tests with test cases, and to include clear descriptions and a list of steps, to set statuses, and to attach test logs and their results. This simplifies the work for testers and consolidates tests and test cases in one place.
After you install TestLink on your server, you must install the TestLink plugin.
Here’s the Testlink Jenkins plugin: https://plugins.jenkins.io/testlink/
Note: Remember that this plugin is open source (https://github.com/jenkinsci/testlink-plugin) and can be easily modified to suit your needs.
Where
- URL - your local Testlink URL
- Developer Key - Personal API access key which should be generated on the user “My Settings” page with appropriate field - API interface (Personal API access key)
To connect a test case in TestLink with a corresponding test class in Git, you first must add a new custom field.
Then, you must put the Git test class name into this field.
Finally, you must setup the Jenkins TestLink plugin as we previously described. You will now be able to execute tests from Jenkins, and your test results — as well as any JIRA bug issues opened for the failed tests — will automatically be added to the corresponding test execution records.
To review them, just click on the gamepad icon on the top panel (“Test Execution”) and select the appropriate test build number or name.
SonarQube is an industry-leading tool for code analysis that automatically finds and corrects logical, security, and vulnerability issues in source code. It supports PL/I and COBOL and integrates well into mainframe CI/CD processes.
SonarQube has two parts —a Scanner and a Server — and both should be installed on the machine that’s running your pipeline. The Scanner can be called with Jenkins or Azure, and once it completes its scan it passes the results to the Server. From there, you can continue the pipeline operation or stop it, depending on the Scanner’s results.
You can find more information at https://www.sonarqube.org/
Here’s SonarQube’s latest documentation: https://docs.sonarqube.org/latest/
And here’s a guide to try out the product: https://docs.sonarqube.org/latest/setup/get-started-2-minutes/
How to install a local instance of SonarQube from the zip file (Community Edition)
- Download zip file from the official site.
- As a non-root user, unzip it.
- As a non-root user, start the server:
- **binwindows-x86-64StartSonar.bat - for Windows
- **bin[Your OS]sonar.sh (console) - for others OS
- Log in to http://localhost:9000 using System Administrator credentials:
login: admin
password: admin
Here’s a trial for analyzing the project (Community Edition)
- Click the Create new project button.
- Give your project a Project key and a Display name, and click the Set Up button.
- Under Provide a token, select Generate a token. Give your token a name, click the Generate button, and click Continue.
- Select your project's main language under Run analysis on your project, and follow the instructions to analyze your project. Here you'll download and execute a Scanner on your code (if you're using Maven or Gradle, the Scanner is automatically downloaded).
Here's SonarQube’s full installation guide: https://docs.sonarqube.org/latest/setup/install-server/
Here’s a separate article about SonarQube installation and usage:
When we talk about “Deploy”, we include every step, process, and activity needed to make a software system or update available for its intended users. Like with Build, this require running a wide range of jobs. Here are a few examples.
You don’t need to manually install the new PTF into SMP/E. Instead, you can call JCL jobs to RECEIVE/APPLY/ACCEPT PTF directly from CI/CD pipeline. If there is any failure, you can perform automatic rollbacks by calling REJECT/RESTORE JCLs.
Here’s a list of JCL scripts and templates we use to automate work with SMP/E:
- JCL job for loading PTF information into global zone and SMPe datasets
- JCL job for installing PTF in the SMPe target libraries
- JCL job for installing PTF in the SMPe distribution libraries
- JCL job for rejecting PTF to clean up the global zone
- JCL job for restoring PTF to clean up the target zone
- JCL job for creating SMPe zones environment and user datasets
- JCL job for customizing SMPe distribution zone environment
- JCL job for customizing SMPe global zone environment
- JCL job for customizing SMPe target zone environment
Operational tasks include running a wide range of jobs on mainframe. These include:
- Helping QA create conditions for stress tests and load testing.
- Transferring code to production.
- Transferring files from one system to another.
- Any other manual operation that can be included in the pipeline and automated.
For example: After deploying an application you may need to transfer a file that is being kept by the customer on Windows onto the mainframe where it will be used. To automate this process, you can use the following script:
You can call this job from Jenkins and include it in your general pipeline.
Here are a few more scripts that automate operational tasks and add them to the DevOps pipeline.
- Shell script for datasets to transfer from UNIX to the mainframe via ftp
- Shell script to run JCL jobs on the mainframe via ftp
- Shell script to transfer changed sources in git to the mainframe via ftp
- How do I copy data sets to other LPAR?
With Planning and Monitoring, teams can quickly and automatically respond to degradation in the customer experience, and implement the product. A bug tracking system can help with this.
Integrating Jira into the Jenkins pipeline’s processes — like tracking issues and statuses, and managing documents — and into test management systems — like tracking deployment status, and automatically generating reports
JIRA is a common choice for project management and bug tracking, and it includes a range of useful features. These include:
- Task tracking.
- Decorating pipeline execution.
- Interaction with other tools via REST API.
You can automate all of these features into the software development process by integrating JIRA with your Jenkins pipeline and test management tools. You can integrate JIRA with these tools using Jenkins plugins or REST API-based scripts.
Here’s how we integrate Jira with our Jenkins pipeline and test management tools.
- In Jira we create a ticket with the task’s description, assignee, priority and other necessary fields for the development of a new functionality or software module.
- The developer starts working on the task.
- When the new functionality is ready, and when the developer is ready to start a merge request to the required branch, the ZIGI ISPF interface will automatically issue the merge request on the mainframe with a Jira ticket name. This will also trigger the launch of the entire Jenkins pipeline.
- The Jenkins pipeline then converts the ticket status to “In Progress” (or any other custom status) and the magic begins :).
- In Jira — during the work of the pipeline — comments are added for each step of the pipeline, whether it is successful or not. This lets us track the status of the deployment and its history directly from JIRA.
- During testing, we also add test results to the ticket in the form of .xml files or any others.
- When the pipeline completes its execution, the ticket is set to the “Done” status. If the pipeline execution was interrupted or terminated with an error, it is set to the “To Do” status with slack and email notification.
For example, let’s look at this integration using the simplest and most functional Jenkins plugins. As an alternative to the plugin, you can use a set of REST-based, easy-to-customize shell scripts. Here are a few examples of these scripts, including instructions.
- How to add comments to the Jira ticket
- How to attach files to the Jira ticket
- How to change of statuses in Jira ticket
- How to track Jira ticket status (trigger desired action after ticket status change)
Here are our most-used Jenkins Jira Plugins. Some have similar functionality , including a few differences that are necessary in some cases:
- Jira plugin - https://plugins.jenkins.io/Jira/
- Jira Pipeline Steps - https://plugins.jenkins.io/Jira-steps/
- JiraTestResultReporter - https://github.com/IBA-mainframe-dev/Global-Repository-for-Mainframe-Developers/wiki/Shell-script-to-track-Jira-ticket-status
Note: All of these plugins are open-source, and you can use them to build custom hpi modules that better fit your unique tasks and requirements.
To set up your plugins, and to connect them to your Jira instance, you will use the Jenkins System Configuration.
Here are Jenkins pipeline code examples (declarative syntax):
Сhange Jira ticket status from ‘TO DO’ to ‘IN PROGRESS’
Here, you can determine the ID of each status — including the custom one —by selecting statuses and viewing the html code of the Jira ticket page. IDs differ between Jira instances, including for standard statuses, so this is the fastest way to find them.
They don’t change, so you can save them in advance:
def transitionToDo = [ transition: [ id: '11' ] ]
def transitionInProgress = [ transition: [ id: '21' ] ]
def transitionDone = [ transition: [ id: '31'] ]
- Jira ticket status transfer example:
JiraTransitionIssue idOrKey: "${JiraID}", input: transitionInProgress, site: "${JiraSite}"
Where
-
${JiraID}- your Jira ticket ID (like “'DVSS-19'”) -
${JiraSite}- your Jira location name (from Jira System Configuration)
- Add comments to the Jira ticket
JiraAddComment idOrKey: "${JiraID}", comment: 'Jenkins Pipeline is started', site: "${JiraSite}"
- Test failure - open new bug ticket, example:
- Upload attachment to the Jira ticket attachments
JiraUploadAttachment idOrKey: "${JiraID}", site: "${JiraSite}", file: 'changesList.txt'
Where
-
${JiraID}- your Jira ticket ID (like “'DVSS-19'”) -
${JiraSite}- your Jira location name (from Jira System Configuration) - file: - name of your attachment file