• Currency work process
• RTC ticket format for Currency work
• Historical report format
• Dockerfiles creation story
• Dockerfile Format
• Changes.txt format
• Recipe update
• Currency Optimization (PPT contents)

• Currency work is ongoing activity where recipes listed on IBM community are updated to latest version. Every time the new version is released for a package, its respective recipe needs to be updated for s390x. One needs to identify changes required for latest version and accordingly update the recipe.

• We design recipes which support various distributions like RHEL 6.9, RHEL 7.1, 7.2, 7.3, SLES 11-sp4, SLES 12, 12 SP1/12 SP2 and Ubuntu 16.04/16.10 on Linux on IBM z Systems.

• IMP: Many packages are already provided by default repositories of that distribution. For ex, MySQL-5.6 we have built from source for RHEL and SLES distributions, however it is already present on one of the Ubuntu default repositories. If you get the package with latest version, then convey this to team and no need to work on that. For RHEL, SLES and Ubuntu, you would get it using 'yum search <pkg_name>', 'zypper search <pkg_name>' and 'apt-get cache <pkg_name>' commands, resp. Please check the same here.

• If the package is not provided in default repositories, you need to refer its respective recipe and modify it in such a way that it would point to latest stable version. Refer the recipe from IBM community page.

• If extra changes are required for building latest version, add them in the SVN recipe and commit it on SVN. On SVN, you would get recipe which are kept on IBM community page. Add required changes in it.

• Also you need to create 'changes.txt' file in the same folder. This file consists of diff of changes between IBM recipe and your recipe. For ex. If existing version of a package is 1.2.3 on IBM page and latest version is 2.3.4, then this file should have diff of 1.2.3 and 2.3.4.

• Your recipe will later replace IBM community recipe and table links will be updated accordingly.

• Also add/update jenkins jobs on ecos0023 for the package accordingly on SVN.

• RTC i.e. Rational Team Concert is a tool where you can plan and track your development project, perform change and configuration management, and manage your build processes.

• To explore Clm RTC web user interface, click here.

• Go to Project Dashboards -> LOZ_CM -> Current Plans widget and click on <current sprint>. You will get the list of work items (user stories, tasks, etc) per owner.

• To create RTC task for currency work, follow below steps:

  • Go to Currency work user story (currently assigned to Koumudini Bhute)
  • Select 'Create child work item' from the drop down at left-> Task -> <package_name> - <version>
    ex. cAdvisor - 0.20.2

    • Add necessary details like Owned by, Planned for, Priority, Estimate, Time Remaining, Due Date, tags, Task Complexity, etc.

    • This task should ideally be owned by yourself.

    • Please refer sample RTC task format here

    • This task should include total efforts spent for that package by currency developer (analysis + recipe update + Jenkins job on ecos0023).

    • Add Estimate and Time remaining accordingly.

    • Set complexity

    • In Description field of this task, please add following details in Yes or No format:

      1. Checked Latest stable version: Yes/No
      2. Package available in distribution: Yes/No
      3. Is current published recipe working: Yes/No
         Note: You can verify above point by checking its Jenkins jobs.
      4. Is Groovy Script added in Jenkins job: Yes/No
      5. Is Poll SCM option set for Jenkins job: Yes/No
      6. Removed Developers from Mail recipients: Yes/No
      7. Test reports created: Yes/No
      8. Removed ROMAN numerals from recipe: Yes/No
      9. Recipe updated for RH(7.1,7.2,RHEL7.3), SL(12,SP1), RH(6.9), SL(11-SP4) & Ub(16.04,16.10): Yes/No
      10. Took changes from Published Recipes folder: Yes/No
      11. Updated recipe to SVN: Yes/No
      12. Updated Jenkins jobs on ecos0023: Yes/No

    • Recipe should be uploaded on SVN. Format of recipe name MUST be <pkg_name>_<version>_<applicable_distro_names_seperated_by_underscore>.md. ex. cAdvisor_0.20.2_rh7_sl12.md

    • You would need to add/update jenkins jobs for the package you are working on. To go to Jenkins Dashboard, click here.

• Once above task is created, add below mentioned sub tasks under this task following same procedure as above:

  1. Update jenkins jobs on ecos0017

     • This task is required to create so that once the item goes in currency, whatever changes are incorporated in recipe should be directly updated in ecos0017 jobs to keep consistency.
     • This task should be assigned to Permanent owners of the jenkins jobs. Refer the owners and packages here.

  2. <Package name> - PR - <PR_Number> (if Applicable)

     • This task should be assigned to yourself.
     • This task should have all details related to PR like Status, Id, Created on, reference links, etc.
     • Once the PR is merged to master branch and next stable release is announced, the recipe should be updated to use next release version directly from main branch and references of github/linux-on-ibmz/ should be removed. This is the moment where this ticket can be closed.

Few Points to note:

• For typical currency recipe, Name of the ticket will be PackageName – version.
• For Broken recipe, Name of the ticket should be followed by (Broken) e.g. PackageName – version(Broken).
• If more than one person is working on the currency task, then separate tickets should be created with Part I, Part II, etc.
  For example, PackageName – version(Part I), PackageName – version(Part II) and so on.
• If ticket is not completed and forwarded to next sprint, then close the existing ticket and create new one for current sprint with same name followed by (Backlog)
  For example, PackageName – version(Backlog)

Once the recipe for your version is uploaded on IBM community, you can mark the main sub task as completed.

• We usually create these reports to transfer our knowledge to next person who is going to work on respective task. It will be helpful if he/she works on higher version of same package or on the same version.
• For this, you need to go to svn inside LoZ -> Recipes -> <package-name> folder. Create Reports folder inside it. Add one more folder named as <version> inside Reports folder. Create report.xlsx and all_test_logs.txt files inside <version> folder.
• Click [here](https://svn.persistent.co.in/svn/Opensource_Ecosystem_for_POWER/LoZ/docs/Historical Report Format.xlsx) to get the standard template for writing reports. Copy this template in your report.xlsx file and fill the details.
• In report.xlsx, add details like intermittent test case failures, any specific fix you did to run the failed tests, build failures and its fixes, comparison of test cases between SystemZ and x86 arch, etc.
• Add Build time issues and fixes in Build issues Report tab and test failures in Test case Failures Report tab.
• In all_test_logs.txt, add complete test cases result logs.
• For example, consider two versions i.e. 2.2.1 and 2.3.1 are published for Elasticsearch. Its folder structure should look like below(also check this):

<Your-SVN-folder> -> Loz -> Recipes -> Elasticsearch -> Reports ->
2.2.1 ->
report.xlsx all_test_logs.txt
2.3.1 ->
report.xlsx all_test_logs.txt

Following would be the structure of this tasks:

1. Go to Verify Dockerfiles user story here
2. Create a task named as Ubuntu - Dockerization of <package-name> - <version>
3. Create 3 subtasks under this as follows: 1. Dockerization of <package-name> - <version> 2. Technical review -<package-name> - <version> 3. Formatting review - <package-name> - <version>
   • Add description as same as specified in Formatting review of Currency work tasks.

Below is the process to be followed while creating Dockerfiles for packages.

• The packages will be separated as either a Binary package or a Service package.

• List of packages which have been separated as Binary/Service can be found here.

• If any of the packages are not listed in the above list, please update the sheet with the package name and type.

• NOTE: Replace the command-line as per the distro for which the Dockerfile is being created (i.e. yum-rhel, zipper-sles, apt-get Ubuntu).

  1. Binary Packages

     • Can be used as base for building an application.
     • Can be run as a container to perform various operation.
     • May/may not contain CMD at end of dockerfile (depends on the functionality required to be performed by dockerfile).
     • Refer Binary_Dockerfile_format for Binary Dockerfile format.

  2. Service Packages

     • Contains CMD at end of dockerfile.
     • The resultant packages can be a server or viewed as web UI.
     • Refer Service_Dockerfile_format for Binary Dockerfile format.

================================= <version> ==================================
Date: <mm/dd/yyyy>
Category: <Currency/Broken Recipe/Added Header/Added Ubuntu Support/Updated header for available versions/Other(specify something)>
Changes:

================================== end =========================================

1. Once you analyze the required changes for latest stable version, DO NOT update or copy-paste the previous recipe from SVN. Instead take recipe from Published Folder(Github) ONLY and add your changes in that.
2. Keep header section as it is. DO NOT update versions even if you feel the mentioned version is outdated. Ex. For Apache Zookeeper 3.5.1, we should keep header tags as below:

<!---PACKAGE:Apache ZooKeeper--->
<!---DISTRO:RHEL 6.6:3.4.8--->
<!---DISTRO:RHEL 7.1:3.4.8--->
<!---DISTRO:SLES 11:3.4.8--->
<!---DISTRO:SLES 12:3.4.8--->
<!---DISTRO:Ubuntu 16.x:3.4.8--->

3. If above things are not present in recipe from Published Recipes (GitHub), then no need to add that your recipe by yourself. Its Gary's job to add that.
4. As you all aware, we have introduced a little new format in upcoming recipes i.e. in next line, following contents should be present, ex. Apache Zookeeper:

<!---PACKAGE:Apache ZooKeeper--->
<!---DISTRO:RHEL 6.6:3.4.8--->
<!---DISTRO:RHEL 7.1:3.4.8--->
<!---DISTRO:SLES 11:3.4.8--->
<!---DISTRO:SLES 12:3.4.8--->
<!---DISTRO:Ubuntu 16.x:3.4.8--->

Building Apache ZooKeeper

Below versions of Apache ZooKeeper are available in respective distributions at the time of this recipe creation:

Ubuntu 16.04 has `3.4.8`

Note: In above example, Apache Zookeeper was present only on Ubuntu repository by default. Your package may present on various distributions, list them one by one accordingly.

5. IMP: Even if a package is present on distribution by default, we have to provide instructions for building latest stable version of that package on all distributions. For ex, in case of Apache Zookeeper, version 3.4.8 is present on Ubuntu repository by default, we still MUST create recipe for Ubuntu for version 3.5.1 along with RHEL and SLES.

6. This line which we used to add in header “Apache ZooKeeper v3.5.1 has been successfully built and tested for Linux on z Systems. The following instructions can be used for RHEL 7.1/6.6 and SLES 12/11.” is now replaced with “The instructions provided below specify the steps to build Apache ZooKeeper v3.5.1 on Linux on the IBM z Systems for RHEL 6/7, SLES 11/12 and Ubuntu 16.04.“ as follows:

<!---PACKAGE:Apache ZooKeeper--->
<!---DISTRO:RHEL 6.6:3.4.8--->
<!---DISTRO:RHEL 7.1:3.4.8--->
<!---DISTRO:SLES 11:3.4.8--->
<!---DISTRO:SLES 12:3.4.8--->
<!---DISTRO:Ubuntu 16.x:3.4.8--->

Building Apache ZooKeeper

Below versions of Apache ZooKeeper are available in respective distributions at the time of this recipe creation:

Ubuntu 16.04 has `3.4.8`

The instructions provided below specify the steps to build Apache ZooKeeper v3.4.8 on Linux on the IBM z Systems for following distributions:

• RHEL (6.8, 7.1, 7.2, 7.3)
• SLES (12, 12 SP1, 12 SP2)
• Ubuntu (16.04, 16.10)

This line should be followed by General Notes section and rest of the recipe is as same as we do.

• Add references if missing
• For more details related to formatting, please refer Recipe format.doc in ~/LoZ/recipes/Recipe Template/ on svn.

Agenda –

1. Starting a service in jenkins job
2. Including all test cases in jenkins job
3. Task Complexity
4. Currency automation
5. New jenkins setup : ecos0023

Starting a service in jenkins job
Categories:
a. Package as a Binary
e.g. doxygen -v
b. Package as a Service
* Start a service e.g. $CATALINA_HOME/bin/startup.sh
* Check its response e.g. ps –aef | grep tomcat OR curl/wget http://localhost:
c. Package as a Database Service
* Start a database server e.g. /usr/bin/mysqld_safe –user=mysql &
* Check few basic queries like insert, select, etc. e.g. show databases

  Note: Sleep() should be added for services which take time to start. 

Including all test cases in jenkins job
* Include all test cases in jenkins job.
* Exclude those are intermittent, equivalent to X86, known, etc.
* For timeout related test cases, increase timeout value.
* Compulsorily keep a note in jenkins job itself with the reason for excluded test cases.

Complexity
* Once currency item is assigned to a developer, at max 1 day he/she should spend on analyzing the complexity. * Depending upon complexity, developer must add no. of hours in his analysis ticket.
* The total number of hours is an addition of three tickets i.e. analysis, update recipe & update Jenkins jobs
* Anytime when a developer feels the complexity needs to be corrected, raise a flag with proper justification.

Currency automation
* Automation of porting of package with the help of Jenkins job
* Starting point should be Jenkins itself.
* Only 3 RTC children tickets will be created: Analysis of latest version, Update recipe & Update jenkins job
* Put 'Automation' tag in main ticket. eg. 31722. Docker Compose - 1.8.0(Automation)
* Update changes.txt accordingly.
* Total efforts ideally be 4 hours. This effort is an addition of analysis, update recipe and update Jenkins job.
* No technical and formatting review will be needed.

New jenkins setup: ecos0023
* http://ecos0023.pok.stglabs.ibm.com:8080/jenkins
* This Jenkins should be used by currency team for automation.
* Privates nodes should be used to run the jobs.
* Also can be used for any Adhoc tasks, debugging, etc.
* Read/write permissions are given to everyone in the team.
* Ecos0017 will be just for continuous integration. All permissions are revoked. Only read access to all.

• Difference between ecos0017 and ecos0023:

  • ecos0017 has docker cloud support. The jobs on ecos0017 are triggered in particular interval i.e. mostly once in a week.
  • Jobs should be created on ecos0017 for all packages present on IBM table.
  • No. of distributions present in a recipe = no. of jenkins jobs to be created
  • Same jobs should be updated if recipe is updated on IBM table for latest version
  • In short, ecos0017 is used for maintenance of IBM table recipes.
  • One has just read only rights of ecos0017. No slave nodes can be created.
  • On ecos0023, one can create n number of slave nodes.
  • Its mainly used for debugging, exploring any adhoc task, etc.

• Currency flow for new comers:

  • We generally have currency meeting every alternate Monday.
  • Refer "Currency targeted for " mail on outlook. This mail will include packages on your name which are targeted for next currency meeting
  • Pick up packages based on priorities. Check jenkins jobs(ecos0017) for the same package for earlier version. Get them on ecos0023 and try to execute same with just version change on private slave node.
  • If all jobs pass for latest version, then only one section will be changed in recipe i.e. checking out version. Changes.txt should be updated saying 'Updated to new version' only. This is called Jenkins Automation where we use previous jenkins jobs for latest version. Such packages will belong 'Simple' with respect to Complexity.
  • If fails, you can either try it in separate container or can do ssh to the slave node and start debugging.
  • Once task is done for a distribution e.g. RHEL 7.1, you can update jenkins job on ecos0023 for RHEL 7.1 and then can try jenkins jobs for similar distributions like RHEL 7.2/7.3 by cloning the same job of ecos0023.
  • Every currency owner MUST update their jobs on ecos0023 once they are done with their currency tasks. This job can then be updated to ecos0017 once the recipe is published on IBM table.
  • One should make sure all the distributions are added in the recipe. Even if some are missing in previous recipe until there is some genuine reason for skipping that.
  • Formatting should be well checked.
  • RTC tickets should be updated properly with proper estimates.