ODS Bulletins - opendevstack/PMC GitHub Wiki

Next issue

https://github.com/opendevstack/ods-core/pull/831

OpenShift 4 compatibility work

Bitbucket PR functionality

Issue #13 (2020-08-14)

Hello,

welcome to no. 13 of the ODS bulletin! This time we can celebrate the release of ODS 3!

What happened lately?

ODS 3.0.0 released

Finally, we tagged v3.0.0 of OpenDevStack! I want to use this opportunity to thank everyone who contributed to it - be it questions, feedback, documentation, coordination, code, reviews, comments, ... ODS 3 is a great release with improvements in so many areas over ODS 2 that it is quite hard to summarise everything that is new :) For the major new features, see https://www.opendevstack.org/blog/2020/08/13/announcing-ods-version-3.0. As we also worked a lot on the documentation, check out https://www.opendevstack.org/ods-documentation/opendevstack/3.x/getting-started/index.html to find out more about some of the smaller changes. A very detailed changelog is available in every repository, for example https://github.com/opendevstack/ods-core/blob/master/CHANGELOG.md.

Noteworthy issues and pull requests

That's it for today! I'm going to adjust the frequency of the bulletin from theory to reality, so expect the next update in about 3 weeks :)

Issue #12 (2020-07-17)

Hello,

welcome to the 12th edition of the ODS bulletin! ODS 3.0 is nearing its release end of this month with lots of exciting new features and stability improvements!

What happened lately?

Avoiding to rebuild a container image for the same Git commit

ODS 2 and earlier would build a new container image in every pipeline run - even if there was already an image for the Git commit being built in the same OpenShift project (or another OpenShift project, e.g. if the image existed in *-dev and the pipeline would deploy to *-test). https://github.com/opendevstack/ods-jenkins-shared-library/pull/400 adds a new feature for ODS 3, so that images in the same or other projects can be reused. For more details how that works exactly and how you can use it in your pipelines once ODS 3 is installed in your cluster, see https://www.opendevstack.org/ods-documentation/opendevstack/3.x/jenkins-shared-library/component-pipeline.html#_odscomponentstageimportopenshiftimageorelse.

New Provisioning App Interface

There has been a lot of work on a new user interface for the provisioning app! See a screenshot and the rationale behind the change in https://github.com/opendevstack/ods-provisioning-app/issues/518. For ODS 3.0, the new UI will be hidden behind a feature flag, but rolled out as the default in a future release.

Quickstarter Testing Framework

To ensure that the provisioning of a quickstarter, and the build of the resulting component works reliably, a new "test framework" has been established in https://github.com/opendevstack/ods-quickstarters/pull/357. It allows to define for each quickstarter which Jenkins stages have to run for both provisioning and building, and then checks if those have been executed when the tests run. A great improvement towards more stability and automated tests! This can be extended in the future to include even more checks (such as generated source code etc.)

Noteworthy issues and pull requests

And that's a wrap - next update once ODS 3.0 is released :)

Issue #11 (2020-07-01)

Hello,

welcome to the 11th edition of the ODS bulletin! We are getting much closer to a ODS 3.0 release with only 10 open issues on GitHub!

What happened lately?

Bringing deployment events into the Jenkins log

If a deployment fails, the problem can usually be found in either the replication controller events or the pod events. However, those are a bit hidden in the OpenShift interface. In the next version of ODS, those events will be displayed in the Jenkins log, so you can see right away why the pipeline / deployment failed. For more details and an example, head over to https://github.com/opendevstack/ods-jenkins-shared-library/pull/385.

Polishing quickstarters

The official quickstarters are a crucial part of ODS and many have been updated for the next release, making the out-of-the-box experience much better. For example, recently the Node/TypeScript/Express quickstarter was updated to avoid using an outdated generator (https://github.com/opendevstack/ods-quickstarters/pull/326) and the Scala quickstarter agent image was updated to the latest versions and a simpler setup (https://github.com/opendevstack/ods-quickstarters/pull/335/files).

Noteworthy issues and pull requests

That's it for today - see you again in two weeks!

Issue #10 (2020-06-08)

Hello,

welcome to the 10th edition of the ODS bulletin!

What happened lately?

Separate Quickstarter configuration

With 2.x, we laid the foundation to configure custom quickstarters to use in the provisioning app. However, doing so was tricky when upgrading between versions. In 3.x, this will be easy as the config of the quickstarters has now been separated from the normal application configuration. For more information, see https://github.com/opendevstack/ods-provisioning-app/issues/363. There's also a new section in the rendered documentation that explains how to configure custom quickstarters: https://www.opendevstack.org/ods-documentation/opendevstack/3.x/provisioning-app/configuration.html#_quickstarters.

Cleaning up resource constraints

2.x already set resource constraints in many places, but there were still lots of places missing. 3.x will ship with resource constraints in every place, such as for Nexus and SonarQube (https://github.com/opendevstack/ods-core/pull/596), for webhook proxies (https://github.com/opendevstack/ods-core/pull/603), for the provisioning app (https://github.com/opendevstack/ods-provisioning-app/pull/397) and also for Jenkins agent build configs (https://github.com/opendevstack/ods-quickstarters/pull/297). Maybe you want to go through your own DeploymentConfig and BuildConfig resources to check if you have specified resource constraints for all of them?

Noteworthy issues and pull requests

That's it for now - next update in two weeks time!

Issue #9 (2020-05-20)

Hello,

another Wednesday, another edition of the ODS bulletin!

What happened lately?

Automated Deployment of OpenShift Templates via Tailor

For a long time, the Jenkins Pipeline made use of existing OpenShift resources, and just created a new image / a new deployment from them. However, this gets you only so far: when you want to make a change to a DeploymentConfig setting (e.g. add a new environment variable), you had to do this manually in the OpenShift web console, and in every namespace. Good news: in ODS 3.0, you may also control your resources as OpenShift Templates in your repository, and have them deployed automatically with the pipeline. See the documentation at https://www.opendevstack.org/ods-documentation/opendevstack/3.x/jenkins-shared-library/component-pipeline.html#_deploying_openshift_resources_from_source_code.

Further Central Namespace Improvements

As mentioned in the last bulletin, OpenDevstack will be located in one central namespace called "ods" instead of needing 3 namespaces "cd", "prov-cd" and "prov-test". Further, the provisioning app and doc gen images are now prebuilt on Docker Hub, which makes setup a lot easier. Along with those changes, we also worked to make things a bit more flexible, and now the central namespace can be configured (allowing to potentially have multiple instances of ODS running next to each other), as well as the "opendevstack" project on Bitbucket so that different sources can be referenced.

Noteworthy issues and pull requests

See you next time!

Issue #8 (2020-05-06)

Hello,

I'm back again with the 8th edition of the ODS bulletin! Let's dive right in.

What happened lately?

Central "ODS" namespace and public Provisioning App image

Up until 2.x, the shared resources (Jenkins images, SonarQube, Nexus) have been located in a central "cd" namespace. The provisioning app was deployed like a typical ODS component in a prov-cd/prov-dev/prov-test setup. This was rather tedious to install for someone who just wanted to consume the provisioning app service (like SonarQube) instead of developing it. Version 3.x will make this dramatically easier. First, the previously named "cd" namespace is being renamed to "ods" to make its purpose clearer - it is the central OpenDevStack namespace (https://github.com/opendevstack/ods-core/issues/493). Secondly, the provisioning app will be deployed into it, without the need to build it yourself. Instead, you can simply pull an image from the Docker Hub (https://hub.docker.com/r/opendevstackorg/ods-provisioning-app) which is pushed there automatically from Github Actions (https://github.com/opendevstack/ods-provisioning-app/pull/419). The image of the document generation service will follow soon.

MRO merged into general shared library

For some time we had two Jenkins shared libraries: one to build components, and one to orchestrate building, testing, deploying and creating documents of multiple repositories (MRO). Both libraries had many touch points and shared similar approaches - but duplicated quite some code because they were in separate repos. https://github.com/opendevstack/ods-jenkins-shared-library/pull/271 merged the MRO repo into the shared lib, making it easier to maintain in the future. If you're unsure what exactly the shared library offers now, have a look at the improved documentation at https://www.opendevstack.org/ods-documentation/opendevstack/3.x/jenkins-shared-library/index.html.

SonarQube and Nexus setup automation

In our quest to automate as much as possible of the setup and upgrade, we have added configuration of SonarQube (https://github.com/opendevstack/ods-core/pull/494) and Nexus (https://github.com/opendevstack/ods-core/pull/508) to the Makefile targets. So instead of having to e.g. enforce user authentication for SonarQube manually after running "make install-sonarqube", that is now done automatically. Further, this configuration is tested automatically via Github Actions - even for different SonarQube versions (https://github.com/opendevstack/ods-core/pull/500). Note that for now, Nexus configuration is limited to the initial installation.

Noteworthy issues and pull requests

And that's a wrap!

Issue #7 (2020-04-22)

Hello,

welcome to the 7th edition of the ODS bulletin! Work on 3.0 is progressing a lot, with a target release date in June. As always, the project status is tracked at https://github.com/orgs/opendevstack/projects/9.

What happened lately?

Monorepo support

ODS has only supported one repo = one component (meaning one Deployment with one container) so far. However, there are many use cases where this is not ideal (e.g. Node-based backend and frontend in one repo). Pull request https://github.com/opendevstack/ods-jenkins-shared-library/pull/242 allows to handle these use cases much easier, and paves the way to have multiple deployments from one repository, or multiple containers within one deployment.

Pull Request Decoration from SonarQube

Finally, decorating pull requests in Bitbucket with analysis from SonarQube has landed in master: https://github.com/opendevstack/ods-jenkins-shared-library/pull/243. This means that if you push a branch, and SonarQube finds a PR for it, it will add comments in the PR.

Documentation Update

Our hosted documentation (https://www.opendevstack.org/ods-documentation/) has received significant updates: it is now more targeted towards users of ODS (and information only relevant to administrators or developers has been extracted into separate areas). Also, the documentation has been re-organised internally to allow more consistency and easier contribution. Finally, the default displayed version is 2.x now. If you want to see the documentation for master, you need to switch to version "3.x Preview".

Noteworthy issues and pull requests

That's it for today!

Issue #6 (2020-04-09)

Hello,

welcome back to a new edition of the ODS bulletin! It's been a really long while since the last one ... but here's to a new start!

What happened lately?

ODS 2.x tuning

There has been lots of work on the 2.x branch to squash bugs, especially around quickstarters (just see the huge list of closed PRs there: https://github.com/opendevstack/ods-quickstarters/pulls?page=2&q=is%3Apr+is%3Aclosed. As so many things have been fixed, we'll tag this new state with v2.1 soon.

Shared library improvements

The ods-jenkins-shared-library is, as a Jenkins Pipeline which basically consists only of side-effects, tricky to test. However, some recent restructuring (https://github.com/opendevstack/ods-jenkins-shared-library/pull/202) has made testing a lot easier and with ongoing work we're improving the quality bit by bit. Further, there is a new pipeline for quickstarters now, which should make it easier to author quickstarters and which will lead to more consistency and fewer bugs. Most quickstarters still need to be converted to it, but you can see the impact on the be-golang-plain quickstarter here: https://github.com/opendevstack/ods-quickstarters/pull/222.

Noteworthy issues and pull requests

That's it for now - next update in two weeks time!

Issue #5 (2020-01-17)

Hello,

welcome to the first edition of the ODS bulletin in 2020!

What happened lately?

ODS 3.0 planning

The theme for ODS 3.0 is "quality", and we have collected some work packages that are relevant for this release at https://github.com/opendevstack/PMC/wiki/ODS-3.0-Work-Packages. If you want to join forces with an owner of a work package there, reach out! Also, you might want to work on a package that has not found an owner yet - feel free to pick it up :)

Further, https://github.com/opendevstack/PMC/wiki/ODS-3.0-Quickstarters makes quickstarter ownership for the 3.0 cycle transparent. See the "Ownership Overview" section there for goals and scope of this initiative. You'll also see that some quickstarters have not found an owner yet - please help out if you can. If no owner will be found for them, those quickstarters might have to be moved to another repository to mark their lack of maintenance.

Testing setup for ODS core

Already mentioned in the last bulletin, @hugowschneider has done some truly amazing work on adding tests for the core infrastructure (https://github.com/opendevstack/ods-core/pull/356). Definitely check it out! It also transitions the tests from Travis to GitHub Actions, which were found to be more reliable.

Noteworthy issues and pull requests

So far - see you next time!

Issue #4 (2019-12-19)

Hello,

welcome again to the ODS bulletin. It's time to celebrate as ODS version 2.0 has been released last Friday! Yay! Thanks again to all contributors - great work!

What happened lately?

ODS 2.0 release and beyond

All closed issues are referenced in https://github.com/orgs/opendevstack/projects/6. Also, each repository has an updated CHANGELOG.md where you can read up on all changes. The update guide for administrators (https://github.com/opendevstack/opendevstack.github.io/blob/master/docs/modules/ROOT/pages/update-guide.adoc) has been updated too, while a new user update guide is in the process of being written (using information collected in https://github.com/opendevstack/ods-documentation/issues/23). The next planned ODS release will be 3.0, targeted May 2020. There have been already many ideas what to do next and we will plan things in more detail in January. Stay tuned!

CPU quotas

A bit late to the game, but https://github.com/opendevstack/ods-quickstarters/issues/74 still made it into ODS 2.0. Now every quickstarter has CPU requests and limits configured!

Testing setup for ODS core

There's an exciting pull request (https://github.com/opendevstack/ods-core/pull/342) that adds testing of project creation on Travis. Testing all pieces can be quite hard and there is still a long way to go, but it's a great start. Check it out!

Noteworthy issues and pull requests

And that's a wrap for this year.

Merry Christmas to everyone and see you again in 2020!

Issue #3 (2019-12-04)

Hello,

it's me again with the third instalment of the ODS bulletin. We are getting close to ODS version 2, so a huge amount of work as happened in the last two weeks!

What happened lately?

Rundeck Removal

The pull requests to remove Rundeck from OpenDevstack have been merged, which means that master does not include Rundeck any longer. Project provisioning and quickstarter creation happens via Jenkins now! This should be a huge win.

Project specific technical user (cd_user)

To ensure that initiatives are really separated technically, ODS 2 will introduce one "cd_user" per project. Previously, there was one global user which caused a variety of problems. Unfortunately, having a distinct user has the downside of making project creation more cumbersome as the technical user will have to be created in a separate process ahead of time, and its credentials will have to be updated in the OpenShift secret once the project has been created. Please see https://github.com/opendevstack/ods-quickstarters/issues/57 with a history of the design decisions and a link to the relevant pull requests. There is one final PR (https://github.com/opendevstack/ods-provisioning-app/pull/297) missing before this change can go live - but we are close, and this change will really be worth it!

Quickstarter specifc memory quotas

In order to achieve lower memory usage in the OpenShift cluster, each quickstarter now defines its own memory quotas, see https://github.com/opendevstack/ods-quickstarters/issues/12. Changes like this are much easier to implement now in the new quickstarter structure and I am looking forward to more improvements like this!

Noteworthy issues and pull requests

Thank you to everyone who contributed to the PRs mentioned above but also on all the other issues (see https://github.com/orgs/opendevstack/projects/6 for a full list) to get ODS 2 out the door!

Issue #2

Hi OpenDevStack folks,

Welcome to the second edition of the ODS bulletin! Lots has happened in the last two weeks, so let's get right to it.

What happened lately?

Ongoing work for release 2

As mentioned last time, one major goal is to get rid of Rundeck. Now there is a first implementation to create projects and components via Jenkins pipelines. The PR for the provisioning app is https://github.com/opendevstack/ods-provisioning-app/pull/265, and the corresponding change that adds the project creation resources to ods-core is in https://github.com/opendevstack/ods-core/pull/227. The new component quickstarters live in https://github.com/opendevstack/ods-quickstarters. Browse through the code, provide feedback!

Centralised configuration

To make it easier to setup and configure ODS, we are reducing the number of required configuration files, which also prevents that parameters have to be duplicated. Further, the sample params are now part of ods-core to remove the need for a separate ods-configuration-sample repository. See PR https://github.com/opendevstack/ods-core/pull/93, which adds the new sample files and a script to manage them. Now there will be a few follow-up PRs that adapt the rest of ODS to point to the new configuration. Once that is done, the ods-configuration-sample repository will be removed.

Noteworthy issues and pull requests

That's it for today, see you again in two weeks with an update!

Issue #1

Hi OpenDevStack folks,

Welcome to the ODS bulletin 😊 I’ll try to send an email like this (roughly) every two weeks now, summarizing what happens in OpenDevStack land!

What happened lately?

2.0 release update

We finalized the scope for version 2.0 and narrowed down the release date to the week of 9th December. All issues that are part of the release are tracked in the 2.0 project on Github (https://github.com/orgs/opendevstack/projects/6). 2.0 will focus around 3 topics:

  1. Rundeck removal (to be replaced with a central Jenkins instance in the cd namespace)
  2. New quickstarter architecture (cleanup; allowing multiple repos to have e.g. private quickstarters; execution via Jenkins Pipelines within each initiative)
  3. Security improvements: no shared technical users (cd_user et al) anymore, and OAuth for SonarQube and Nexus To find out more about (1) and (2), check out https://github.com/opendevstack/ods-project-quickstarters/issues/283, which also links to a PoC.

Versioning

We are revising the versioning scheme and release management. We will move away from SemVer, which we anyway never really followed, to a simple version number which increases with every version. The next version (currently known as β€œ2.0”) will simply be β€œ2”, followed by β€œ3”, then β€œ4” and so on. A new major version is expected every half year (likely in May and November). Patch releases are made as necessary. Further, all pointers (e.g. to the shared library, Jenkins master and agents) will target those major versions to ensure stability for projects.

Noteworthy issues and pull requests

See you again in two weeks with an update!