The Current State of Compliance

Two Years?

Source

I was working for a large systems integrator on my very first project in my career.

I started as a software engineer building large custom systems for large organizations. Even in those days, I was always interested in and participated in building the end to end systems - from security, to app development, build and deployment, databases, messaging, and installations. What you might call a full-stack developer - well before there was such a thing.

One of the key lessons I learned on this project was what motivates us. Releasing and receiving software motivates both the creators (engineers and such) and the users. Not silly hawaiian shirt days or contrived "team building" exercises. It is about releasing software.

After two long years on this project, I was thrilled when the software I helped create was in use at a hospital logistics center. I immediately knew that this was what motivated my work. I think if you asked most people developing and delivering software systems and then talk with users, you would get near universal agreement. Yet, we waited two years to release the software system to users. The primary reason for this delay was system complexity. This complexity included a 75+ person development team across many teams, software that had to be manually installed at the logistics centers, etc. So, the work got batched and delayed and delayed. I found it maddening that it would take us days to get just the application portion (i.e. not the environment or databases) of the software system built and deployed within a simple shared integration environment.

It was also while I was on this project that I saw a single-page industry magazine advertisement with a picture of a keyboard button and the word "Integrate" on it. To me, it symbolized the means to the end. The more we could integrate the complete software system from recent changes, the better chance we could regularly release software to users while collectively motivating the software users and the software creators. This is what motivates me every day and why I create continuous delivery systems for customers today. With Continuous Delivery, you are integrating the entire software system (application code, configuration, infrastructure and data) with every code commit. And - as I will show you - things like security and compliance as well. In doing this, you are reducing the time it takes to release software to users while providing rapid and actionable feedback to team members. This gets both the users and creators closer to our collective goals.

"These go to eleven"

So, I have always been interested in turning things up to 11. That is what I will be going over in this LiveLesson: how to apply this "continuous mindset" to compliance. Treat compliance as a code asset that can be versioned, tested, and run with every change. Break down the organizational silos that can occur and ensure that compliance is something that is everyone's job.

How does compliance work in many organizations? From what I have seen, it might look something like this:

Internal compliance is conducting an audit in three weeks, we need to make sure we are compliant with all of our internal policies.

A heroic effort then ensues of people discovering or creating documents and other artifacts that detail their policies, along with checklists stipulating that they have been executed against "the software system". This often includes signed approvals from bodies like Change Advisory Boards and such. If the scope of the audit is at the enterprise level, people might spend late nights and weekends preparing for the audit and running through mock Q&As. After the audit, the team receives a report based on a snapshot in time and—viewed through a cynical lens—people try to determine who was most at fault for any issues that arose.

Most of the typical activities involved in "compliance" are nothing more than "Risk Management Theatre": a mere spectacle that has very little to do with assessing or reducing actual risk. These activities often focus on human failure versus systems failure (aka: "someone to blame" without fixing the root cause of the problem). Moreover, the activities are focused on a single point in time—a single event—and do not consider the reality that software is a dynamic system.

Challenges

• Inability to Scale – Inability to adapt to the speed of change at scale. In the cloud, large scale can be achieved instantly, and manual compliance cannot adapt to this type of scale.
• Inability to Properly Assess – Inability to quickly and accurately assess infrastructure state changes. Even before the cloud, infrastructure changes that might have violated an organization’s policies were regularly occurring without the knowledge of those who managed this infrastructure. With the ability to make changes through APIs via the cloud, these changes are can be exponential, which makes them extremely difficult to assess without a systematic and automated approach.
• Compliance “Events” – Monumental efforts to comply with internal and external audits. Martin Fowler is fond of saying that Continuous Integration makes integration a "nonevent". Without Continuous Compliance, audits and other reviews become an event that take up a lot of time and resources.
• Manual Efforts – Repeatedly performing the same manual efforts to achieve “compliance” with organizational policies. Without continuous compliance, your organization is always “noncompliant” because of the rapid change in state and the inability to assess compliance without a fully-automated approach.
• Less Effective Feedback and Release Less Often – Reduce product teams’ ability to increase the speed of change for their end users. Because compliance is performed manually, either fully or partially, there is often a separate event in the release lifecycle during which a person or team audits the software system. Since there is minimal integrity in a manual process, the results are suspect and often do not reduce risk. Even worse, because it takes so much time, teams will forego these checks thereby exposing them to greater risk. This approach reduces effective feedback from end users and increases risk when software systems are noncompliant with the organization’s policies.

Continuous Delivery for Compliance

If that is the current state of compliance, I want you to consider how we will fix the problems that occur when you have organizational silos, manual processes, compliance "events", and so on.

The good news is that the answer is that the answer is right under our nose. In this LiveLesson, you will learn how you can apply these "continuous" practices in which we automate and test compliance on every change. To conclude this section, I want you to watch a short video on continuous delivery and consider how we can apply continuous delivery to compliance as the entire purpose of this LiveLesson is for you to learn how to apply continuous delivery practices to compliance.

One way to visualize Continuous Delivery is to imagine a factory assembly line where cross-functional engineers work on various parts of the software as it moves in its lifecycle.

In the past, teams delivered software only every few months or so, because of manual, error-prone processes. Often pieces of the delivery package went back and forth between teams separated by silos. This was a long, drawn-out, and expensive exercise, with the final package assembled at the 11th hour with teams vowing never to repeat it.

With Continuous Delivery, software can be delivered several times a day, once a week or as often as you want. It is always in a releasable state.

When compared to traditional delivery methods, Continuous Delivery offers two keys advantages. The first is that the software is always ready to release, meaning you do not stop and make a special effort to release the software. The second is that there are not any walls between the teams. Instead, there are autonomous teams made up of developers, testers, infrastructure, etc. all contributing to a single path to production.

Continuous Delivery is not exclusive to application developer changes. No matter whether it is a change to the infrastructure, data, or whatever, the entire package is built, tested, analyzed, and deployed. When it passes all these checks, it becomes a release candidate and could potentially be released to users, if the business chooses to do so.

When a problem is discovered, a quick feedback loop ensures that team members are notified that something is wrong and needs to be corrected, and the assembly line stops. This is when team members prioritize correcting the error and committing it so that it moves along in the path to production.

In summary, Continuous Delivery is concerned with all parts of the software system and how all team members work as part of this single path to production. This way, software gets to users quicker and in a more predictable manner.