Application Development and DevOps - BKJackson/BKJackson_Wiki GitHub Wiki

Subpages

Python apps

Awesome Python Applications - Free software that works great, and also happens to be open-source Python.

Tutorials

How To Integrate Unit Testing, Linting, and Continuous Integration Into Your Python Projects - Earth Lab
Enabling CI/CD for Machine Learning project with Azure Pipelines - Great lab with exercises on how to create a CI/CD pipeline for ML/AI projects in Azure, Last updated Aug 21, 2019
How to CI and CD a Node.JS Application Using GitHub Actions - Learn how to CI/CD with GitHub Actions and Docker, Sept. 2, 2019
Deploying Machine Learning Models

Articles

The Relationship Between Microservices, Docker and Kubernetes
What does it mean to “productionize” data science? - If we drop awesome technology into a sloppy system, the technology won’t do anything but let us look sexy as we fail.
Data science productionization: portability - Portability decreases the time it takes to get value from your code by decreasing the amount of code you have to rewrite when your goals change.
Google Cloud Continuous Delivery Platform - End-to-end automation from source to production.

DevOps and Data Science/Machine Learning

DeepOps: Accountability and reproducibility in deep learning through DeepOps

Licenses

Choosealicense.com

Why containers are great

"Containers are an ideal delivery vehicle for microservices – a software architecture that is becoming increasingly important when building large, distributed systems that need to scale and be resilient." Source

Containers vs. Virtual Machines - Explained by 12 Docker pros

Short one-liner explanation: Containers are processes. Virtual machines are servers.

Combining build-deploy tools

Configuring Kubernetes Docker Deployments With Makefiles and Envsubst - Useful for a project that requires frequent Docker builds & pushes to a Kubernetes cluster.
Rewriting our deploy tooling: from Makefile to Bash and back again - GNU Make as part of deployment tooling for a Kubernetes clusters in staging and production AWS accounts and testing with dynamic stacks.

Does this need to be automated now? - The Automation Checklist

When you ask, “Does this need to be automated now?” run through the following checklist:

  1. How frequently is the process or scenario repeated?
  2. How long is the process?
  3. What people and resource dependencies are involved in the process? Are they causing delays in CI/CD?
  4. Is the process error-prone if it is not automated?
  5. What is the urgency in getting the process automated?

Using this checklist, you can prioritize the steps in a CI/CD implementation. First and foremost, automate the process for compiling code. Ideally, you will integrate code multiple times per day (1). Manually, the process takes a few minutes to a couple of hours (2). That stalls output until the compiler finishes the task (3). It is also susceptible to human error (4), and because CI/CD is a pipe dream without automated integration, this is urgent (5). Source

Continuous Integration (CI)

Continuous integration (CI) is a process in which developers and testers collaboratively validate new code. Traditionally, developers wrote code and integrated it once a month for testing. That was inefficient—a mistake in code from four weeks ago could force the developers to revise code written one week ago. To overcome that problem, CI depends on automation to integrate and test code continuously. Scrum teams using CI commit code daily at the very least, while a majority of them commit code for every change introduced. Source

Continuous Delivery (CD)

Continuous delivery (CD) is the process of continuously creating releasable artifacts. Some companies release to users once or even multiple times a day, while others release the software at a slower pace for market reasons. Either way, the ability to release is tested continuously. Continuous deployment is possible thanks to cloud environments. Servers are set up such that you can deploy to production without shutting down and manually updating servers. Source

The 3 Musketeers: How Make, Docker and Compose enable us to release many times a day - Feb. 1, 2018

Continuous Deployment (also CD)

Continuous deployment is the concept that every change made in the code base will be deployed almost immediately to production if the results of the pipeline are successful. This is terrifying to most organizations because rapid product changes can scare away users.

Companies believe that if they do not practice continuous deployment, they are not doing CD. They fail to distinguish between continuous deployment and continuous delivery.

Continuous delivery is the concept that every change to the code base goes through the pipeline up to the point of deploying to nonproduction environments. The team finds and addresses issues immediately, not later when they plan to release the code base.

The code base is always at a quality level that is safe for release. When to release the code base to production is a business decision.

Whereas continuous deployment unsettles most organizations, continuous delivery resonates with them. Continuous delivery gives them control over product rollout, functionality, and risk factors. There is time for alpha testing, for beta customers, for early adopters, and so on.

Continuous Integration and Continuous Delivery (CI/CD)

Thus, CI/CD is a process for continuous development, testing, and delivery of new code. Some companies like Facebook and Netflix use CI/CD to complete 10 or more releases per week. Other companies struggle to hit that pace because they succumb to one or more of five pitfalls:
5 common pitfalls of CI/CD—and how to avoid them - Mar. 28, 2018

CI/CD pitfalls

  1. Automating the wrong processes first
  2. Confusing continuous deployment for continuous delivery
  3. Lack of meaningful dashboards and metrics
  4. Lack of coordination between continuous integration and continuous delivery
  5. Balancing the frequency of running continuous integration jobs and resource utilization

Companies like Netflix can complete integration, testing, and delivery in a matter of two to three hours. They established a system that passes code from hand to hand without indecision and discussion.

Software Doneness and Technical debt

DoD Working Document: Software is Never Done: Refactoring the Acquisition Code for Competitive Advantage
What is technical debt and what should you do about it?
Software project estimation: handling the dreaded ‘how long will it take?’
A programmer’s work is never done
Why Can't We Tell When Software is Done?
Software is never finished, only released
5 Reasons Modern Software is Never “Done”