FAQ - microsoft/bedrock GitHub Wiki

General

Why Bedrock?

Bedrock is a set of automation, tooling and infrastructure for deploying production-level Kubernetes clusters with a GitOps flow. The goal is to make development teams productive through automating DevOps best practices through Continuous Integration and Continuous Delivery. As a developer, a commit into application code will pass through the GitOps pipeline and end up in production Kubernetes cluster with a single approval click or code commit.

Setup

How do I generate a deploy key for flux?

First, generate an ssh key locally:

$ ssh-keygen -t rsa -b 4096 -C "[email protected]"

This creates a new ssh key, using the provided email as a label.

> Generating public/private rsa key pair.

When you're prompted to "Enter a file in which to save the key," enter any name for this file. It will be saved in your current directory.

Now, start the ssh-agent in the background

$ eval "$(ssh-agent -s)"

Add your private key to the ssh-agent:

$ ssh-add -K <path to your newly created private key>

Now we're ready to add this deploy key to the repository. If you're using GitHub, navigate to the repository's settings and click on Deploy Keys > Add deploy key and paste the newly created public key, with any title. Make sure to give write access to this deploy key as flux will show errors otherwise. If you're using Azure DevOps, the steps are documented here.

When you're ready to start a deployment via terraform, copy the path to the private key you created using these steps above and use it for gitops_ssh_key in terraform.tfvars. (More info here)

How do I edit the flux deployment details?

Run command kubectl edit deployment/flux -n flux to load the deployment file for flux, and make the necessary modifications. Then exit out of the editor as you save the file, and run kubectl describe deployment flux -n flux to make sure your changes are reflected in the flux instance!

How do I edit the deploy key for an existing flux instance?

Make sure you have the full absolute path to your new deploy key. Then run the following command:

kubectl create secret generic flux-ssh --from-file=identity="<absolute-path-to-your-newly-created-deploy-key>" -n flux

If you get error that the key already exists, run kubectl delete secret flux-ssh -n flux and then try the command above once again.

GitOps

How does Bedrock automate deployment and make sure clusters are up to date?

Bedrock uses Flux to synchronize all manifests in a repository with a Kubernetes cluster and monitors container registries for new images while updating the manifests accordingly. We also include alerting using Prometheus which would help monitor the deployments in case of any issues.

How do I use an orchestrator besides Azure DevOps?

Bedrock at the moment is setup to work end-to-end with Azure DevOps, but plans are in place to support Travis, CircleCI, Jenkins, Spinnaker, Tekton and Brigade. We're also welcoming pull requests for anything that is currently not supported.

What are environments?

Different environments are needed for an application to have fool proof deployment and user experience, for example a different environment for production and development. Bedrock supports deploying multiple platforms using the terraform config in azure-multiple-clusters.

What is Fabrikate?

Fabrikate simplifies the front end of the GitOps workflow: it takes a high level description of your deployment, a target environment config (such as prod or staging), and renders the Kubernetes resource manifests for that deployment utilizing templating tools like Helm.

What is a high level definition (HLD)?

High level definition is a description of the Kubernetes deployment which includes dependencies on microservices and any external infrastructure (stacks) they consume. We use a tool called Fabrikate to install dependencies specified in this high level definition and create the Kubernetes manifest yaml definitions that are necessary to setup the cluster.

How do I get started on making a high level definition (HLD)?

Here is a good guide on how to get started writing a high level definition.

How does the high level definition (HLD) support multiple environments and clusters?

You can specify multiple configurations in a high level definition and fabrikate can build combinations of these environments. For example:

alt text

If you would like to generate prod-east from this configuration, simply type fab generate prod-east and staging-central using fab generate staging-central for example.

When this configuration is generated for all specified environments and lands in the manifest repository, we get manifest folders generated for each of the configs as shown below.

alt text

Flux can be configured in each cluster to listen to a specific path in this repository and it will only apply changes that happen in that folder.

How does the HLD support multiple environments on a single cluster?

Even though we recommend using multiple clusters for multiple environments (for simplicity and security purposes), there's ways we can configure all of them to run on a single cluster, for example by isolating them with a namespace prefix. In this case, a single instance of flux would update all the environments at once.

What is environment promotion?

Environment promotion is the process dev teams use to move across different environments, for example when smoke tests and QA tests are performed on a build, it's ready to be deployed to staging. When the working functionality is verified in staging, it's ready to be deployed to production.

alt text

Some teams perform a swap between staging and production so that they can swap back in case of any errors or failures that were not caught during testing.

How does bedrock perform environment promotion?

Bedrock updates the image tag that is referred in the high level definition, for let's say the dev environment. When the HLD receives an update, it fires the manifest yaml generation pipeline for the dev environment and produces manifest files which are pushed into the manifest repository folder dev/. Flux picks up this change and applies it to the Kubernetes cluster automatically.

alt text

These steps can be repeated for n amount of environment promotions depending on the number of Kubernetes environments a dev team maintains for an application.

When the user/operator is ready to promote this dev environment to QA, depending on the rules for this environment, they may kick off a release for QA which will perform the repeat steps as above for the QA environment, and the cycle continues.