The Three Musketeers Explained - BKJackson/BKJackson_Wiki GitHub Wiki

Additional Resources

• About the 3 Musketeers
• 3 Musketeers Hello World Example
• 3 Musketeers - Cookiecutter Echo - A very simple Cookiecutter template to show the 3 Musketeers in action.
• 3 Musketeers Docker Cookiecutter containerized with automatic updates - Example of pipelines with GitLab and Docker Hub uses 3 Musketeers to be tested, built, and deployed.
• 3 Musketeers Make Docs

What is it?

In a nutshell, the 3 Musketeers is a pattern for developing software in a repeatable and consistent manner.

The pattern now uses the combination of Docker, Docker Compose, and Make, which we’ve dubbed “The 3 Musketeers”.

It leverages Make as an orchestration tool to test, build, run, and deploy applications using Docker and Docker Compose. The Make and Docker/Compose commands for each application are maintained as part of the application’s source code and are invoked in the same way whether run locally or on a CI/CD server.

The Tools

Docker

Docker is the most important musketeer of the three. I’ve always loved Docker since I started learning it years ago. It changed the way I perceived software development: testing, building, running, and deploying can all be done inside a lightweight Docker container — which can be run on different operating system.

• Install Docker Community Edition

Docker Compose

Docker Compose, or simply Compose, manages Docker containers in a very neat way. It allows multiple Docker commands to be written as a single one, which allows our Makefile to be a lot cleaner and easier to maintain. Testing also often involves container dependencies, such as a database, which is an area where Compose really shines. No need to create the database container and link it to your application code container manually — Compose takes care of this for you.

• Install Docker Compose
• flemay/docker-musketeers - A lightweight Docker image with Docker, Compose, make, zip, git, curl, and Cookiecutter

Make

Make is a cross-platform build tool to test and build software and it is used as an interface between the CI/CD server and the application code. A single Makefile per application defines and encapsulates all the steps for testing, building, and deploying that application. Of course other tools like rake or ant can be used to achieve the same goal, but having Makefile pre-installed in many OS distributions makes it a convenient choice.

The Makefile contains targets that reflect the application’s life cycle like test, build, and deploy.

It is common for each of these stages to have dependencies. Test and build may require development packages, whereas deploy may need the application as a build artifact prior deploying.

From a CI/CD perspective, each stage can be executed on a different agent, and therefore, it is important to check the presence of the dependencies before executing the tasks.

For example, our simple application has few Go dependencies defined in Gopkg.toml that are needed for testing and building. Thus Makefile has a target deps which follows a specific pattern: it calls the Compose service golang to execute the target _depsGo inside a Docker container.

_depsGo installs the required packages using the Go dependency management tool dep. It also makes an artifact $(GOLANG_DEPS_ARTIFACT) as a zip file for dependencies to be passed along through the stages. This step is quite useful as it acts as a cache and means subsequent CI/CD agents don’t need to re-install the packages again when testing and building.

deps:
    docker-compose run --rm golang make _depsGo 
_depsGo:
    dep ensure zip -rq $(GOLANG_DEPS_ARTIFACT) $(GOLANG_DEPS_DIR)/

test: $(GOLANG_DEPS_DIR) 
    docker-compose run --rm golang make _test 
_test:
    go test -v

build: $(GOLANG_DEPS_DIR)
    docker-compose run --rm golang make _build 
_build:
    GOOS=linux go build -o bin/main zip -r $(PACKAGE).zip bin

deploy: $(ARTIFACT_NAME)
    docker-compose run --rm serverless make _deploy 
_deploy:
    rm -fr .serverless sls deploy -v  

What we have now is a Makefile with the targets based on the life cycle of our simple application forming a pipeline. Each of the stages relies on Docker and Compose to be executed. This means the pipeline can be tested and executed on Windows, Linux, and MacOS. In other words, it is run the same way whether it is local or on a CI/CD server.

Environment variables and the .env file

Development following the twelve-factor app uses the environment variables to configure their application.

Often there are many environment variables and having them in a .env file becomes handy. Docker and Compose do use environment variables file to pass the variables to the containers.

• 3 Musketeers Environment variables & .env file Doc

Source: The 3 Musketeers: How Make, Docker and Compose enable us to release many times a day