Docker - sgml/signature GitHub Wiki

Golang Docker API

docker_golang_api_advantages:
  - programmatic_control:
      description: >
        The Go API allows developers to programmatically interact with Docker, enabling
        seamless automation and integration into applications. This provides more flexibility
        than manually executing Docker CLI commands.
      url: https://pkg.go.dev/github.com/docker/docker/client
  - fine_grained_control:
      description: >
        The Go API provides low-level control over Docker objects (e.g., containers, images,
        networks, volumes), allowing detailed customization of workflows.
      url: https://docs.docker.com/engine/api/
  - error_handling:
      description: >
        With the Go API, developers can implement robust and precise error handling directly in
        their applications, unlike parsing CLI error messages.
      url: https://pkg.go.dev/github.com/docker/docker/client#Client
  - integration_with_go_apps:
      description: >
        The Go API seamlessly integrates with Go applications, eliminating the need to
        invoke external processes or parse CLI output.
      url: https://pkg.go.dev/github.com/docker/docker/client
  - asynchronous_operations:
      description: >
        The Go API supports non-blocking, asynchronous operations for tasks such as
        container creation, which can improve efficiency in complex workflows.
      url: https://docs.docker.com/engine/api/v1.41/
  - custom_tooling:
      description: >
        Developers can create custom tools tailored to specific use cases using the API,
        such as deployment pipelines, monitoring tools, and container orchestration systems.
      url: https://pkg.go.dev/github.com/docker/docker
  - access_to_events:
      description: >
        The Go API enables subscribing to Docker events, such as container start/stop or
        image pull completion, which allows for real-time event-driven workflows.
      url: https://docs.docker.com/engine/api/v1.41/#tag/System/operation/SystemEvents
  - cross_platform_compatibility:
      description: >
        Applications built with the Go API can interact with Docker on any platform
        (Linux, macOS, Windows), ensuring consistent behavior across environments.
      url: https://pkg.go.dev/github.com/docker/docker
  - reduced_overhead:
      description: >
        By using the Go API, developers avoid spawning new processes to execute CLI commands,
        reducing latency and improving performance.
      url: https://pkg.go.dev/github.com/docker/docker
  - version_compatibility:
      description: >
        The Go API allows specifying the Docker API version to ensure compatibility
        with specific Docker Engine versions, providing greater control over behavior.
      url: https://docs.docker.com/engine/api/version-history/

Cookbook

# Status
systemctl status docker.service

# Stop
ps axf | grep docker | grep -v grep | awk '{print "kill -9 " $1}' | sudo sh
systemctl stop docker

# Start
systemctl start docker

# The Bad Parts
* https://dagster.io/blog/fast-deploys-with-pex-and-docker

# Verify Certificate Chain
openssl verify -CAfile RootCert.pem -untrusted Intermediate.pem UserCert.pem

# Verify Self-Signed Certificate
openssl verify -CAfile test.crt test.crt

QA

Error guessing is a testing technique that leverages the tester’s experience, intuition, and knowledge of the system to predict where defects might occur. Here's a general methodology:

  • Leverage Experience: Testers use their past experience with similar applications or common error patterns to guess where errors might occur in the current application.

  • Understand the System: Testers need a good understanding of the system, its context, and its users. This knowledge can help them predict what might go wrong.

  • Identify Risky Areas: Certain parts of an application might be more prone to errors. These could be complex features, areas where new changes have been made, or parts of the application that have had issues in the past.

  • Design Test Cases: Based on their guesses, testers design and execute test cases specifically to expose potential errors.

  • Learn and Adapt: As testers find defects, they learn more about the system’s behavior and can refine their error guessing strategy.

Daemon

https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html

Engine

Mutual TLS

Client

Troubleshooting

Five W's

Internals

Optimized Images

Commands

Exec

docker exec -it postgres-localhost bash
psql -U postgres
CREATE DATABASE mytestdb;

Volume Mappings

docker volume create --name my_dag_log_volume --opt type=none --opt device=$PWD/airflow/logs --opt o=bind
    volumes:
      - my_dag_log_volume:/usr/local/airflow/logs
volumes:
  my_dag_log_volume:

Concepts

https://docs.docker.com/glossary/

https://dockerbook.com/TheDockerBook_sample.pdf

https://docs.docker.com/compose/release-notes/

https://docs.docker.com/compose/gettingstarted/

https://vsupalov.com/docker-arg-env-variable-guide/

https://www.techrepublic.com/article/what-is-the-difference-between-dockerfile-and-docker-compose-yml-files/

YAML

https://github.com/compose-spec/compose-spec/blob/main/spec.md

https://docs.docker.com/config/containers/live-restore/

https://docs.docker.com/compose/compose-file/

https://github.com/docker-library/postgres/issues/581

Debugging

https://www.joyfulbikeshedding.com/blog/2019-08-27-debugging-docker-builds.html

https://medium.com/@betz.mark/ten-tips-for-debugging-docker-containers-cde4da841a1d

https://rasa.com/docs/rasa-x/installation-and-setup/docker-compose-manual/

Directives

https://design.jboss.org/redhatdeveloper/marketing/docker_cheatsheet/cheatsheet/images/docker_cheatsheet_r3v2.pdf

https://duo.com/decipher/docker-bug-allows-root-access-to-host-file-system

Environment Variables

https://docs.docker.com/compose/environment-variables/

https://www.quora.com/Is-there-a-way-to-change-environment-variables-from-outside-a-Docker-container-If-we-have-a-server-already-running-is-there-a-clever-technique-we-can-use-to-change-the-environment-variables-without-rebuilding-the

https://code.visualstudio.com/docs/remote/containers-advanced

https://docs.docker.com/compose/reference/envvars/

User/Group Permissions

https://docs.docker.com/engine/install/linux-postinstall/

https://dev.to/acro5piano/specifying-user-and-group-in-docker-i2e

https://stackoverflow.com/questions/40462189/docker-compose-set-user-and-group-on-mounted-volume

https://code.visualstudio.com/docs/containers/python-user-rights

https://medium.com/@nielssj/docker-volumes-and-file-system-permissions-772c1aee23ca

https://stackoverflow.com/questions/52754149/operation-of-the-mkdir-command-with-dockerfile/52754409

https://github.com/moby/moby/issues/20920

https://stackoverflow.com/questions/52737690/how-to-copy-a-directory-over-the-docker-cp

Race Conditions

https://labouardy.com/preventing-race-conditions-in-docker/

https://github.com/moby/moby/issues/26768

https://github.com/moby/moby/issues/38064

https://www.digitalocean.com/community/tutorials/how-to-debug-and-fix-common-docker-issues

https://www.digitalocean.com/community/questions/how-to-fix-docker-got-permission-denied-while-trying-to-connect-to-the-docker-daemon-socket

Wordpress

https://devtidbits.com/2019/11/02/customise-wordpress-on-docker/

Postgres

https://github.com/Radu-Raicea/Dockerized-Flask/wiki/[Docker]-Access-the-PostgreSQL-command-line-terminal-through-Docker

https://stackoverflow.com/questions/26598738/how-to-create-user-database-in-script-for-docker-postgres

https://davejansen.com/how-to-set-up-and-use-postgres-using-docker/

https://markheath.net/post/exploring-postgresql-with-docker

https://medium.com/better-programming/connect-from-local-machine-to-postgresql-docker-container-f785f00461a7

https://medium.com/coderbunker/rapid-api-development-tutorial-with-docker-postgresql-postgraphile-e387c1c73dd8

https://hackernoon.com/dont-install-postgres-docker-pull-postgres-bee20e200198

https://medium.com/@mikaelino/querying-a-database-with-graphql-and-dataloader-an-introduction-in-go-d8d2609bc635

https://medium.com/@harshityadav95/postgresql-in-windows-subsystem-for-linux-wsl-6dc751ac1ff3

https://medium.com/@stephanedmonson/solution-for-connecting-postgresql-via-wsl-windows-subsystem-for-linux-ubuntu18-c79940fa5742

Docker Engine API

https://docs-stage.docker.com/engine/api/v1.24/

https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-docker/configure-docker-daemon

https://jackiechen.org/2015/04/20/understanding-docker-directory-structure/

Compose

https://docs.docker.com/compose/compose-file/compose-file-v2/

https://gist.github.com/eliashussary/379e44a99e2389bd6a8ea6a23c2d5af8

https://bobcares.com/blog/docker-change-container-configuration/

https://linuxhint.com/run_postgresql_docker_compose/

https://stackoverflow.com/questions/44198512/howto-pass-postgres-user-env-variable-when-using-docker-compose-yml-for-docker

Inspect

https://www.mankier.com/1/docker-inspect

Debian

https://wiki.debian.org/nftables

https://lwn.net/Articles/676831/

Network

https://docs.docker.com/network/

https://stackoverflow.com/questions/36454955/docker-and-netstat-netstat-is-not-showing-ports-exposed-by-docker-containers

https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/

https://forums.docker.com/t/docker-pull-not-using-correct-dns-server-when-private-registry-on-vpn/11117

Clustering

https://www.godaddy.com/engineering/2018/05/02/kubernetes-introduction-for-developers/

https://azure.microsoft.com/en-us/resources/cloud-computing-dictionary/what-is-kubernetes/#beyond-kubernetes

File/Directory/Volume Bind Mounts

X11

Containers

https://docs.docker.com/config/containers/live-restore/

Startup/Shutdown

https://docs.docker.com/config/daemon/systemd/

https://coreos.com/os/docs/latest/customizing-docker.html

http://www.softpanorama.org/Commercial_linuxes/Startup_and_shutdown/systemd.shtml

http://www.softpanorama.org/Commercial_linuxes/RHEL/index.shtml

https://www.slideshare.net/SreenivasMakam/docker-networking-common-issues-and-troubleshooting-techniques

https://docker-py.readthedocs.io/en/stable/volumes.html

https://docs.docker.com/storage/volumes/

Daemon

https://docs.docker.com/config/daemon/

https://docs.docker.com/engine/security/https/

https://www.bennadel.com/blog/3419-from-noob-to-docker-on-digitalocean-with-nginx-node-js-datadog-logs-dogstatsd-and-letsencrypt-ssl-certificates.htm

https://www.linkedin.com/pulse/use-apache-ssllets-encrypt-docker-leon-sczepansky/

https://docs.docker.com/engine/reference/commandline/dockerd/

http://dockerlabs.collabnix.com/beginners/components/daemon/

https://support.coreos.com/hc/en-us/articles/115002311127-Debugging-docker-daemon-Issues

https://github.com/docker/cli/issues/2104

https://serverfault.com/questions/681571/docker-exposed-ports-accessible-from-outside-iptables-rules-ignored

Compile from Source(s)

Gotchas

One issue that occurs semi-regularly for LtU is that bot attacks - often just from overly aggressive crawlers - can cause the database container to run out of resources and crash. In that case, Kubernetes usually restarts the container and everything recovers automatically within a few minutes.

This process worked well for a few years, but at some point, it stopped working seamlessly. Even after manual attempts to recover, there would still be an intermittent but frequent symptom where it seemed like the app was still trying to reach the old crashed container, or something like that. I spent a bit of time looking at it, but soon decided it would be easier to switch to a new cluster than debug that years-old setup.

Git Commit in a Docker Container

# Once you're in the container, you can install git.
apt-get update
apt-get install -y git

# Configure your Git user (replace with your own name and email).
git config --global user.name "Your Name"
git config --global user.email "[email protected]"

# Clone your repository (replace with your repository's URL).
git clone https://github.com/username/repository.git

# Navigate into your repository.
cd repository

# Make changes to your files here.

# Stage and commit your changes.
git add .
git commit -m "Your commit message"

# Push your changes (replace 'main' with your branch name if different).
git push origin main

Environment Variables

{
  "DOCKER_CERT_PATH": "Path to the directory containing Docker TLS certificates (e.g., ca.pem, cert.pem, key.pem). Used for secure communication with the Docker daemon.",
  "DOCKER_HOST": "Specifies the address of the Docker daemon (e.g., tcp://localhost:2376). Determines where Docker CLI communicates with the Docker engine.",
  "DOCKER_TLS_VERIFY": "Set to '1' to enable TLS verification for secure communication with the Docker daemon. If unset or set to '0', TLS verification is disabled.",
  "DOCKER_API_VERSION": "Specifies the version of the Docker API to use. For compatibility, it's recommended to set this explicitly.",
  "DOCKER_BUILDKIT": "Enables BuildKit, an advanced build subsystem for Docker. Set to '1' to enable BuildKit features during image builds.",
  "DOCKER_REGISTRY": "Specifies the default registry to use for pulling images. Useful when working with private registries.",
  "DOCKER_USERNAME": "Username for authentication when pushing or pulling images from a registry.",
  "DOCKER_PASSWORD": "Password for authentication when pushing or pulling images from a registry.",
  "DOCKER_IMAGE_TAG": "Default tag to use when pulling or pushing images. If not specified, 'latest' is assumed.",
  "DOCKER_NETWORK": "Specifies the default network mode for containers (e.g., 'bridge', 'host', 'none').",
  "DOCKER_LOG_DRIVER": "Sets the default logging driver for containers (e.g., 'json-file', 'syslog', 'journald').",
  "DOCKER_STORAGE_DRIVER": "Specifies the storage driver for managing container filesystems (e.g., 'overlay2', 'aufs', 'zfs')."
}
{
  "options": {
    "-inform": "DER|PEM|NET",
    "-outform": "DER|PEM|NET",
    "-in": "filename",
    "-out": "filename",
    "-md2|-md5|-sha1|-mdc2": "digest to use",
    "-engine": "id",
    "-text": "prints out the certificate in text form",
    "-certopt": "customizes the output format used with -text"
  },
  "descriptions": {
    "-inform": "Specifies the input format (DER, PEM, or NET)",
    "-outform": "Specifies the output format (DER, PEM, or NET)",
    "-in": "Input filename to read a certificate from",
    "-out": "Output filename to write to",
    "-md2|-md5|-sha1|-mdc2": "Digest options for signing or display",
    "-engine": "Specifies an engine by its unique id",
    "-text": "Displays full certificate details",
    "-certopt": "Customizes the output format used with -text"
  }
}
+---------------------+
|                     |
|   Kubernetes API    |
|                     |
+---------------------+
          |
          v
+---------------------+
|                     |
|   Kubelet (Windows) |
|                     |
+---------------------+
          |
          v
+---------------------+
|                     |
|   Containerd        |
|                     |
+---------------------+
          |
          v
+---------------------+
|                     |
|   Windows Host OS   |
|                     |
+---------------------+
          |
          v
+---------------------+
|                     |
|   Windows Containers|
|                     |
+---------------------+

MTLS

package main

import (
    "crypto/rand"
    "crypto/rsa"
    "crypto/x509"
    "crypto/x509/pkix"
    "encoding/pem"
    "math/big"
    "os"
    "time"
)

func generateCert() {
    // Generate a private key
    priv, _ := rsa.GenerateKey(rand.Reader, 2048)

    // Create a certificate template
    template := x509.Certificate{
        SerialNumber: big.NewInt(1),
        Subject: pkix.Name{
            Organization: []string{"My Organization"},
        },
        NotBefore: time.Now(),
        NotAfter:  time.Now().Add(365 * 24 * time.Hour),
        KeyUsage:  x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature,
        ExtKeyUsage: []x509.ExtKeyUsage{
            x509.ExtKeyUsageServerAuth,
            x509.ExtKeyUsageClientAuth,
        },
        BasicConstraintsValid: true,
    }

    // Create a self-signed certificate
    certDER, _ := x509.CreateCertificate(rand.Reader, &template, &template, &priv.PublicKey, priv)

    // Save the certificate and key to files
    certOut, _ := os.Create("cert.pem")
    pem.Encode(certOut, &pem.Block{Type: "CERTIFICATE", Bytes: certDER})
    certOut.Close()

    keyOut, _ := os.Create("key.pem")
    pem.Encode(keyOut, &pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(priv)})
    keyOut.Close()
}

func main() {
    generateCert()
}

Postgresql.conf

ssl = on
ssl_cert_file = 'cert.pem'
ssl_key_file = 'key.pem'

References

https://github.com/golang/go/blob/master/src/crypto/tls/example_test.go"

⚠️ **GitHub.com Fallback** ⚠️