DockerQuickNotes - henk52/knowledgesharing GitHub Wiki

Docker

Open issues

  • How to provide ssh .pub to a container during startup
    • test volume use after fixing the HOME thing

References

  • Hec18 - Kubernetes for developers, Joseph Heck

  • Dont use the 'docker' group:

  • Network

Docker vocabulary

  • Docker hub: docker repository. Default: index.docker.io
  • Docker image: A collection of all the files that make up an executable software application.
    • This collection includes the application plus all the libraries, binaries, and other dependencies such as the deployment descriptor, just needed to run the application everywhere.
    • files in the Docker image are read-only and hence the content of the image cannot be altered. If you choose to alter the content of your image, the only option Docker allows is to add another layer with the new changes. In other words, a Docker image is made up of layers
    • Aka: 'Dockerized application'.
  • Base image: Bottom layer image, that only holds the base necessities to provide an operational entity, no kernel nor device drivers.
  • Docker Registry: a place where Docker images can be stored in order to be publicly or privately found, accessed, and used.
    • You can have multiple versions of an image in the registry and use tags on those versions.
    • You can prepend the image name with a contributor path like: docker pull happycontributor/newimage

docker Commands

  • docker container prune - remove all stopped containers.
  • docker help - list the docker commands.
  • docker history - show the layers?
  • docker images - list the images, downloaded locally.
    • docker images -a
  • docker inspect <container-id | image-name> - Return low-level information on a container, image or task
  • docker logs - list the console output? for a given container id
  • docker ps --all
  • docker push - push an image to the docker registry.
  • docker pull - pull an image from the docker registry.
    • sudo docker pull busybox
    • sudo docker pull busybox:1.24 (Pulls a specific version).
    • sudo docker pull registry.example.com/myapp
      • Pulls from an alternative site. (will try http, https, ftp)
  • docker rmi [OPTIONS] IMAGE [IMAGE...]
    • If you get Error: Conflict, cannot delete image fd484f19954f because it is tagged in multiple repositories, use -f to force
    • Then you need to remove all repo references to that image; e.g. $ docker rmi test1:latest
  • docker run [options] IMAGE [COMMAND] [ARGS]
    • options:
      • --add-host
      • -p public:container (you can have multiple of these
      • --restart=always (Start at boot time)
      • --hostname=
      • -i, --interactive Keep STDIN open even if not attached
      • --cap-add=[]
      • --cpuset-cpus=
      • --device=[]
      • --link=[]
      • --memory=
      • --memory-swap=
      • --memory-swappiness=
      • --name=
      • -t, --tty Allocate a pseudo-TTY
      • --volume=[]
      • --workdir=
    • sudo docker run -i -t fedora:25 /bin/bash
      • Start an interactive session
  • docker search - will list docker images found
    • sudo docker search mysql

quick commands

  • get a shell inside a running container:
    • docker exec -it <CONTAINER_ID> /bin/sh
  • Override an entry point in a container
    • docker run -it --entrypoint /bin/bash [docker_image]
      • See:
  • provide a host directory inside a container
  • copy a file from a running container to the host
    • docker cp ID:PATH LOCAL_PATH
      • the ID can be either the container id or the container name.
    • docker cp 92636c5fc5c5:/etc/hostname .
  • copy a file from the host to a running container

Attach a container to a specific bridge

  • docker create --name lighttpd --network my-net --publish 80:8080 lighttpd:0.1.0

Build a docker image

  • docker build --tag lighttpd:0.1.0 .

Dockerfile

FROM alpine:3

EXPOSE 8080/tcp

RUN adduser -D -H www

RUN apk add lighttpd
RUN apk add perl

RUN mkdir -p /var/www/html/cgi-bin
COPY index.pl /var/www/html/cgi-bin
COPY index.html /var/www/html
RUN chown -R www:www /var/www

RUN mkdir -p /var/cache/lighttpd
RUN chown -R www:www /var/cache/lighttpd

RUN chown -R www:www /var/log/lighttpd


COPY lighttpd.conf /etc/lighttpd

# -D : don't go to background
CMD /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf

Build a rust compiler docker image

# Dockerfile for building rust applications
FROM alpine:3.20

RUN apk add rust cargo

WORKDIR /app

CMD [ "/bin/sh" ]

Build a docker image from rust source code

docker build --tag hello:0.1.0 .

Dockerfile

FROM rust_build:0.1.1 AS build

WORKDIR /app

COPY . .

RUN cargo build --release

# https://docs.docker.com/build/building/multi-stage/

FROM alpine:3.20

RUN apk update && \
    apk add libgcc

COPY --from=build /app/target/release/hello /bin/hello

# Create a new user and group
RUN addgroup -g 1000 myappgroup && \
    adduser -u 1000 -G myappgroup -s /bin/sh -D myappuser

# Switch to the non-root user
USER myappuser

CMD [ "/bin/hello" ]

troubleshoot docker build image

How to debug Docker build failures

  • DOCKER_BUILDKIT=0 docker build .

Dockerfile

  • ADD -
  • COPY -
  • FROM
    • FROM x AS stage
      • FROM
      • STAGE

Docker good manufacturing practices

  • Keep a Dockerfile in your source repository(Hec18, p58).
    • In the root of your source code.
  • Build your container with production in mind(Hec18, p59).
    • Do not use default values for unit testing etc(Hec18, p59).
  • Keep debug tools, diagnostic tools etc out of the production image
    • consider making a second Dockerfile in a subdirectory that adds to the first and only includes the debugging tools you want to add. If you do this, I recommend you add a name -debug to the name of the image to make it clear that the image has the additional tooling installed(Hec18, p58).
  • Output your logs to stdout/stderr(Hec18, p59).
    • This can be picked up by k8s and docker.
    • TODO is there a reason to use stderr?

Docker volumes

Making a host directory available in a container

  • docker run --detach --volume /home/myuser/subdir:/opengrok/src opengrok-dev

Docker networking

  • Docker secures the network by managing rules that block connectivity between different Docker networks.
  • Behind the scenes, the Docker Engine creates the necessary Linux bridges, internal interfaces, iptables rules, and host routes to make this connectivity possible.

Unpredictable interface order

IPtables and docker

  • All of Docker’s iptables rules are added to the DOCKER chain.

    • Do not manipulate this chain manually.

  • If you need to add rules which load before Docker’s rules, add them to the DOCKER-USER chain.

    • These rules are applied before any rules Docker creates automatically.

  • Rules added to the FORWARD chain -- either manually, or by another iptables-based firewall -- are evaluated after these chains.

    • This means that if you expose a port through Docker, this port gets exposed no matter what rules your firewall has configured.
    • If you want those rules to apply even when a port gets exposed through Docker, you must add these rules to the DOCKER-USER chain.

TODO:

  • What are chains
  • Syntax:
    • :FORWARD DROP [0:0] - What does it mean? should it be removed?
    • :DOCKER-ISOLATION-STAGE-1
    • -A FORWARD -j DOCKER-USER
    • -A FORWARD -o catapultb2 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
    • -A FORWARD -o catapultb2 -j DOCKER
    • -A FORWARD -i catapultb2 ! -o catapultb2 -j ACCEPT
    • -A FORWARD -i catapultb2 -o catapultb2 -j ACCEPT
    • -A DOCKER -i catapultb2 -j RETURN

IPtables commands

  • sudo iptables -L
  • sudo iptables-save

Get docker network interface used by container

docker inspect 7c17181abcd4 | grep NetworkID | awk -F"\"" '{ print $4}' | cut -c -12 | xargs docker network inspect

use the Gateway IP address to find the host interface to tcpdump

Update the /etc/hosts in the container

Update /etc/hosts at run time

  • --add-host list Add a custom host-to-IP mapping (host:ip)

e.g. : docker run -it --add-host gitlab:192.168.122.83 pytools:0.1.0 sh

Update /etc/hosts at build time

  • --add-host strings Add a custom host-to-IP mapping (format: "host:ip")

Docker cookbook

  • override entry points in images
    • docker run -it --entrypoint sh container

Docker host stuff

Limit log size

See:

e.g.: docker run --log-driver json-file --log-opt max-size=10m alpine echo hello world

run tcpdump in a container directly from the host

  1. docker inspect testcontainer | grep -i pid
  2. sudo nsenter -t PID -n tcpdump

Installing the docker-ce on Ubuntu

See:

  1. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
  2. echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list
  3. sudo apt-get update
  4. sudo apt-get install docker-ce docker-ce-cli containerd.io
  5. sudo usermod -a -G docker $USER
  6. exit from machine
  7. login again
  8. docker ps (should give you a single HEADER line)

Installing the docker-ce on Fedora 29

See:

  1. sudo dnf install -y yum-utils
  2. cd /etc/yum.repos.d/
  3. wget
  4. sudo dnf install -y docker-ce docker-ce-cli containerd.io
  5. sudo usermod -aG docker your-user
  6. vi /etc/docker/daemon.json add: { "group": "dockerroot" }
  7. sudo systemctl enable docker
  8. sudo systemctl start docker
  9. sudo systemctl status docker
  10. logout
  11. login again
  12. docker images

Installing the Docker host

  • sudo apt install docker.io
    • This seems to be outdated(in 22.04 is 20.xx when the docker-ce is 24)

Installing docker on Fedora 23

Docker storage

Move docker storage to /home

  • sudo systemctl stop docker
  • sudo mkdir /home/dockerdata
  • sudo rsync -aP /var/lib/docker/ /home/dockerdata
  • sudo mv /var/lib/docker /var/lib/docker.old
  • sudo vi /etc/docker/daemon.json
  • sudo systemctl start docker
  • it seems take many minutes before the dockerd actually stop trying to use /var/lib/docker
    • Maybe it was because it was restarted multiple times?
{ 
   "data-root": "/home/dockerdata" 
}

Installing docker on windows

  1. Enable hyper-v 1.* Turn windows featrues on or off
  2. Download docker for windows:

  • Unable to translate current working directory. Using C:\WINDOWS\system32

udo apt remove docker.io

  1. sudo debootstrap xenial xenial > /dev/null
  2. sudo tar -C xenial -c . | docker import - xenial
  3. docker run xenial cat /etc/lsb-release

Create a Raspberry Pi Base image

. sudo apt-get install debootstrap . cat /etc/os-release . sudo debootstrap stretch stretch . sudo tar -C stretch -c . | sudo docker import - stretch . sudo docker run stretch cat /etc/os-release

WatchList

Create RHEL base image

using a local docker registry

Localhost registry

The localhsot registry does not require a certificate. But you can only access it via 127.0.0.1, not on the LAN address(that requires a certificate).

  • sudo docker run -d -p 5000:5000 --restart=always --name registry registry:2
    • or: sudo docker run -d -p 5000:5000 --restart=always --name registry -v pwd/data:/var/lib/registry registry:2
  • sudo docker ps -a
  • sudo docker images
  • sudo docker tag hello-world localhost:5000/hello-world
  • sudo docker push localhost:5000/hello-world
  • sudo docker pull localhost:5000/hello-world

Running a LAN registry

Image manipulation

Copy an Image from one machine to another machine

  1. docker pull registry
  2. docker images | grep registry
  3. docker save --output registry.dkr 2ba7189700c8
  4. scp registry.dkr SomeUser@SomeHost:/tmp
  5. login to the destination machine
  6. docker load --input /tmp/registry.dkr
  7. docker images
  8. docker run -d --restart=always -p 5000:5000 2ba7189700c8 --name=registry

Image info

Get the IP address of a container

docker inspect <containerNameOrId> | grep '"IPAddress"' |

Container best practices

Allow a user to execute docker commands without sudo

Generate an image

Dockerfile entries:

  • FROM - must be the first entry

  • CMD - The main purpose of a CMD is to provide defaults for an executing container.

  • RUN -

  • mkdir rhel7bld

  • vi Dockerfile

    • FROM rhel7_base:latest
    • CMD cat /etc/redhat-release

8 best practices for building containerized applications

  1. Instances are disposable
  2. Retry instead of crashing
  3. Persistent data is special
  4. Use stdout not log files
  5. Secrets (and other configurations) are special too
  6. Don't assume co-location of services
  7. Plan for redundancy / high availability
  8. Implement readiness and liveness checks

Pods

  • Pod name: flask-1599974757-b68pw(Hec18,p41).
    • The first part of the name (flask) is from the deployment
    • the second part (1599974757) is the unique name assigned to the ReplicaSet that was created
    • the third part (b68pw) is the unique name assigned to the Pod that was created.

Methods for troubleshooting

  • journalctl -u docker

Debugging

debug app in container

  1. docker exec -it CONTAINER_ID sh
  2. cd /tmp
  3. dbg
  4. attach 1
  5. continue
  6. generate-core-file
  7. wait for the core dump
  8. thread apply all bt
  9. quit

get the interrupt reason:

  • dbg APP_WITH_PATH CORE_WITH_PATH

docker-compose

  • docker compose up

docker-compose installation

Installing docker-compose on ubuntu

See:

  1. sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  2. sudo chmod +x /usr/local/bin/docker-compose

docker-compose networking

private and public nic in docker-compose

See also:

version: "3.9"

services:
  alpha:
    image: alpine
    command: sleep 300
    networks:
      default:
      direct:
          ipv4_address: 10.1.2.1
  bravo:
    image: alpine
    command: sleep 300
    networks:
      default:
      direct:
          ipv4_address: 10.1.2.253

networks:
  direct:
    driver: bridge
    ipam:
      config:
        - subnet: 10.1.2.0/24
          gateway: 10.1.2.254

Docker security

Investigate image supply chain

For each tools in an image investigate(LFS260):

  • Open or closed source
    • If open source
      • does anyone offer support
      • look at the source repo to identify how active the project/community is
      • How many critical issues exists
    • If closed source
      • what is the pricing scheme
      • how many op critical issues exists

Tools:

  • Trow - Container Registry and Image Management for Kubernetes Clusters.
  • Prisma Cloud - Cloud-Native Application Protection Platform (CNAPP)
    • Prisma Cloud secures applications from code to cloud, enabling security and DevOps teams to effectively collaborate to accelerate secure cloud-native application development and deployment.
  • NeuVector - open source, zero-trust container security platform.
  • Clair - static analysis of vulnerabilities in application containers (currently including OCI and docker).
  • Aqua - See and stop threats across every phase of your software development lifecycle, from code to cloud and back.
  • Trivy - a comprehensive and versatile security scanner. Trivy has scanners that look for security issues, and targets where it can find those issues.
  • Notary

Installing and using the trivy security scanner

kernel vulnerabilities

  • git clone https://github.com/jondonas/linux-exploit-suggester-2.git
  • cd linux-exploit-suggester-2
  • ./linux-exploit-suggester-2.pl

Mitigating kernel vulnerabilities

(LFS260)

  • Block dynamic modules
  • kernel.modules disabled
    • kernel.modules_disabled 1
  • Adaptive Stack Layout Randomization
    • kernel.randomize_va_space sysctl 2
  • Hardware Security Features
    • NX(No eXecute)
      • Look for the nx flag in /proc/cpuinfo. It may be enabled/disabled in BIOS.
    • VT-d
      • hardware feature to be enabled in BIOS - look for vmx or svm in /proc/cpuinfo.
    • TPM
      • TPM is a hardware feature that provides a way to store checksums
    • TXT(Trusted Execution Technology)
      • used to isolate the memory used by guest virtual machines.
      • A guest VM could have a malicious kernel that tried to access memory owned by a host or another guest. TXT is an efficient, hardware-based method for preventing this from happening.
    • SMAF(Secure Memory Allocation Feature)
    • hw cryptographics devices
  • ExecShield
    • The kernel caches the user segment descriptor value, so the overhead in the context-switch path is a very cheap, unconditional 6-byte write to the GDT, costing 2-3 cycles at most.
    • the kernel also remaps all PROT_EXEC mappings to the so-called ASCII-armor area, which on x86 is the addresses 0-16MB.
    • These addresses are special because they cannot be jumped to via ASCII-based overflows.
    • echo 0 > /proc/sys/kernel/exec-shield
    • In many recent kernels the ExecShield feature was replaced with NX (not executable) hardware support.
  • Integrity Management
    • The kernel’s integrity management subsystem may be used to maintain the integrity of files on the system.
    • The Integrity Measurement Architecture (IMA) component performs runtime integrity measurements of files using cryptographic hashes, comparing them with a list of valid hashes.
    • Look for CONFIG_IMA in /boot/config*
    • Boot with ima_tcb and ima=on kernel parameters.
  • Integrity Management with dm-verity
    • A simpler approach to integrity management is the dm-verity module. This is a device mapper target which manages file integrity at the block level.
    • Check for CONFIG_DM_VERITY under /boot/config*.
  • Linux Security Modules (LSM)
    • implements hooks at all security-critical points within the kernel.
    • A user of the framework (an LSM) can register with the API and receive callbacks from these hooks.
    • All security-relevant information is safely passed to the LSM, avoiding race conditions, and the LSM may deny the operation.
    • This is similar to the Netfilter hook-based API, although applied to the general kernel.
  • Secure computing mode (seccomp)
    • a mechanism which restricts access to system calls by processes.
    • The idea is to reduce the attack surface of the kernel by preventing applications from entering system calls they do not need.

Image generation

Troubleshooting

troubleshooting daemon

Error response from daemon: Cannot start container xxx: [8] System error: exec format error

Error response from daemon: Cannot start container d81646156ff60a44599a7fdb750821e51bddeb9b4ae0a53ce145092828ee1f4e: [8] System error: exec format error

Error user can't run docker without sudo

See:

  1. sudo groupadd docker
  2. sudo usermod -aG docker jenkins
  3. systemctl restart docker

lstat filebeat-6.0.0-x86_64.rpm: no such file or directory

Do check the .dockerignore file too.

WARNING: Error loading config file: .dockercfg: $HOME is not defined

See:

Troubleshooting docker-compose

  • docker ps --filter "status=exited"

  • docker inspect 1956e6b3f2c3 | grep LogPath

  • sudo less /var/lib/docker/containers/1956e6b3f2c3a3ceb75a162b20a57df2a0ee51726b14f3af7e8abce15a70bd68/1956e6b3f2c3a3ceb75a162b20a57df2a0ee51726b14f3af7e8abce15a70bd68-json.log

docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

Docker was not installed

Troubleshooting docker build

user homedir ownership change to root after command is done

  • sudo apt remove docker.io
  • sudo apt autoremove
  • install docker.ce

Update to docker-ce 24

Dockerfile entries

RUN chown -R steam:steam ${HOMEDIR} && ls -l /data
RUN ls -l /data

output:

Step 8/13 : RUN useradd --home-dir "${HOMEDIR}" --create-home -u "${PUID}" --shell /usr/bin/bash ${USER} && ls -l /data
 ---> Running in 68d5f062a6a0
total 4
drwxr-x--- 2 steam steam 4096 Aug 19 09:18 steam
Removing intermediate container 68d5f062a6a0
 ---> b67a5698aea0
Step 9/13 : RUN ls -l /data
 ---> Running in 13f9b0545416
total 4
drwxr-x--- 2 root root 4096 Aug 19 09:18 steam