docker - RicoJia/notes GitHub Wiki

========================================================================

Motivation:

========================================================================

so many operating systems to update.
- docker is a process that runs separately on your system, it has its own file system, network. If you don't set a port to connect to it, you won;t get it.
Container has been around since 2010, but docker is the most popular one
- Kubernetes is a ochestrator of docker, i.e, managing multiple containers at a time
Technology: emulates CPU usage, RAM of an operating system, by taking slices of the existing operating system. Then, you install software in designated places.
- The area of the docker is called "sandbox".
Docker vs Virtualization:
1. Docker is a parent process that spans child processes, with cache loaders, cache manager, etc. the kernel space is the same for Linux containers. So the docker will just contain the required User space files to a specific distro. Also, you can run Linux dockers on windows, but currently not windows dockers on Linux.
2. virtualization fakes the entire operating system, which needs the a lot of disk space and processing power.
Versions: Docker EE: (enterprise) and CE (Community Edition)
- EE: verify if an image is legit
Update Channel: Stable (gives you the latest releases), Test(Pre-releases for testing), Nightly(latest builds of work)
Docker Enterprise: not related to Docker at all. It's a container platform that scans container vulnerabilities

========================================================================

Concepts

========================================================================

Registry
1. registry is the docker hub, repo is a collection of images.
2. each image is stored as a tag
3. docker pull docker.io/ubuntu:bionic
  - docker.io is the DNS name of the registry
  - full name: docker.io/library/ubuntu:bionic, where library is the name of the repo ("namespace")
  - Official images don't show repo name, but for others, you must include it
  - Because it's not latest, you must delete it with tag.
4. docker pull registry.hub.docker.com/library/ubuntu:bionic
  - Has same sha, so it was built on the same image. But registry is different.
  - BUT with a different registry name, the image is still a different image!!
5. Pulling non-official latest image: docker pull mysql/mysql-server
  - namespace/repo, this is a common pattern for image naming
6. Pulling images from the internet may pose a security threat. Set up your own repository
Image: bundle up all the dependencies.
- They're just copies of each other. You want to build images. You can give it a "name" as well. You can build an image using docker build
Container: an instance of an image? you can execute that container, but not that image.
Ephemeral: means short-lived things, like an application, server

Some Interesting facts

Images are stored in "Docker hub"
- You should have your private image registry.
- Base_image is a tiny but functional OS
- An image is built on a layered model
  - A layered system means: you can add to the base system. Every time you add something to the system, you're adding a layer.
  - a difference is also called "delta"
- Dockerfile is required
"Container escape" is a successful attack on a container. Since containers share the host operating system, that could be a security loophole
Isolation Techniques
- cgroups: operating system to make sure resources are not consumed by specific groups only
- namespace: on kernel level, docker uses namespace for access.

Image Management

docker search --limit 5 httpd, show top 5 images with this name for downloading
docker images --no-trunc shows sha256, the internal name for the image.
- You will see the old tag and the new tag. Feel free to delete the old one
See Processes: common: docker ps: (ps from processes) to see what's running on your docker.
- docker ps -l -l means "the latest launched"
- docker ps -a gives you what has exited too.
Delete container, images
- docker container rm CONTAINER_NAME
- docker image rm IMAGE_NAME #this has to come after removing the container
- docker container prune: remove all the containers
- docker rmi httpd:2-alpine, you need the tag cuz you don't have the latest.

========================================================================

Building an Image:

========================================================================

layer model: saves space cuz it will keep track of changes we've made.
- Each layer is a "delta", or changes, compared to the layer underneath it.
- So these deltas allow us to use caching.
Build a docker image
1. docker pull nginx:
  - docker run will check if we need to pull, but that will run a container first.
2. docker history nginx: shows the commands to create such an image line by line.
  - missing means somthing not available locally_
3. check tag: latest is labelled by human, so it may not be latest.
```
#tag is optional, cuz it's latest by default
docker tag SOURCE_IMAGE[:tag] TARGET_IMAGE[:tag]
# example, creating a tag, which will NOT erase the original one.
docker tag nginx:latest nginx:my_blog_stable
```
  - hub.docker.com
  - a docker image on hub.docker.com links to a dockerfile on github, very common pattern.
4. Find the docker file on dockerhub, then start building your own:
```
ENV my_env 1.1.7    #like linux environment variables
COPY my_config.yam /var/www/config    # Copying a local file onto a docker image.
EXPOSE 80     #exposing port 80 for TCP communication
CMD["nginx", "-g", "daemon off;"]    #run the binary, using JSON array (java script object notation). Daemon off is to run the binary in foreground. Else, it will terminate as soon as it starts?? Most applications require slight changes like this when containerized.
Run apt-get update \
  && apt-get install -y nginx \   #&& will make all changes here in one layer. The fewer the layer, the faster.
  && apt-get clean    # this will remove the source code of the installed packages.
```
5. build the image
```
docker build -t my_nginx_tag .    #-t allows tag, . means to use the Dockerfile in the current directory
```
  - You may see other images as they are required to build the current image.
  - Where do docker containters exist? In /var/lib/docker. IMPORTANT TO MAKE A BACKUP here!
6. Run the image
```
docker run -dit --name=my_nginx_tag
```
7. --help
  - Find help, their help is very well-written!!: docker run --help
8. docker image prune to prune dangling images, which are not associated with any tagged images.
  - docker system prune : remove dangling images, built cache, stopped containers, networks not used by at least one container.
9. check disk space used by docker docker system df
10. Another option: docker build - < Dockerfile, - is used to take a docker file from stdin
11. make sure you have all dependencies in apt. Otherwise apt install may not install it.
12. always do
```
RUN apt update \
&& apt install ros-noetic-joint-state-publisher-gui ; exit 0
```
force remove docker image: docker rmi -f ...
Download git repo and install it: see code
- Experiment to get all the installation steps right
- start building from small images. ========================================================================

Run A Container

========================================================================

Basic Management workflow
- docker run -dit --name tag_name IMAGE_NAME
  - will keep running even tho the terminal is closed.
  - by default, docker starts a container on the foreground. -d is to detach and make it run in the background
- docker stop tag_name
- docker ps -a: see all containers.
- docker rm tag_name: remove a container
- docker start Stopped_Container_Name
- docker inspect my_container see restart, and other info about a container.
- docker kill my_container like docker stop, both terminates process, and both needs docker rm
  1. docker kill sends SIGKILL directly.
  2. docker stop sends SIGTERM, then after a grace period, SIGKILL
  3. SIGTERM is graceful, as it can be interpreted and ignored, which can be used to save state. SIGKILL will just kill the process.
  4. SIGTERM almost the same as SIGINT
- docker run -dit IMAGE_NAME --restart=always: regardless of connection issues, unless there's docker stop/kill, it always tries to restart.
  - the host may get rebooted
- docker run UNKNOWN_IMAGE will pull the image
- docker run --rm IMAGE_NAME automatically deletes container after stop
- docker logs container shows all output from a container
  - docker logs -t tag_name shows each output with a time stamp
  - docker logs -f tag_name, f means follow, that's to print outputs real time. You can see this on a running container!!
docker run specifics
- get into a running container
```
docker run -itd IMAGE_NAME --name CONTAINER_NAME
#and in every new terminal session:
docker exec -ti CONTAINER_NAME bash 
```
- Run to mount a home directory docker run -itd --name CONTAINER_NAME -v $homedir:/home/$USER IMAGE_NAME
- You can append other options to this too docker run -p Your_value:docker_env_value map Your_value to env variable in docker
- a container will automatically run some processes, and comes with a file system
docker container prune: remove all the containers

Running a webserver in a container

docker run -d --name=my_nginx -p 8080:80 nginx, -p redirects everything from 0.0.0.0:8080 to port 80 of the container.
docker ps and you will see 0.0.0.0:8080->80/tcp under ports.
curl http://localhost:8080 and you will see the webpage spat out
If you want to use web browser, get the host machine ip address, then put IP:8080
1. You can use 127.0.0.1 (local host) but perfer 192.168... (local network, so you can access it on another computer)
docker logs -tp my_nginx you will see the connection details of the webserver.
docker run -d --name=another_nginx -p 8080:80 -v ${PWD}/webpages:/usr/share/nginx/html:ro nginx
1. -v creates a volume for file sharing bw the host and container.
2. the first arg after -v is the path on the host
3. ro is an option for read-only, i.e, files can be changed by host, not by container.
4. then inside ${PWD}/webpages, you supply an index.html. The content will be pushed to the web-browser.

Execute a command in a container

launch a container real time:
- docker run -it --name=my_apache httpd /bin/bash:
  - no -d cuz it will be detached;
  - /bin/bash means to run the bash binary, so you can use bash. Some ppl will just put bash,
  - Some images based on a minimal Linux Distro - Alpine Linux. That doesn't have /bin/bash, it has /bin/sh, older style, no auto completion, etc.
  - so you can do sh as well
docker exec -it SHA bash: this will enter a running container.
- without -it docker will immediately stop
- If the container is already running with -d, exiting docker would NOT terminate the container!
docker exec -d SHA touch hello_world will create hello_world, so exec can execute any command - but an image is not the full distribution, so some commands are missing, such as ps, vim...
docker run -it --name my_redis redis bash enters the container, while without the bash it just launches the processes. Ctrl-D is to quit the container.
docker top SHA: see some processes running in the container.

Docker logs can be used to achieve password spat out by detached container

docker run -dit -p 8080:8080 -p 50000:50000 --name=my_jenkins jenkins/jenkins:lts
- Docker_User_namespace/Image_Name ?
- lts is "long term support"
jenkins needs a password for what installation?
- see the password: use docker logs -tf my_jenkins
- to download page: localhost:8080
you can see logs on /var/log/syslog on your host system too
or to see the log generated by journald, by using command journalctl as well. journalctl collects data from systemd, which can get stuff from rsyslog. rsyslog owns /var/log/*
1. pipes output to less, a pager system. you need arrows to go horizontally.
2. journalctl --no_pager will squish everything on one page.
3. journalctl to see all logs from systemd, journalctl -n 25 shows the last 25 lines.
  - journalctl -u docker.service shows logs only containing "docker.service". It comes with a delay though!

========================================================================

Docker Volume

========================================================================

docker volumes: used to save data in host filesystems from a running container, so data will persisit

-v is --mount, but --mount is easier to use.

Create volume

docker volume create test_volume
docker volume list
docker volume rm test_volume

inspect volume

if it's a permanent volume

docker inspect test_volume    # see "Mountpoint": "/var/lib/..." that's its position on the host machine

To see tmp volume:

docker inspect CONTAINER_NAME | grep Mounts -A 10   #grep "Mounts", show the following 10 lines

run a container with volume

docker run -d --name my_container --mount source=test_volume,destination=/root/volume ros_image   #VERY IMPORTANT: no space before/after = and ,
docker run -d --name my_container --mount src=test_volume,dst=/root/volume

start volume with "ephemeral" container:

docker run -d --name my_container --mount type=tmpfs,dst=/root/volume ros_image
docker run -d --name my_container --mount type=tmpfs,tmptf-size=256M,dst=/root/volume ros_image  #restrict volume size to 256M

Short way, use -v and , becomes :

docker run -d --name my_container -v /root/webpages:/usr/share/nginx/html:ro -p 8080:80 nginx   #src:dst. Whatever that dst is, container will create it for you :)

this is not being managed by docker, so less flexible than --mount

Interacting with volume: add to .../_data/

#use docker inspect volume_name to see mount point 
[
    {
        "CreatedAt": "2021-08-22T22:10:48-05:00",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/rico_v1/_data",
        "Name": "rico_v1",
        "Options": {},
        "Scope": "local"
    }
]

echo 'Hello world' > /var/lib/docker/volumes/test_volume/_data/index.html

You can mount the same volume multiple containers

Making container have read-only access to the volume:

docker run -d --name read_only_container --mount src=test_volume,dst=/root/volume,ro ros_image
# ro can be replaced by readonly

Good security practice if the container gets hacked in!

Deleting a volume after use: after a container is destroyed, the volume is STILL INTACT! This is a feature. So delete the volume separately
- docker volume prune, deletes unused volumes

========================================================================

More advanced operations

========================================================================

Run a command in this running container. docker exec -ti my_container sh -c "echo a && echo b"
login to a container as root: docker exec -it --user root omnid_dockint /bin/bash
How to decrease build time? 1. Separate docker files 2. time DOCKER_BUILDKIT=1 docker build -t node-14-second-bad-cache-with-buildkit . 3. https://geshan.com.np/blog/2020/10/docker-build-example-faster-docker-build/

Docker commit: save a container into a new image.

docker commit CONTAINER_ID  svendowideit/testimage:version3
docker images

========================================================================

Docker File

========================================================================

How Dockerfile works: docker engine reads Dockerfile to build the image

FROM ros:noetic-ros-core-focal     #start building on top of a verified image on dockerhub, 12.6.3 is the tag of the image. 
WORKDIR   /code   #make a new directory for all subsequent commands
ENV ENV_NAME val  # make an environment variable for all your processes. 
COPY FILE1.json /code/FILE2.json    #copy in application dependencies (.json) to the image
RUN npm install   #run this command, which is a package manager that install all dependencies .json has
COPY EVERYTHING inside the current directory to the code directory 
CMD ["node", ".json"]    #default command for the docker to run  upon booting up. Run node upon booting up

there's also a docker_ignore file to ignore what not to go into docker
There's also docker-compose file, which can create multiple containers at once (whatever you put on the docker-run).

each command creates a new layer. Another Example
1. FROM: sets the base image
2. CMD: a command and more
3. RUN: execute a command
4. EXPOSE: expose a network port
5. VOLUME: sets a disk share
6. COPY: creates files from disk into image
7. LABEL: maintainer of the image. old way is "MAINTAINER"
8. ENV: environment variable to a container. Dream Mobile Platform Example. It can persist across images.
9. ENTRYPOINT: decides what executable to run when container starts.
  - should be at the end of the docker files
  - CMD will follow it to specify the args to the executable
  - CMD arg can be overriden by command linearg : docker run ricojia/test arg.
  - If arg is missing in CMD in dockerfile, we can even use command line to add args to it
  - the ENTRYPOINT command itself is overridable.
```
#Dockerfile
ENTRYPOINT["bin/bash"]
# execute curl instead of "bin bash", then have google.com as arg.
docker run --entrypoint /usr/bin/curl -it ricojia/test google.com
```
    - or If you want to execute a command that wish to be overriden, change it to CMD
      #instead of this: ENTRYPOINT ["bin/bash"] #try this: CMD["bin/bash"] #then on commandline we can execute curl instead of ```bin/bash```, then google.com is the arg. docker run -it ricojia/test curl google.com
10. RUN: && will execute one by one, if one fails, the subsequent ones won't get executed.
build by docker build -t ricojia/ping_repo:latest .
- if no tag, then latest is assumed
- . means to use the Dockerfile in current directory. Else, supply the dir path
- will download the base image separately.
- push to docker hub:
```
docker login 
docker push ricojia/test:latest
```
  - Image name must be Docker_ID/repo:latest for successful pushes
- add missing libs to dockerfile and rebuild
- docker build -t ricojia/ping_repo:allow_ocerride -f Dockerfile_test ., where we're using Dockerfile_test, and we need -f
How to run a container as the same user?
1. uid: without uid, container will run as root. uid and gid of a process is examined by the kernel for privileges when it asks to modify a file
  - A container shares the same kernel as the host, which is an advantage of Docker.
  - so the same UID, GID space is shared.
  - so to run container as the current user, you can: see here
2. Methods2
  1. use export UID=$(id -u); export GID=$(id -g); docker run --user ${UID}:${GID} my_image
    - There's no easy way to share user name with the container, so you will see I have no name!
    - you may see bash: UID: readonly variable, then ignore it
  2. Build another image from the supplied image, with UNAME, UID, GID, etc.
```
# new image
'ARG BASE=ubuntu:focal
FROM $BASE

# Make Xwindows work with the native x server and qt
ENV DISPLAY=:0
ENV QT_X11_NO_MITSHM=1

# Create env variables for user information with placeholder values. The real args are passed at build time, and set up the docker container to have the same users as the host system
ARG UNAME=robot
ARG UID=1000
ARG GID=1000

# create a normal user called "$USER"
RUN groupadd -f -g $GID $UNAME
RUN id -u $UNAME || useradd -m -u $UID -g $GID -s /bin/bash $UNAME

# change to the desired user
USER $UNAME
WORKDIR /home/$UNAME'
```
    - then plug in the variable names docker build --build-arg UID=$(id -u) --build-arg GID=$(id -g) --build-arg UNAME=$USER --build-arg BASE=$base_image_tag --tag "$new_image_tag" - <Dockerfile

========================================================================

Docker interactive

======================================================================== The goal is to do ROS development on non-Ubuntu machines with minimal differences, so we can make running programs in the docker container be as close to the host user experience as possible: this means operating with the same user and group ids as the host user, seeing all devices on the host machine, and being accessible via the host’s network settings.

========================================================================

Reminders

========================================================================

docker exec -it container_name bash is to enter a running container
- docker exec container COMMAND: COMMAND should be an executable, quoted or chained command WON'T work
- docker exec -t container_name df -h see disk space of container, useful for seeing the usage of volumes
docker entrypoint, exit immediately even with -it - need /bin/bash CMD /bin/bash -c "source /opt/ros/noetic/setup.bash && /bin/bash"
docker exec - start a new bash session, which is NOT a child session of the current container.
If there's an error in the middle, then the rest of the docker file will fail. CMD will be omitted. You can choose to omit errors:
```
RUN make; exit 0
#or
RUN make \
; exit 0
```

Docker Registry

docker private registry: it's on a local machine, and can be pushed to and pulled from by other machines.

docker - RicoJia/notes GitHub Wiki

Motivation:

Concepts

Some Interesting facts

Image Management

Building an Image:

Run A Container

Running a webserver in a container

Execute a command in a container

Docker logs can be used to achieve password spat out by detached container

Docker Volume

More advanced operations

Docker File

Docker interactive

Reminders

Docker Registry

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️