pkgcache - ewsdocker/ewsdocker.github.io GitHub Wiki

Advanced Docker Concepts - pkgcache

Goals

The goals of this exercise are to become familiar with the Docker network and volume commands, as well as to realize the versatility of user-defined network bridges.

Refer to the following Docker documents:

And to the following Docker Hub image descriptions:

Additional reading:

Brief overview

Sometimes externally available files (e.g. tar, tgz, zip) are required to complete a Docker build process. The task is to create a method, using only existing Docker containers and Docker resources, to add files to a file-cache and retrieve files from the file-cache by the Docker build process.

The following components will be used:

  • pkgcache
    A Docker volume for storing (caching) and retrieving the files. This is the file-cache.

  • pkgnet
    A Docker network bridge for communications between the various components.

  • pkgnginx
    This is an un-modified NGINX server image in a container providing download (read-only) access to the pkgcache files. No index is produced, and packages (files) are accessed by their full package name using http.

  • pkgcli
    The nimmis/alpine-micro image is used as a pkgcache client, providing read/write access to the pkgcache and Linux commands to download from external (LAN/WAN) sources to the pkgcache.

This exercise consists of the following steps:

  1. Create the pkgnet network bridge;
  2. Create the pkgcache Docker volume;
  3. Download the ewsdocker/alpine-nginx and nimmis/alpine-micro:edge Docker images;
  4. Create the pkgnginx server container;
  5. Create the pkgcli client container from nimmis/alpine-micro:edge;
  6. Run tests;
  7. A real-life example.

Creating the pkgcache docker environment

Create a user-defined network bridge.

Enter the following command on the docker host console:

docker network create -d bridge --subnet=172.32.0.0/16 pkgnet

This will create a new user-defined bridge network on subnet 172.32.0.0 named pkgnet. Choose an unused network ip address for the subnet, or the host won’t allow the network to be created (no overlaps of IP addressing with other subnets).

If access to other networks from this subnet is to be isoloated, use the internal flag:

docker network create -d bridge --internal --subnet=172.32.0.0/16 pkgnet

NOTE: The --internal flag must NOT be used if access to the external LAN / WAN is required for the build (e.g. - external repositories). This flag will isolate the build process to the pkgnet network ONLY, so all required resources must be available on the pkgnet network.

The new network settings can be viewed by

docker network inspect pkgnet

Create the pkgcache volume

Create the pkgcache volume as follows:

docker volume create pkgcache

The volumes on the docker host can be listed with

docker volume ls

and viewed with

docker volume inspect pkgcache

Simple, so far.

Now it gets easier.

Get the docker images

Pull down the ewsdocker/alpine-nginx image from Docker Hub and re-tag it with a shorter name:

docker pull ewsdocker/alpine-nginx:edge
docker tag ewsdocker/alpine-nginx:edge alpine-nginx:edge
docker rmi ewsdocker/alpine-nginx:edge

Also, pull down the nimmis/alpine-micro:edge image from docker and re-tag it with a shorter name:

docker pull nimmis/alpine-micro:edge
docker tag nimmis/alpine-micro:edge alpine-micro:edge
docker rmi nimmis/alpine-micro:edge

Create containers from the images

Now that the images are loaded, they can be turned into the required containers.

Create a pkgcache client

Now, create an alpine-micro client container with access to pkgcache and the internet via wget or curl:

docker run -it \
           --rm \
           --mount source=pkgcache,target=/pkgcache \
           --network=pkgnet \
           --name=pkgcli \
       alpine-micro:edge /bin/ash

When the # prompt appears, you are connected to pkgcache (the name given to alpine-micro) as root and should be able to access the pkgcache volume:

ls /pkgcache

Since this is the first access to the pkgcache, the cache should be empty.

Create pkgnginx

create an alpine-micro client container with access to pkgcache and the pkgnet network.

docker run -d \
           --restart unless-stopped \
           --mount source=pkgcache,target=/usr/share/nginx/html \
           --network=pkgnet \
           --name=pkgnginx \
       alpine-nginx:edge

Checking the docker run schedule should show a running alpine-nginx named pkgnginx

docker ps -a

The newly created pkgcache docker volume is mounted on the html directory in pkgnginx, at /usr/share/nginx/html, allowing information stored in /pkgcache to be served instead of whatever is in the default html directory.

Test the connection

Copy a file to the pkgcache.

cp /boot.sh /pkgcache

A listing will show the file in the pkgcache volume.

Using alpine-micro for a client, we’ll attempt to download boot.sh to the root (current) directory from pkgnginx:

wget http://pkgnginx:8100/boot.sh

The requested boot.sh file should now be in the current (/root) directory.

An example use.

We can use the basic cache system that was just created to provide a cached-file "plug-in" to a Dockerfile to make it possible to pull a file into the running Docker build process by utilizing an http server and Docker bind-mount volumes. This is more elegant than downloading the file and adding it to the Docker container source before building it, and it is more maintainable than a manual solution.

The NGINX server in an Alpine Linux container has a very small footprint, less than 18 MB, loads quickly, and is perfect for embedded services such as the pkgcache "plug-in" cache server. It can be downloaded, built and run very quickly and automatically removed by terminating the pkgnginx server. And, it can all be done from a simple script.

A sample plug-in

We will develop a method of deploying pkgcache in a real project.

During the Docker build of the ewsdocker/debian-libreoffice, it is necessary to download a rather large tarball from the Libre Office download site. One of the requirements of the debian-libreoffice project is to download the file directly from the Libre Office servers during the Docker build, so this file must be re-loaded for each test of the build.

During the container development, In order to reduce the overall network load on both the Libre Office servers and the local Internet link, the file will be downloaded once to the local Docker host pkgcache, then built by transferring the cached-file to the Docker build process.

Parsing the URL

The source for LibreOffice 6.1.0 for Debian 9 is available from the Libre Office download site:

http://mirror.switch.ch/ftp/mirror/tdf/libreoffice/stable/6.1.0/deb/x86_64/LibreOffice_6.1.0_Linux_x86-64_deb.tar.gz

We can break the URL up into 2 fields:

We also need the name of the installation directory in the tar file.

  • ODIR
    This is the installation directory, LibreOffice_6.1.0.3_Linux_x86-64_deb. It can be determined at the LibreOffice download site, or from inspecting the tar file (tar -t): it occurs between libreoffice and DEBS in the installation directory path.

In the released Docker image source, the "plug-in" is disabled, and the needed resource will download directly from the original source site during the build, using the settings determined above.

In the development Dockerfile, the "plug-in" is enabled, and the needed resource will download from the pkgcache. In this case, the url is http://pkgnginx/LibreOffice_6.1.0_Linux_x86-64_deb.tar.gz, and the breakdown of the URL is the same as above, except for the OHOST setting:

  • OPKG
    LibreOffice_6.1.0_Linux_x86-64_deb.tar.gz

  • OHOST
    http://pkgnginx

  • ODIR
    LibreOffice_6.1.0.3_Linux_x86-64_deb

Sample Segment

This is a segment of the released Docker image source script for ewsdocker/debian-libreoffice:

ENV OHOST=http://mirror.switch.ch/ftp/mirror/tdf/libreoffice/stable/6.1.0/deb/x86_64
#ENV OHOST=http://pkgnginx

ENV OPKG=LibreOffice_6.1.0_Linux_x86-64_deb.tar.gz
ENV ODIR=LibreOffice_6.1.0_Linux_x86-64_deb

ENV OURL=${OHOST}/${OPKG}

RUN mkdir -p /usr/local/share/libreoffice \
 && wget -qO- ${OURL} | tar xvz -C /usr/local/share/libreoffice \
 && dpkg -i /usr/local/share/libreoffice/${ODIR}/DEBS/*.deb \
 && rm -R /usr/local/share/libreoffice/${ODIR}

Change the comment character as shown below for the development Dockerfile script:

#ENV OHOST=http://mirror.switch.ch/ftp/mirror/tdf/libreoffice/stable/6.1.0/deb/x86_64
ENV OHOST=http://pkgnginx

Steps to perform