Docker Swarm on Photon OS Study Case - dcasota/photonos-scripts GitHub Wiki

Recently, I came across an issue on the Photon OS GitHub repository, see here. The issue is intriguing because it involves Docker Swarm, yet the stack experiences crashes. I resolved to analyze the problem. In this blog post, I provide a brief description of the x86_64 lab setup using Microsoft Windows 11 and VMware by Broadcom Workstation 17.5. The focus is on running a Docker Swarm, with particular attention to potential issues related to the 'swarmpit' application.

x86_64 lab setup with Microsoft Windows + VMware By Broadcom Workstation

Microsoft Windows 11, VMware By Broadcom Workstation 17.5.1

Import photon-hw15-5.0-dde71ec57.x86_64.ova. Remove floppy, remove CD/DVD. Configure bridged network. For the use case swarmpit, use 4gb ram. Altered vhw upgrade with compatibility 17.5.x. Login, change password.

Optional: tdnf update -y. Update to latest.

DHCP

DHCP for swarm nodes is commonly a bad choice. Use static ip adresses. Nevertheless, if dhcp, have a look that the DHCP lease time is not too short.

Optimize the dhcp configuration, e.g.

cat > /etc/systemd/network/99-dhcp-en.network << "EOF99dhcp"
[Match]
Name=eth0

[Network]
DHCP=yes
IPv6AcceptRA=no

[DHCP]
UseDNS=true
UseRoutes=true
EOF99dhcp
chmod 644 /etc/systemd/network/99-dhcp-en.network

# see https://github.com/vmware/photon/issues/1321#issuecomment-1112158237
# Check journalctl as soon as the docker swarm is up and running about ChecksumOffload eligibility.
# Actually, do not use TransmitChecksumOffload=false for a non-physical ethernet adapter e.g. Microsoft hyperv vethernet adapter. 
# cp /usr/lib/systemd/network/99-default.link /etc/systemd/network/99-default.link
# echo "TransmitChecksumOffload=false" >> /etc/systemd/network/99-default.link

systemctl stop systemd-networkd
systemctl stop systemd-networkd.socket
systemctl start systemd-networkd
systemctl restart systemd-resolved

Install and configure docker. Initialize swarm.

# see https://github.com/vmware/photon/issues/1321
iptables -A INPUT -p tcp --dport 2377 -j ACCEPT
iptables -A INPUT -p tcp --dport 7946 -j ACCEPT
iptables -A INPUT -p udp --dport 7946 -j ACCEPT
iptables -A INPUT -p udp --dport 4789 -j ACCEPT
iptables -A INPUT -p esp -j ACCEPT
iptables -A OUTPUT -p esp -j ACCEPT

# Allow icmp (ping)
iptables -A INPUT -p icmp -j ACCEPT

iptables-save > /etc/systemd/scripts/ip4save

tdnf install -y docker apparmor-parser
systemctl enable docker

systemctl start docker
docker swarm init

Check the docker setup more granularly.

cd $HOME
tdnf install -y curl
curl https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh > check-config.sh
chmod a+x ./check-config.sh
./check-config.sh

All entries should be enabled, and the following entries are declared as missing.

- CONFIG_MEMCG_SWAP: missing
    (cgroup swap accounting is currently enabled)
[...]
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

ZFS is missing and acceptable so far.

Add nodes to the docker swarm.

Check the output of journalctl. Investigate in issues.

The following use case assumes a fully functional dhcp/ntp/dns/docker-swarm setup.

Study Case Swarmpit

In this study case, a tool named swarmpit is used. It is available on https://swarmpit.io.

  • Initial Issue
    Running Swarmpit as specified with docker run -it --rm --name swarmpit-installer --volume /var/run/docker.sock:/var/run/docker.sock swarmpit/install:1.9 does not work out-of-the-box on Photon OS. It fails with out-of-memory messages.

    Using the manual installation with git clone https://github.com/swarmpit/swarmpit -b master, docker stack deploy -c swarmpit/docker-compose.yml swarmpit and journalctl -f exposes the following:

    Mar 15 13:25:28 photon-machine kernel: __vm_enough_memory: pid: 2346, comm: java, no enough memory for the allocation
Mar 15 13:25:28 photon-machine kernel: __vm_enough_memory: pid: 2346, comm: java, no enough memory for the allocation
Mar 15 13:25:28 photon-machine kernel: __vm_enough_memory: pid: 2346, comm: java, no enough memory for the allocation
Mar 15 13:25:28 photon-machine kernel: __vm_enough_memory: pid: 2346, comm: java, no enough memory for the allocation
Mar 15 13:25:28 photon-machine audit[2298]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=unconfined pid=2298 comm="java" 
exe="/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java" sig=11 res=1

    There are many issue constellations:

    1. basic networking dhcp/ntp/dns
    2. docker swarm overlay network
    3. inter-service connectivity inside docker swarm service network
    4. docker compose service version
    5. cascade sequence of starting service subcomponents
    6. relaxed no file handling
    7. subcomponents release couchdb, influxdb, swarmpit app and swarmpit agent

    All these 7 issue constellations and how to prevent are discussed later.

Let's provision carefully a modified swarmpit service on Photon OS.

Setup

Create the docker swarm network and specify it e.g. "swarmpit".

docker network create --scope=swarm --attachable=true --driver=overlay swarmpit

Check inter-service connectivity. (todo: check to add for inter-service connectivity).

  • Optional: To use the docker-compose.yml from below, change the content for each service element with:

    networks: 
  - public

    and replace the service element networks at the bottom with this content.

    networks:
  public:
    name: swarmpit_net
    external: true

Clone the swarmpit github repository.

cd $HOME
tdnf install -y git
git clone https://github.com/swarmpit/swarmpit -b master

Replace docker-compose.yml and deploy.

cat > swarmpit/docker-compose.yml << "EOFdockercompose"
version: '3.9'

services:
  app:
    image: swarmpit/swarmpit:1.9
    depends_on:
      - db
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    environment:
      - SWARMPIT_DB=http://db:5984
      - SWARMPIT_INFLUXDB=http://influxdb:8086
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - 888:8080
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080"]
      interval: 60s
      timeout: 10s
      retries: 3
    networks:
      - net
    deploy:
      placement:
        constraints:
          - node.role == manager

  db:
    image: couchdb:2.3.1
    ulimits:
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - db-data:/opt/couchdb/data
    networks:
      - net

  influxdb:
    image: influxdb:1.7
    volumes:
      - influx-data:/var/lib/influxdb
    networks:
      - net

  agent:
    image: swarmpit/agent:2.2
    depends_on:
      - app
      - db
      - influxdb
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - net
    deploy:
      labels:
        swarmpit.agent: 'true'

networks:
  net:
    driver: overlay

volumes:
  db-data:
    driver: local
  influx-data:
    driver: local
EOFdockercompose


docker stack deploy -c swarmpit/docker-compose.yml swarmpit

Check the service status with docker service ls.

Also, check if the swarmpit_app component is up with docker service logs swarmpit_app.

If the swarmpit service has successful started, you get a log similar to the following.

swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:20:15 PM swarmpit.server invoke
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: Swarmpit is starting...
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:20:15 PM swarmpit.database invoke
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: Waiting for CouchDB ...
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:21:41 PM org.apache.http.impl.execchain.RetryExec execute
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: I/O exception (java.net.NoRouteToHostException) caught when processing request to {}->http://db:5984: No route to host (Host unreachable)
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:21:41 PM org.apache.http.impl.execchain.RetryExec execute
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: Retrying request to {}->http://db:5984
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:21:41 PM swarmpit.database invoke
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: ... connected after 76 sec
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:21:41 PM swarmpit.database invoke
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: DB schema created
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Single node setup finished
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Default token secret created
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Change reg types finished
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:21:41 PM swarmpit.database invoke
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: Waiting for InfluxDB ...
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:21:41 PM swarmpit.database invoke
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: ... connected after 0 sec
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:21:41 PM swarmpit.database invoke
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: InfluxDB RP: #{autogen an_hour a_day}
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:21:41 PM swarmpit.database invoke
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: InfluxDB CQ: #{}
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:21:41 PM swarmpit.server invoke
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: Swarmpit running on port 8080
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:21:41 PM swarmpit.setup invoke
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: Docker API: 1.43
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:21:41 PM swarmpit.setup invoke
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: Docker ENGINE: 24.0.5
swarmpit_app.1.t0ede7zjexpv@photon-machine    | Mar 16, 2024 8:21:41 PM swarmpit.setup invoke
swarmpit_app.1.t0ede7zjexpv@photon-machine    | INFO: Docker SOCK: /var/run/docker.sock

In web browser, enter to the docker swarm ip address and port 888.

Possible culprits and how to avoid them

1. basic networking dhcp/ntp/dns

Check with journalctl -f if there are any dhcp/ntp/dns issues during a running docker swarm stack. For instance, with DHCP avoid bridged binding to VMware By Broadcom Workstation for docker swarm using

  • Wifi adapter
  • usb-2-ethernet-adapter I haven't found out why those docker swarm setups with dhc/ntp/dns in cascade with Microsoft Windows 11 networking + VMware Workstation 17.5.1 networking + vhw21 + Photon OS 5.0 networking on Linux kernel 6.1.79 + Docker networking are not stable enough.

2. docker swarm overlay network

Photon OS supports docker overlay network. But depending on Photon OS release&build and docker version, double check the ports for overlay network, Also, see https://github.com/vmware/photon/issues/1321.

3. inter-service connectivity inside docker swarm service network

(to be added)

4. docker compose service version

To avoid a few limitation service version limitations, use version 3.9. Be care ful: Using the manual installation with original docker-compose.yml and version 3.9, docker stack deploy -c swarmpit/docker-compose.yml swarmpit does not work, because of the agent deploy mode global. It fails with failed to update service swarmpit_agent: Error response from daemon: rpc error: code = Unimplemented desc = service mode change is not allowed. With service version 3.9 and applying depends_on dependencies, agent deploy mode global can be safely deleted. Avoid 'API_VERSION

5. cascade sequence of starting service subcomponents

The cascade sequence is:

-- couchdb -- \
---------------- | -- app -- agent
-- influxdb --- /

This cascade sequence can be specified with the 'depends_on' element.

6. relaxed no file handling

Even with the cascade sequence, there can be situations of no attached files in app and couchdb. Therefore, add 'ulimits:nofile' expressions.

7. subcomponents release couchdb, influxdb, swarmpit app and swarmpit agent

First, start without any cpu and ram limitations and reservations. The application initial scale runs fine with 4gb ram. Cpu and ram limitations and reservations help to avoid security issues by specifying only the needed amount of resources. It also can be used to detect memory leaks. With new releases the setup has to be verified very carefully. Also, see Also, see on the forum e.g. https://github.com/swarmpit/swarmpit/issues/643.

Couchdb up to 2.3.1 supports 'admin party mode' and can be used without any modifications on the logic. Do not use Influxdb 1.8 because of issues. The setup uses 1.7, maybe release above 2.0 e.g. 2.7 would be fine, too.

Avoid too low initial memory and use cascade sequence and 'ulimits:nofile' specifications. The original service description docker-compose.yml crashes and you might see swarmpit_app entries (docker service logs swarmpit_app) like the following:

With 2gb ram only, even with the Java modification, the start of the influxdb container produces a crash. Without the deploy resources limits and reservations, and applying having applied 4 GB ram to the virtual machine, influxdb starts successfully.

swarmpit_app.1.ys5btih8201i@photon-machine    | java: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by java)
swarmpit_app.1.ys5btih8201i@photon-machine    | java: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/../lib/jli/libjli.so)

Useful commands

docker swarm init
docker node ls
docker swarm leave --force

docker stack deploy -c swarmpit/docker-compose.yml swarmpit
docker start ls
docker stack rm swarmpit

docker service ls
docker service logs swarmpit_app
docker service logs swarmpit_influxdb
docker service logs swarmpit_db
docker service logs swarmpit_agent

docker image prune --all --force
docker volume prune --all --force

Weblinks