openSUSE MicroOS notes

To be unbiased I gave try to MicroOS (containerd based openSUSE distribution).

WARNING!

Old installation (Mar 31 2024) has backdoored xz. From https://news.opensuse.org/2024/03/29/xz-backdoor/:

... long text removed ...

Update: As of Dec 2024 there is now version 5.6.3 (was 5.6.1).
$ ( source /etc/os-release && echo "$PRETTY_NAME $VERSION_ID" )

openSUSE MicroOS 20241226

$ xz --version

xz (XZ Utils) 5.6.3
liblzma 5.6.3

To quickly start:

Tip

There is now minimalist MicroOS + Combustion project that uses libvirt and virt-install to setup MicroOS just with root Login and root SSH login. Project is available on: https://github.com/hpaluch/microos-examples

Downloaded and flashed to USB stick:

Tested "System Roles":

MicroOS Desktop (GNOME) [RC]
MicroOS Container Host

WARNING! MicroOS installer is extremely aggressive - it by default removes all existing partitions! I strongly recommend to use whole drive and to disconnect all other drives on installation

After reboot (Container Host)

If you selected Container host but did not provide device with public SSH key on installation, you can only locally login as root, but not remotely (via SSH).

To enable password root login you can try:

transactional-update shell
echo "PermitRootLogin yes" > /etc/ssh/sshd_config.d/99-permit-root.conf
exit
reboot

Now you can login as root and/or upload public SSH key, etc...

Quick CLI podman example for Server Role Container Host:

we will run simple Bitnami Apache container

create directory for WWW hosting:

mkdir -p /srv/bitnami/apache1
echo '<h1>Hello, apache1 !</h1>' > /srv/bitnami/apache1/index.html
# TODO: should restrict permissions after container is created:
chmod a+rwx /srv/bitnami/apache

create script create_apache1.sh with contents:

#!/bin/sh
set -euo pipefail
set -x
IMAGE=docker.io/bitnami/apache
INST=apache1

podman run --detach \
  --publish 80:8080 \
  --name $INST \
  --restart never \
  --volume /srv/bitnami/$INST:/app:Z \
  --pull missing \
  $IMAGE
exit 0

make it executable and run it:

echmod +x create_apache1.sh
./create_apache1.sh

try: curl -v http://IP_OF_MY_MICROOS

you can also find what user-id is container using with:

podman exec -it apache1 bash
id
  uid=1001(1001) gid=0(root) groups=0(root),1001(1001)
fgrep 1001 /etc/passwd
  1001:*:1001:0:container user:/app:/bin/sh
mkdir testik
exit

Ehm, there is user named 1001 with UID=1001 - this may confuse systemd and other tools that things that number is always UID.

outside container we can see:

ls -ld /srv/bitnami/apache1/testik/

drwxr-xr-x. 1 1001 root 0 Mar 31 09:53 /srv/bitnami/apache1/testik/

so there is UID=1001 inside container.

Please be aware that podman uses iptables - when running above container I can see:

$ iptables -L -n

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
NETAVARK_INPUT  0    --  0.0.0.0/0            0.0.0.0/0            /* netavark firewall rules */

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
NETAVARK_FORWARD  0    --  0.0.0.0/0            0.0.0.0/0            /* netavark firewall rules */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain NETAVARK_FORWARD (1 references)
target     prot opt source               destination         
DROP       0    --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
ACCEPT     0    --  0.0.0.0/0            10.88.0.0/16         ctstate RELATED,ESTABLISHED
ACCEPT     0    --  10.88.0.0/16         0.0.0.0/0           

Chain NETAVARK_INPUT (1 references)
target     prot opt source               destination         
ACCEPT     17   --  10.88.0.0/16         0.0.0.0/0            udp dpt:53

Chain NETAVARK_ISOLATION_2 (1 references)
target     prot opt source               destination         

Chain NETAVARK_ISOLATION_3 (0 references)
target     prot opt source               destination         
DROP       0    --  0.0.0.0/0            0.0.0.0/0           
NETAVARK_ISOLATION_2  0    --  0.0.0.0/0            0.0.0.0/0    

$ iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
NETAVARK-HOSTPORT-DNAT  0    --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
NETAVARK-HOSTPORT-DNAT  0    --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
NETAVARK-HOSTPORT-MASQ  0    --  0.0.0.0/0            0.0.0.0/0           
NETAVARK-1D8721804F16F  0    --  10.88.0.0/16         0.0.0.0/0           

Chain NETAVARK-1D8721804F16F (1 references)
target     prot opt source               destination         
ACCEPT     0    --  0.0.0.0/0            10.88.0.0/16        
MASQUERADE  0    --  0.0.0.0/0           !224.0.0.0/4         

Chain NETAVARK-DN-1D8721804F16F (1 references)
target     prot opt source               destination         
NETAVARK-HOSTPORT-SETMARK  6    --  10.88.0.0/16         0.0.0.0/0            tcp dpt:80
NETAVARK-HOSTPORT-SETMARK  6    --  127.0.0.1            0.0.0.0/0            tcp dpt:80
DNAT       6    --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:10.88.0.2:8080

Chain NETAVARK-HOSTPORT-DNAT (2 references)
target     prot opt source               destination         
NETAVARK-DN-1D8721804F16F  6    --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80 /* dnat name: podman id: a9ebad6ccf484426c7fc308c108a0cd52caba132705332b3b5da5510061cde57 */

Chain NETAVARK-HOSTPORT-MASQ (1 references)
target     prot opt source               destination         
MASQUERADE  0    --  0.0.0.0/0            0.0.0.0/0            /* netavark portfw masq mark */ mark match 0x2000/0x2000

Chain NETAVARK-HOSTPORT-SETMARK (2 references)
target     prot opt source               destination         
MARK       0    --  0.0.0.0/0            0.0.0.0/0            MARK or 0x2000

If you are curios what NETAVARK means - it is network layer used by podman:

https://docs.podman.io/en/stable/markdown/podman-network.1.html
https://github.com/containers/netavark
I was unable to find useful documentation on Netavark, but it is RedHat product, so no surprise...
https://github.com/containers/netavark/issues/573
https://stackoverflow.com/questions/76130480/unable-to-forward-traffic-to-rootless-podman-container-with-firewalld

Installing RPM packages

Nearly all filesystems are read-only (with exception of /etc/ (via overlay), /usr/local, /home, /var/ and few others). You have to use:

wrapper transactional-update (via sudo)
all updates are written to 'forward snapshot' - you have to reboot to apply changes
if you did already one update without reboot you have to add --continue option to avoid loosing previous changes (!)

Example:

sudo transactional-update pkg install mc curl wget lynx vim tmux ncdu busybox-net-tools bind-utils
# changes not visible until you reboot!

# if you want to make additional changes without reboot you have to pass --continue:
sudo transactional-update --continue pkg in man man-pages

# and finally reboot
sudo reboot

Updating grub

I removed splash=silent and quiet from variable GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub.

To apply changes you have to use:

# again - if you make other changes withour reboot add --continue
sudo /sbin/transactional-update grub.cfg
sudo reboot

Installing Cockpit

Have to follow: https://documentation.suse.com/sle-micro/5.3/html/SLE-Micro-all/article-cockpit-slemicro.html

# WARNING! remove "-t" from above guide:
/sbin/transactional-update pkg in patterns-microos-cockpit cockpit
# now must reboot to new filesystem:
reboot
# after reboot:
systemctl enable --now cockpit.socket

Skip firewall sections - it is intentionally omitted:

https://bugzilla.opensuse.org/show_bug.cgi?id=1213627

Using Cockpit

the only account root is not allowed by default (see /etc/cockpit/disallowed-users)

so you need to create non-privileged user:

transactional-update shell
u=MYUSER # replace MYUSER with your username
/usr/sbin/useradd -m -s /bin/bash $u
passwd $u
/usr/sbin/groupadd -r wheel
/usr/sbin/usermod -G wheel -a $u
visudo 
# uncomment one of these lines:
# %wheel ALL=(ALL:ALL) ALL
# %wheel ALL=(ALL:ALL) NOPASSWD: ALL

reboot

if there is no /hoem/MYSER create do it manually:

u=MYUSER
mkdir /home/$u
chown $u:$u /home/$u

now you should be able to login as MYUSER to Cockpit
open https://IP_OF_YOU_MICROS:9090
fill Username of MYUSER and password you specified for Username
after login you can click on Turn on administrative access to have full privileges
now click on "Podman containers"
I had warning "Podman service is not active"
- check ON "Automatically start podman on boot"
- click on "Start podman"

Podman

Podman is RedHa't container management tool - RedHat tries to replace original Docker tools with it. If you know docker you can use most commands also with podman by just replacing 'docker' with 'podman'.

Podman is automatically installed if you selected MicroOS Container Host System Role.

You can use directly podman command or Cockpit Web UI.

To use it from Cockpit

login to Cockpit Web UI (see previous chapter)
after login you can click on Turn on administrative access to have full privileges
now click on "Podman containers"
I had warning "Podman service is not active"
- check ON "Automatically start podman on boot"
- click on "Start podman"

Which registries (Image servers) are used?

looking into /etc/containers/registries.conf

unqualified-search-registries = ["registry.opensuse.org", "registry.suse.com", "docker.io"]

nice quickstart: https://phoenixnap.com/kb/podman-tutorial

To create your first container in Podman

we will Apache web server provided by Bitnami, official page is on: https://github.com/bitnami/containers/tree/main/bitnami/apache#how-to-use-this-image

first prepare Volume for our Web pages:

# runs as your non-privileged user MYUSER
mkdir -p ~/www
# Notice weird escape of '!' - the \! does not work!
 echo "<h1>Hello `id -un` on `hostname -f`"'!</h1>' >  ~/www/index.html

in Cockpit Web UI go to "Podman containers"
click on "Create container"
- set name to "apache1" or so
- Owner: switch from root to your user
  - Note: for best security you should create dedicated user for each container to ensure better isolation
- on Image: type apache
- be sure to select docker.io/bitnami/apache (first name is Image host - docker.io, second is "namespace" - vendor bitnami, last is image name apache)
- uncheck "with terminal" (Apache is normally background process)
- now change tab to integration and:
- add "Port mapping"
  - Host port: 8080 (customizable), container port: 8080 (fixed, defined by Container creator), Protocol: TCP
  - note for ports below 1024 you have to run container as root or add 'net.ipv4.ip_unprivileged_port_start=80' to /etc/sysctl.conf. Not sure what is worse...
- click on Add volume
  - Host path: /home/MYUSER/www (configurable, where we put index.html), container path: /app (fixed, defined by Image creator), SELinux: Private (you can also try Shared)
- double check all values and then:
- click on Create and run
- now watch Container State: should start with downloading
- then created and finally running
- point your browser or curl command to

When something does not work:

try podmain container logs CONTAINER_NAME
in our case podman container logs apache1
to diagnose, what is wrong - we can attach with shell to running container:
```
CONT=apache1
podman exec -it $CONT bash
```

Please see these two guides regarding Podman and SELinux volumes:

NOTE: If something went wrong use:

Delete Container (container is local Instance)
But normally Keep Image - it is reusable read-only Image that can be used to create several containers...

To see containers (Instances) try (append --all to see also stopped containers):

$ podman container list

CONTAINER ID  IMAGE                            COMMAND               CREATED        STATUS        PORTS                   NAMES
714a08904352  docker.io/bitnami/apache:latest  /opt/bitnami/scri...  8 minutes ago  Up 8 minutes  0.0.0.0:8080->8080/tcp  apache1

To see Images (read-only filesystems "Templates"):

$ podman images

REPOSITORY                TAG         IMAGE ID      CREATED      SIZE
docker.io/bitnami/apache  latest      3eb9e9c5a5e5  2 weeks ago  518 MB

Using dedicated IP for container

Quite often we have to provide services on standard ports (22,80,443, etc...). In case of SSH (for GitLab for example) the only way is to use multiple IP addresses. To assing more than one IP address to single LAN card you have to:

install Network Manager text user interface (TUI) with:

sudo transactional-update pkg in NetworkManager-tui
# must reboot
sudo reboot

switch from DHCP assigned IPv4 to Static IPv4
- use sudo nmtui, reboot and verify that everything still works.
- edit Connection (that one with our NIC name)
- switch IPv4 CONFIGURATION to <Manual> and click on <Show>
  - have to fill-in Addresses (for a while only Primary), with Netmask (for example 192.168.0.10/24)
  - and Gateway and DNS Servers...

afer reboot use these commands to verify configuration:

ip l # network interfaces
ip a # IP addresses
ip r # routes
cat /etc/resolv.conf

optionally backup working configuration:

sudo bash
# run directly as root to expand '*.*':
cp /etc/NetworkManager/system-connections/*.* /root/

add additional IP addresses in NetworkManager
- when my Primary Address is 192.168.0.51/24
- I added another with 192.168.0.52/24
again reboot to ensure that new configuration works.

verify that there are both addresses assigned:

$ ip -br -4 a

lo               UNKNOWN        127.0.0.1/8 
enp0s8           UP             192.168.0.51/24 192.168.0.52/24

try ping from other Host to each IP address to verify that MicroOS host responds to both IP addresses.
here is content of my /etc/NetworkManager/system-connections/enp0s8.nmconnection

[connection]
id=enp0s8
uuid=f4477190-f17a-3d75-b7db-fdd594a5f476
type=ethernet
interface-name=enp0s8
timestamp=1711630770

[ethernet]

[ipv4]
address1=192.168.0.51/24,192.168.0.1
address2=192.168.0.52/24
dns=78.157.167.7;78.157.167.57;
may-fail=false
method=manual

[ipv6]
method=auto

[proxy]

Now we my try Apache Container again, this time bound it to 2nd IP address and port 80.

create shared index.html with:

# run as root
mkdir -p /var/www/html
echo "<h1>Hello from /var/www/html"'!</h1>' > /var/www/html/index.html

now run privileged container binding specific IP on port 80 and volume in SELinux shared mode (capital Z)

sudo podman run --detach --name apache80 \
  --publish 192.168.0.52:80:8080 --volume /var/www/html:/app:Z \
  docker.io/bitnami/apache:latest

Verify - STATUS must be Up

sudo podman container list

CONTAINER ID  IMAGE                            COMMAND               CREATED         STATUS         PORTS                      NAMES
7684a6126f54  docker.io/bitnami/apache:latest  /opt/bitnami/scri...  20 seconds ago  Up 16 seconds  192.168.0.52:80->8080/tcp  apache80

And try to access it on complete IP:port: in my case:

http://192.168.0.52
you should see content of your /var/www/html/index.html

Installing GitLab in container

To test something really heavyweight let's try GitLab CE:

official docs: https://docs.gitlab.com/ee/install/docker.html

GitLab container has to be run as root - so we should invoke all podman commands as root.

First download image for later use:

available CE tags are on https://hub.docker.com/r/gitlab/gitlab-ce/tags/

# As of 2024 Mar, image is over 3GB in size!

sudo podman pull docker.io/gitlab/gitlab-ce:latest
sudo podman images

REPOSITORY                  TAG         IMAGE ID      CREATED       SIZE
docker.io/gitlab/gitlab-ce  latest      86ef11c98cc8  26 hours ago  3.09 GB

Because GitLab provides its own SSH that should be on port 22 we have to:

create dedicated IP alias just for GitLab - see previous chapter. I will use 192.168.0.52
reconfigure SSHd on MicroOS to listen on primary IP address only - in my case 192.168.0.51

create file /etc/ssh/sshd_config.d/10-bind.conf with your primary IP:

# Listen on primary IPv4 address only
ListenAddress 192.168.0.51:22

restart sshd with systemctl restart sshd

verify that SSHd is bound to single IP address - 192.168.0.51 in example:

$ netstat -an | fgrep :22 | fgrep LISTEN

tcp        0      0 192.168.0.51:22         0.0.0.0:*               LISTEN

also verify that you can still connect from other PC to MicroOS SSH!

Now we will prepare environment

all do as root (we use privileged ports so we have no other choice)

# suffix is last number in IP address
export GITLAB_HOME=/var/opt/gitlab52
# braces {} work properly only in real BASH shell!
mkdir -p $GITLAB_HOME/{data,logs,config}

To be sure add appropriate entry to /etc/hosts of MicroOS, in my example:

192.168.0.52 gitlab52.example.com gitlab52

Now create script create_gitlab52.sh with contents:

#!/bin/bash
set -euo pipefail
set -x
# WARNING! You should replace 'latest' with your favorite GitLab version tag!
# See https://hub.docker.com/r/gitlab/gitlab-ce/tags/ for list
export IMAGE=docker.io/gitlab/gitlab-ce:latest
export IP=192.168.0.52
export GITLAB_HOME=/var/opt/gitlab52
export INST=gitlab52
export FQDN=gitlab52.example.com

podman run --detach \
  --hostname $FQDN \
  --add-host $FQDN:$IP \
  --env GITLAB_OMNIBUS_CONFIG="external_url 'http://$FQDN'" \
  --publish $IP:443:443 --publish $IP:80:80 --publish $IP:22:22 \
  --name $INST \
  --restart never \
  --volume $GITLAB_HOME/config:/etc/gitlab:Z \
  --volume $GITLAB_HOME/logs:/var/log/gitlab:Z \
  --volume $GITLAB_HOME/data:/var/opt/gitlab:Z \
  --shm-size 256m \
  --pull missing \
  $IMAGE
exit 0

Run it and as root watch logs with:

podman container logs -f gitlab52

Until you see messages with:

==> /var/log/gitlab/gitlab-rails/production_json.log
...

Now ensure that your client can resolve FQDN (gitlab52.example.com in example)

you have to use URL in form http://FQDN to access your GitLab
you should avoid using http://IP, because some links on GitLab will use FQDN instead of IP address (and browser will not find it)
Initial login is root
password can be found on MicroOS host in /var/opt/gitlab52/config/initial_root_password

WARNING! PostgreSQL runs sub-optimally on BTRFS /var/ volume - there should be at least disabled copy-on-write for its directory - or preferably have dedicated ext4 volume.

Proper backup

TODO

Image could be backed-up with podman save
Container could be backed-up with podman export
you also have to backup all volumes - preferably when Container is NOT running.

TODO

TODO:

how to bind on port < 1024, currently only 2 choices: a) running container as root b) avoid non-privileged processes to bind on ports < 1024 (sysctl)

Problems

No firewall included by default:

https://bugzilla.opensuse.org/show_bug.cgi?id=1213627
https://forums.opensuse.org/t/micro-os-suse-aeon-compared-to-fedora-silverblue/167663/11
however you can click on "enable" when installing from ISO - so one does not know what is intended behavior...

Privacy

Here is (incomplete) list of DNS requests made by MicroOS (only it no tunnel or no DoH is used).

While installing from ISO:

doc.opensuse.org
# at the end of installation looks like updates from mirrors:
codecs.opensuse.org
downloadcontentcdn.opensuse.org
download.opensuse.org
ftp.icm.edu.pl
ftp.linux.cz
ftp.man.poznan.pl
ftp.pbone.net
ftp.sh.cvut.cz
ftp.uni-bayreuth.de
ftp.uni-erlangen.de
mirror.alwyzon.net
mirrors.nic.cz
quantum-mirror.hu
tux.rainside.sk

On boot:

3.opensuse.pool.ntp.org
conncheck.opensuse.org

TODO

Quadlets - directly invoke podman from Systemd Units:

https://fale.io/blog/2023/05/17/manage-podman-containers-with-systemd-and-quadlet

Resources

MicroOS pi-hole demo:

https://www.youtube.com/watch?v=w9IimsV807Y
https://en.opensuse.org/SDB:Howto-pihole#Installation_on_MicroOS_-_Container
https://github.com/openSUSE/leapmicro-pihole-nextcloud-demo
- but it is missing Combustion file

Other:

https://en.opensuse.org/Portal:MicroOS

MicroOS - hpaluch/hpaluch.github.io GitHub Wiki

openSUSE MicroOS notes

After reboot (Container Host)

Installing RPM packages

Updating grub

Installing Cockpit

Podman

Using dedicated IP for container

Installing GitLab in container

Proper backup

TODO

Problems

Privacy

TODO

Resources

⚠️ GitHub.com Fallback ⚠️

MicroOS - hpaluch/hpaluch.github.io GitHub Wiki

openSUSE MicroOS notes

After reboot (Container Host)

Installing RPM packages

Updating grub

Installing Cockpit

Podman

Using dedicated IP for container

Installing GitLab in container

Proper backup

TODO

Problems

Privacy

TODO

Resources

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️