CoreOS - hpaluch/hpaluch.github.io GitHub Wiki

Fedora CoreOS (FCOS)

Status: installed under Proxmox, but needs useful workload for example

WARNING!

Apr 1 2024: regarding XZ backdoor (see https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users) my CoreOS installation (stable - slowest stream!) SEEMS to be NOT vulnerable by latest XZ. Please note that it is unclear if there are any yet unknown backdoors in older XZ versions.

Here are details:

 $ rpm-ostree status
 
  fedora:fedora/x86_64/coreos/stable
                   Version: 39.20240309.3.0 (2024-03-23T00:39:35Z)
                    Commit: 0d3740b2c05724aa74d57a27fca2f257d7d691daa2b6bb370cd45655190e0f64
              GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
 
 $ rpm -qi xz
 
 Name        : xz
 Version     : 5.4.4
 Release     : 1.fc39
 Architecture: x86_64
 Install Date: Sat Mar 23 00:37:05 2024
 Size        : 2066126
 Signature   : RSA/SHA256, Wed Aug  2 16:06:17 2023, Key ID 75cf5ac418b8e74c
 Source RPM  : xz-5.4.4-1.fc39.src.rpm
 Build Date  : Wed Aug  2 16:04:18 2023
 Build Host  : buildvm-x86-26.iad2.fedoraproject.org
 ...

However if you are using faster streams (testing or even next) you may have problem...

Important note:

Similar to MicroOS - minimal distribution for containers.

WARNING!

Both SUSE MicroOS and FCOS has issues with proper firewall supports

Visited:

Trying under Proxmox:

curl -fLOJ https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/39.20240309.3.0/x86_64/fedora-coreos-39.20240309.3.0-qemu.x86_64.qcow2.xz
xz -d fedora-coreos-39.20240309.3.0-qemu.x86_64.qcow2.xz
qemu-img info fedora-coreos-39.20240309.3.0-qemu.x86_64.qcow2 


image: fedora-coreos-39.20240309.3.0-qemu.x86_64.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 1.58 GiB
...

Important is virtual size 10 GiB

Now I created VM in Proxmox:

  • OS:
    • DVD: Do not use any media
  • System:
    • GPU: SPICE
    • Firmware: BIOS, Q35
    • enable Qemu Agent
  • 1 CPU (I have only 2 cores on hose, otherwise you should use more cores)
    • host CPU passthrough
  • Disks:
    • SCSI (Virtio SCSI single)
    • Storage local (needed for QCOW2)
    • Size: 10GB (to match image size)
    • Format: qcow2
    • Cache: writeback (unsafe)
    • Discard
  • CPU
    • 1 core (I use Host cores - 1)
    • Type: host
  • Memory: 4GB
  • Network
    • VirtIO
    • disabled firewall

When empty VM is created we will

VMID=125 # replace with your VMID!
ls -lh /var/lib/vz/images/$VMID/vm-$VMID-disk-0.qcow2
cp fedora-coreos-39.20240309.3.0-qemu.x86_64.qcow2 /var/lib/vz/images/$VMID/vm-$VMID-disk-0.qcow2
qm rescan

VM 125 (scsi0): size of disk 'local:125/vm-125-disk-0.qcow2' updated from 32G to 10G
# you can see that I create 32GB image instead of 10GB, but dont' mind

Now before first boot we have to expand Fedora disk (it will resize FS automatically - but ONLY on first boot)

qm disk resize $VMID scsi0 30G

Now very important:

  • create snapshot until it is too late!
  • CoreOS allows configuration only on first boot - including setting up Users...
  • If ignition fails for any reason you have to start from beginning - now you will find snapshot handy
# create snapshot called 'before-boot'
qm snapshot $VMID before-boot

  snapshotting 'drive-scsi0' (local:125/vm-125-disk-0.qcow2)

qm listsnapshot $VMID

  `-> before-boot                 2024-03-30 09:14:06     no-description
   `-> current                                            You are here!

But we are NOT done:

To create "Ignition" configuration we have to:

  • generate SSH key-pair on your Client machine, for example: 
    ssh-keygen -t ed25519
# answered destination file as: /home/USERNAME/.ssh/id_ed25519_coreos
  • next note public ssh key (in .pub) file, in my case: 
    cat ~/.ssh/id_ed25519_coreos.pub 

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKBZD2p///I6TSpWPz5GZd+RFSqC6T5+81cdo5S7LhA4 hpaluch@zotac
  • I used ed25519 to avoid copying long public SSH keys...
  • now we have to install 1 tool, in case of openSUSE LEAP 15.5 using: 
    sudo zypper in python3-yamllint butane
  • NOTE: openSUSE has butane RPM package but it is too old... It will just throw error 
    Error translating config: No translator exists for variant fcos with version 1.5.0
  • now we have to create input YAML file, following https://docs.fedoraproject.org/en-US/fedora-coreos/producing-ign/
  • be sure to replace SSH public key with content of your ~/.ssh/id_ed25519_coreos.pub
  • my example file called example.yaml:
variant: fcos
version: 1.5.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKBZD2p///I6TSpWPz5GZd+RFSqC6T5+81cdo5S7LhA4 hpaluch@zotac'

Validate it with:

yamllint example.yaml
``
You can safely ignore "error" "line too long".

Now we have to install latest `butane` tool, in my case:
```shell
curl -fLOJ https://github.com/coreos/butane/releases/download/v0.20.0/butane-x86_64-unknown-linux-gnu
chmod +x butane-x86_64-unknown-linux-gnu
./butane-x86_64-unknown-linux-gnu --version

   Butane 0.20.0

Now we can transform input YAML file to required JSON output:

./butane-x86_64-unknown-linux-gnu --pretty --strict example.yaml > example.ign

One simple way to "validate" output JSON is to pass it through jq tool:

jq < example.ign

There should be no error reported...

Here is full content of my example.ign - if you want to just replace SSH key you can skip Butane step:

{
  "ignition": {
    "version": "3.4.0"
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKBZD2p///I6TSpWPz5GZd+RFSqC6T5+81cdo5S7LhA4 login@dom"
        ]
      }
    ]
  }
}

Now we have to follow: https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-qemu/ to pass this configuration file to Proxmox VE (QEMU):

  • I copied my example.ign to Proxmox: /etc/local/example.ign
  • now we have to append custom arguments to our VM with: 
    # ensure that you will modify right VM!
qm config $VMID
qm set $VMID --args '-fw_cfg name=opt/com.coreos/config,file=/etc/local/example.ign'

Tip: to be able to watch and log boot messages we can add serial port to VM and use qm terminal to watch it.

qm set $VMID --serial0 socket

Now we will start logging, start VM and watch messages on its virtual serial port:

script ~/coreos-install.log
echo $VMID
# will be likely empty, because command "script" invokes new shell, define it again:
VMID=125
qm start $VMID
qm terminal $VMID

If everything works properly, you should see messages like this:

Fedora CoreOS 39.20240309.3.0
Kernel 6.7.7-200.fc39.x86_64 on an x86_64 (ttyS0)

.. ssh host keys omitted ...

enp6s18: 192.168.0.100 ...

Ignition: ran on 2024/03/30 14:49:23 UTC (this boot)
Ignition: user-provided config was applied
Ignition: wrote ssh authorized keys file for user: core

Now you can try to login to target IP (192.168.0.100 in my example) with username core and SSH key. Here is excerpt from my ~/.ssh/config:

Host coreos
     HostName 192.168.0.100
     HostKeyAlias coreos
     User core
     IdentityFile ~/.ssh/id_ed25519_coreos
     IdentitiesOnly yes

And simply try ssh coreos

Tip: you can use sudo Command to run Command as root

Basic overview of filesystem:

$ lsblk -f -o name,fstype,label,fsavail,fsuse%,mountpoints
NAME   FSTYPE LABEL      FSAVAIL FSUSE% MOUNTPOINTS
sda                                     
├─sda1                                  
├─sda2 vfat   EFI-SYSTEM                
├─sda3 ext4   boot        218.6M    31% /boot
└─sda4 xfs    root         27.2G     8% /var
                                        /sysroot/ostree/deploy/fedora-coreos/var
                                        /usr
                                        /etc
                                        /
                                        /sysroot

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4        30G  2.3G   28G   8% /sysroot
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           982M  168K  982M   1% /dev/shm
tmpfs           393M  844K  392M   1% /run
tmpfs           982M     0  982M   0% /tmp
/dev/sda3       350M  109M  219M  34% /boot
tmpfs           197M   12K  197M   1% /run/user/1000

# there are many invisible sub-volumes (???)

 mount | fgrep sda4
/dev/sda4 on /sysroot type xfs (ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
/dev/sda4 on / type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
/dev/sda4 on /etc type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
/dev/sda4 on /usr type xfs (ro,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
/dev/sda4 on /sysroot/ostree/deploy/fedora-coreos/var type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)
/dev/sda4 on /var type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota)

Trivial container play:

$ podman pull busybox
$ podman images

REPOSITORY                 TAG         IMAGE ID      CREATED        SIZE
docker.io/library/busybox  latest      ba5dc23f65d4  10 months ago  4.5 MB

$ podman run --name bb1 busybox /bin/sh -c "date;sleep 3;date"

Sat Mar 30 15:58:23 UTC 2024
Sat Mar 30 15:58:26 UTC 2024

# container exits after 3 seconds:

$ podman ps -a

CONTAINER ID  IMAGE                             COMMAND               CREATED         STATUS                     PORTS       NAMES
419e9ae11a0a  docker.io/library/busybox:latest  /bin/sh -c date;s...  26 seconds ago  Exited (0) 23 seconds ago              bb1

TODO: Something more useful - with networking...

Customization

Installing additional packages:

  • use rpm-ostree instead of yum or dnf
  • rpm-ostree basically provides 'snapshot like' experience...
  • use rpm-ostree search PACKAGE` to search for PACKAGE
  • to install Midnight Commander, try: 
    rpm-ostree search mc
sudo rpmostree install mc vim-enhanced

Changes queued for next boot. Run "systemctl reboot" to start a reboot

Now notice that there is Layered packages layer:

$ rpm-ostree status

State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Sat 2024-03-30 15:16:40 UTC)
Deployments:
  fedora:fedora/x86_64/coreos/stable
                  Version: 39.20240309.3.0 (2024-03-23T00:39:35Z)
               BaseCommit: 0d3740b2c05724aa74d57a27fca2f257d7d691daa2b6bb370cd45655190e0f64
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
                     Diff: 66 added
          LayeredPackages: mc vim-enhanced

● fedora:fedora/x86_64/coreos/stable
                  Version: 39.20240309.3.0 (2024-03-23T00:39:35Z)
                   Commit: 0d3740b2c05724aa74d57a27fca2f257d7d691daa2b6bb370cd45655190e0f64
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C

To apply changes you have to run sudo systemctl reboot (same as in MicroOS).

Run normally containers

TODO:

Resources