Offline Upgrading Photon OS 3.0 to 5.0 - dcasota/photonos-scripts GitHub Wiki

Offline upgrading Photon OS virtual machines isn't difficult, but it isn't described in the docs because the initial situation, configuration and installed applications lead rapidly to environment-specific cascades. To get an idea of so-called airgap server deployment topologies, have a look to VMware By Broadcom Telco Cloud Automation in which the commercial version of Photon OS is in use as subsystem, see https://docs.vmware.com/en/VMware-Telco-Cloud-Automation/1.9.5/com.vmware.tca.userguide/GUID-27304C0A-2D5C-4E99-8932-E7FE08B92D31.html and the follow-up chapters. The concept and a few commands in the guide are reusable for the open-source version of Photon OS as well.

The offline upgrade requirement isn't new and traces can be found here, here and here.

The following 5-steps-guide is a slightly different approach to not use any network on the target environment. Basically it populates a virtual disk with packages and - after the virtual disk has been transported and attached to the target virtual machine - the upgrade is applied from the repositories on that mounted virtual disk.

Remarks:

Specify $basearch, if target virtual machine $HOSTTYPE is different to helper virtual machine cpu architecture.
The virtual machine virtual hardware remains unchanged, still MBR boot-mode, no secure boot.

1. Populate 3.0, 4.0 and 5.0 repositories on a Photon OS helper virtual machine

A. Create a Photon OS 5.0 helper virtual machine with VMware Photon OS packages connectivity.

Consider the same virtual hardware as the Photon OS 3.0 target virtual machine. If using e.g. https://packages.vmware.com/photon/5.0/GA/ova/photon-hw15-5.0-dde71ec57.x86_64.ova, the virtual hardware of the 3.0 virtual machine must be upgraded at least to virtual hardware 15.
2vcpu, 2gb ram
Consider a ~~250GB~~ 130GB hard disk. The following recipe syncs a lot of repo files. Storing all versions of any packages is unoptimized, but I haven't found an optimized logic yet.

B. Add the Photon OS repositories 3.0, 4.0 and 5.0 and do a reposync.

Hint: The following code snippet adds all repositories but 3.0 GA-, iso- and debuginfo-repos.

cat > /etc/yum.repos.d/photon-updates_3.0.repo << "EOF-updates_3.0"
[photon-updates-3.0-$basearch]
name=VMware Photon Linux 3.0 ($basearch) Updates
baseurl=https://packages.vmware.com/photon/3.0/photon_updates_3.0_$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY-4096
gpgcheck=1
enabled=1
skip_if_unavailable=1
EOF-updates_3.0

cat > /etc/yum.repos.d/photon-release_3.0.repo << "EOF-release_3.0"
[photon-release-3.0-$basearch]
name=VMware Photon Linux 3.0 ($basearch)
baseurl=https://packages.vmware.com/photon/3.0/photon_release_3.0_$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY-4096
gpgcheck=1
enabled=1
skip_if_unavailable=1
EOF-release_3.0

cat > /etc/yum.repos.d/photon-extras_4.0.repo << "EOF-extras_4.0"
[photon-extras-4.0-$basearch]
name=VMware Photon Linux Extras 4.0 ($basearch)
baseurl=https://packages.vmware.com/photon/4.0/photon_extras_4.0_$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY-4096
gpgcheck=1
enabled=1
skip_if_unavailable=1
skip_md_filelists=1
EOF-extras_4.0

cat > /etc/yum.repos.d/photon-updates_4.0.repo << "EOF-updates_4.0"
[photon-updates-4.0-$basearch]
name=VMware Photon Linux 4.0 ($basearch) Updates
baseurl=https://packages.vmware.com/photon/4.0/photon_updates_4.0_$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY-4096
gpgcheck=1
enabled=1
skip_if_unavailable=1
EOF-updates_4.0

cat > /etc/yum.repos.d/photon-release_4.0.repo << "EOF-release_4.0"
[photon-release-4.0-$basearch]
name=VMware Photon Linux 4.0 ($basearch)
baseurl=https://packages.vmware.com/photon/4.0/photon_release_4.0_$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY-4096
gpgcheck=1
enabled=1
skip_if_unavailable=1
EOF-release_4.0

cat> /etc/yum.repos.d/photon-extras_5.0.repo << "EOF-extras_5.0"
[photon-extras-5.0-$basearch]
name=VMware Photon Linux Extras 5.0 ($basearch)
baseurl=https://packages.vmware.com/photon/5.0/photon_extras_5.0_$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY-4096
gpgcheck=1
enabled=1
skip_if_unavailable=1
skip_md_filelists=1
EOF-extras_5.0

cat > /etc/yum.repos.d/photon-updates_5.0.repo << "EOF-updates_5.0"
[photon-updates-5.0-$basearch]
name=VMware Photon Linux 5.0 ($basearch) Updates
baseurl=https://packages.vmware.com/photon/5.0/photon_updates_5.0_$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY-4096
gpgcheck=1
enabled=1
skip_if_unavailable=1
EOF-updates_5.0

cat > /etc/yum.repos.d/photon-release_5.0.repo << "EOF-release_5.0"
[photon-release-5.0-$basearch]
name=VMware Photon Linux 5.0 ($basearch)
baseurl=https://packages.vmware.com/photon/5.0/photon_release_5.0_$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY-4096
gpgcheck=1
enabled=1
skip_if_unavailable=1
EOF-release_5.0

chmod 644 /etc/yum.repos.d/*.repo

sed -i "s/\$basearch/$HOSTTYPE/g" /etc/yum.repos.d/photon-updates_3.0.repo /etc/yum.repos.d/photon-release_3.0.repo
sed -i "s/\$basearch/$HOSTTYPE/g" /etc/yum.repos.d/photon-extras_4.0.repo /etc/yum.repos.d/photon-updates_4.0.repo /etc/yum.repos.d/photon-release_4.0.repo
sed -i "s/\$basearch/$HOSTTYPE/g" /etc/yum.repos.d/photon-extras_5.0.repo /etc/yum.repos.d/photon-updates_5.0.repo /etc/yum.repos.d/photon-release_5.0.repo

# Disable repo which were configured during installation of the helper vm
sed -i "s/enabled=1/enabled=0/g" /etc/yum.repos.d/photon.repo
sed -i "s/enabled=1/enabled=0/g" /etc/yum.repos.d/photon-debuginfo.repo
sed -i "s/enabled=1/enabled=0/g" /etc/yum.repos.d/photon-extras.repo
sed -i "s/enabled=1/enabled=0/g" /etc/yum.repos.d/photon-iso.repo
sed -i "s/enabled=1/enabled=0/g" /etc/yum.repos.d/photon-updates.repo

tdnf makecache
tdnf reposync

Reposync takes hours.

C. Shutdown the helper virtual machine. Transport its virtual disk to the air-gapped target environment.

2. Backup the air-gapped target Photon OS virtual machine(s)

A. Perform a backup.

B. Consider the upgrade first in an air-gapped test environment with the backuped target Photon OS virtual machine.

C. Attach the helper virtual disk to the offline 3.0 target virtual machine.
The tested offline target is a Photon OS 3.0 GA virtual machine without floppy disk, without network adapter and without cd/dvd.

3. Perform upgrade pre-steps.

A. Notice the added virtual disk.

fdisk -l

B. Mount the attached virtual disk e.g. on /dev/sdb3

mkdir /mnt/local
mount /dev/sdb3 /mnt/local

C. Configure the local repositories.

chmod 644 -R /mnt/local/root/photon-release-3.0-x86_64
chmod 644 -R /mnt/local/root/photon-updates-3.0-x86_64
rpm -ivh /mnt/local/root/photon-updates-3.0-x86_64/x86_64/createrepo_c-0.11.1-5.ph3.x86_64.rpm
createrepo /mnt/local/root/photon-release-3.0-x86_64
createrepo /mnt/local/root/photon-updates-3.0-x86_64

cat > /etc/yum.repos.d/photon.repo << "EOF-release"
[Local photon-release $basearch]
name=Local VMware Photon Linux $releasever ($basearch)
baseurl=file:///mnt/local/root/photon-release-$releasever-$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY
skip_if_unavailable=1
EOF-release

cat > /etc/yum.repos.d/photon-updates.repo << "EOF-updates"
[Local photon-updates $basearch]
name=Local VMware Photon Linux $releasever ($basearch) Updates
baseurl=file:///mnt/local/root/photon-updates-$releasever-$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY
skip_if_unavailable=1
EOF-updates

cat > /etc/yum.repos.d/photon-extras.repo << "EOF-extras"
[Local photon-extras $basearch]
name=Local VMware Photon Linux Extras $releasever ($basearch)
baseurl=file:///mnt/local/root/photon-extras-$releasever-$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY
skip_if_unavailable=1
EOF-extras

chmod 644 /etc/yum.repos.d/*.repo

# 3.0 cannot read variables, hence replace them with hardcoded values.
sed -i "s/\$basearch/$HOSTTYPE/g" /etc/yum.repos.d/photon-extras.repo /etc/yum.repos.d/photon-updates.repo /etc/yum.repos.d/photon-release.repo
sed -i "s/\$releasever/3.0/g" /etc/yum.repos.d/photon-extras.repo /etc/yum.repos.d/photon-updates.repo /etc/yum.repos.d/photon-release.repo

# these repos should already be disabled
sed -i "s/enabled=1/enabled=0/g" /etc/yum.repos.d/photon-debuginfo.repo /etc/yum.repos.d/photon-iso.repo

tdnf makecache

4. Apply the upgrade.

A. Save the status before upgrade for debug purposes. Save the following commands output.

cat /etc/photon-release
uname -a
systemctl list-unit-files --all 2>/dev/null
networkctl
resolvectl
iptables --list
docker container ls
ls -ll /etc/yum.repos.d/
ls -ll /etc/systemd/network

B. Make virtual disk mount permanently. Check /dev/sdb3 to the virtual disk attached.

tdnf install awk -y
cp /etc/fstab /etc/fstab.org
UUID=`blkid /dev/sdb3 | awk '{ print $2 }' | awk -F '=' '{ print $2}' | awk -F '"' '{ print $2}'`
echo "UUID=$UUID    /mnt/local ext4 defaults 0 2">>/etc/fstab

C. Apply a 3.0 update

tdnf update tdnf -y
# solve distroverpkg issue
tdnf update rpm -y --releasever=3.0
tdnf update photon-release -y --releasever=3.0
# solve issue https://github.com/vmware/photon/issues/1530
echo "skip_md_filelists=1">>/etc/yum.repos.d/photon.repo
echo "skip_md_filelists=1">>/etc/yum.repos.d/photon-updates.repo
echo "skip_md_filelists=1">>/etc/yum.repos.d/photon-extras.repo
rm -rf /var/cache/tdnf/
tdnf makecache --refresh

# Put here packages which cause conflict on update/upgrade
# tdnf remove bc liota -y

# apply update
tdnf update -y
rm /etc/yum.repos.d/*.rpmnew

# apply mkinitrd (no need to reboot)
tdnf install dracut -y
mkinitrd
# mkinitrd -q /boot/initrd.img-$(uname -r) $(uname -r)

D. Apply the upgrade to 4.0

# Prepare the 4.0 repositories without declaring them yet in /etc/yum.repos.d/.
chmod 644 -R /mnt/local/root/photon-extras-4.0-x86_64
chmod 644 -R /mnt/local/root/photon-release-4.0-x86_64
chmod 644 -R /mnt/local/root/photon-updates-4.0-x86_64
createrepo /mnt/local/root/photon-extras-4.0-x86_64
createrepo /mnt/local/root/photon-release-4.0-x86_64
createrepo /mnt/local/root/photon-updates-4.0-x86_64

# upgrade without reboot
tdnf install -y photon-upgrade

# Before upgrade, redirect to $releasever offline repositories
cat > /etc/yum.repos.d/photon.repo << "EOF-release"
[Local photon-release $basearch]
name=Local VMware Photon Linux $releasever ($basearch)
baseurl=file:///mnt/local/root/photon-release-$releasever-$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY
gpgcheck=1
enabled=1
skip_if_unavailable=1
EOF-release

cat > /etc/yum.repos.d/photon-updates.repo << "EOF-updates"
[Local photon-updates $basearch]
name=Local VMware Photon Linux $releasever ($basearch) Updates
baseurl=file:///mnt/local/root/photon-updates-$releasever-$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY
gpgcheck=1
enabled=1
skip_if_unavailable=1
EOF-updates

# Add extras-repository and transport it, or delete the entry.
rm /etc/yum.repos.d/photon-extras.repo

/usr/bin/photon-upgrade.sh --precheck-only

/usr/bin/photon-upgrade.sh --upgrade-os --assume-yes

E. Reboot
Proceed a reboot.

F. Apply the upgrade to 5.0

# Prepare the 5.0 repositories without declaring them yet in /etc/yum.repos.d/.
chmod 644 -R /mnt/local/root/photon-extras-5.0-x86_64
chmod 644 -R /mnt/local/root/photon-release-5.0-x86_64
chmod 644 -R /mnt/local/root/photon-updates-5.0-x86_64
createrepo /mnt/local/root/photon-extras-5.0-x86_64
createrepo /mnt/local/root/photon-release-5.0-x86_64
createrepo /mnt/local/root/photon-updates-5.0-x86_64

# Before reboot, redirect to offline repositories
cat > /etc/yum.repos.d/photon.repo << "EOF-release"
[Local photon-release $basearch]
name=Local VMware Photon Linux $releasever ($basearch)
baseurl=file:///mnt/local/root/photon-release-$releasever-$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY-4096
gpgcheck=1
enabled=1
skip_if_unavailable=1
EOF-release

cat > /etc/yum.repos.d/photon-updates.repo << "EOF-updates"
[Local photon-updates $basearch]
name=Local VMware Photon Linux $releasever ($basearch) Updates
baseurl=file:///mnt/local/root/photon-updates-$releasever-$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY-4096
gpgcheck=1
enabled=1
skip_if_unavailable=1
EOF-updates

cat > /etc/yum.repos.d/photon-extras.repo << "EOF-extras"
[Local photon-extras $basearch]
name=Local VMware Photon Linux Extras $releasever ($basearch)
baseurl=file:///mnt/local/root/photon-extras-$releasever-$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY-4096
gpgcheck=1
enabled=1
skip_if_unavailable=1
EOF-extras

rm /etc/yum.repos.d/photon-release.repo

chmod 644 /etc/yum.repos.d/*.repo

# upgrade without reboot
/usr/bin/photon-upgrade.sh --precheck-only
/usr/bin/photon-upgrade.sh --upgrade-os --assume-yes

G. Reboot
Proceed a reboot.

H. Save the status after upgrade for debug purposes. Save the following commands output. Analyze the differences.

cat /etc/photon-release
uname -a
systemctl list-unit-files --all 2>/dev/null
networkctl
resolvectl
iptables --list
docker container ls
ls -ll /etc/yum.repos.d/
ls -ll /etc/systemd/network

5. Perform upgrade post-steps.

A. Check configuration, applications' functionality.

B. Cleanup of the local repositories.

  ```
  # fill-in
  rm /etc/fstab.rpmnew
  rm /etc/yum.repos.d/*.rpmnew
  ```

C. Install additional software e.g. tdnf install open-vm-tools -y

D. Delete snapshot(s) if not needed anymore.

E. Apply the recipe in production.

Before, consider the following TODO list.

~~How to copy the repo to the first disk and to relink the packages before detaching the virtual disk?~~
For testing purposes, the custom recipe syncs lot of packages from packages.vmware.com/photon. Hence the disk file needed is ~~250GB~~ 130GB. How to identify only necessary packages 3.0, 4.0 and 5.0 ? see https://github.com/vmware/photon/issues/1544#issuecomment-2031080336.
The custom recipe requires that the virtual disk with packages is attached, but it doesn't copy the packages to the local disk. Hence it is not possible yet to get rid of the attached virtual disk. How to relink copied packages as '@system' ?
~~gpgcheck isn't implemented, no cert-checks.~~
Differences between a vanila Photon OS 5.0 and an upgraded system? [fill-in observations]