Configure PXE boot for Hypershift Agent Power CI - hypershift-on-power/hack GitHub Wiki

Setup PXE boot in Bastion Node:

Bastion node and supporting resource creation(one time process):

1. PowerVS workspace creation

  • Create a PowerVS workspace in desired location

2. Network creation

  • We need both public and private network to setup PXE boot.
  • Use 1.1.1.1 as DNS server in both networks.
  • For private network CIDR 192.168.140.0/24 can be used, which is used as reference in upcoming sections.

3. Internetwork communication

  • Post creation of private network, need to enable the internetwork communication, there are 2 ways of achieving this
  • Raise a support ticket to enable the internetwork communication for the private network(manual) - sample support ticket
  • Create a cloud connection with that private network to enable communication(automatic)

4. Bastion node creation

  • Need to create bastion node with both public and private network attached to it. Only minimum compute is required for bastion so 0.25 CPU and 4 GB of memory is enough.
  • Use UI which allow you to enter static IP address 192.168.140.2 for the private network to keep a static ip for bastion.

5. Dummy image creation

  • Dummy image used for the deployment for the root disk for the compute node for the agent cluster.
  • Will boot initially with this image then later will boot it with discovery ISO via DHCP bootp request.
  • Initial boot image link
  • Use below pvsadm command to import it into your workspace.
pvsadm image  import --pvs-instance-id <pvs_instance_id> --bucket power-objs-bucket  --object rhcos-none.ova.gz --api-key <my-api-key> --pvs-image-name rhcos-none --bucket-region us-south --accesskey <hmac_access> --secretkey <hmac_secret>
  • You can get the access and secret key from bucket's Service Credentials section here.

Bastion node preparation(one time process):

Need to hosts following services in bastion node

  • SNAT
  • DHCP
  • PXE

Network Configuration(SNAT)

Create the following file and run the script. This script will allow the private network to talk to public world(snat).

$ cat ip-forward.sh
#!/bin/bash -x

set -x

PRIVATE_INT=$1
PUBLIC_INT=$2

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -A FORWARD -i $PRIVATE_INT -o $PUBLIC_INT -j ACCEPT
iptables -A FORWARD -i $PUBLIC_INT -o $PRIVATE_INT -m state --state ESTABLISHED,RELATED \
         -j ACCEPT
iptables -t nat -A POSTROUTING -o $PUBLIC_INT -j MASQUERADE

iptables -A FORWARD -i $PRIVATE_INT -j ACCEPT
iptables -A FORWARD -o $PRIVATE_INT -j ACCEPT

ethtool --offload $PRIVATE_INT rx off tx off
ethtool --offload $PUBLIC_INT rx off tx off
ethtool -K $PUBLIC_INT tso off
ethtool -K $PRIVATE_INT tso off
ethtool -K $PRIVATE_INT gso off

ifconfig ${PUBLIC_INT} mtu 1450 up
ifconfig ${PRIVATE_INT} mtu 1450 up

Run the script

ip-forward.sh env3 env2

Here env3 is private interface and env2 is public interface. Use ip a or other suitable command to look at the network configuration of bastion node and run the script with proper args.

Note: For heterogeneous clusters, add route between PowerVS and VPC using the following command,

ip route add $VPC_IP_RANGE via $GATEWAY_IP dev $PRIVATE_INT

Here, $VPC_IP_RANGE is the IP Range of the VPC, $GATEWAY_IP is PowerVS Private Network Gateway IP and $PRIVATE_INT private interface.

DHCP

# Install the dhcp-server

$ yum install -y dhcp-server
$ systemctl enable dhcpd
# Modify the dhcpd.conf file with your private network configuration, typical file looks like this:
$ cat /etc/dhcp/dhcpd.conf
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp-server/dhcpd.conf.example
#   see dhcpd.conf(5) man page
#

ddns-update-style interim;
default-lease-time 14400;
max-lease-time 14400;

option interface-mtu 1450;
option routers                  192.168.140.2;
option broadcast-address        192.168.140.255;
option subnet-mask              255.255.255.0;
option domain-name-servers      1.1.1.1;
allow bootp;

subnet 192.168.140.0 netmask 255.255.255.0 {
    interface env3;
    # Static entries

    # Example host entry:
    # host rhcos-worker-1 { hardware ethernet fa:37:97:b7:6a:20; fixed-address 192.168.144.211; }
    # this will not give out addresses to hosts not listed above
    
    # deny unknown-clients;

    # this is PXE specific
    filename "boot/grub2/powerpc-ieee1275/core.elf";

    next-server 192.168.140.2;
}
# Restart the dhcpd service
$ systemctl restart dhcpd

PXE

# Install tftp server and configure
$ yum install -y tftp-server
$ systemctl enable tftp
$ systemctl start tftp
# Prepare a GRUB netboot directory.
$ grub2-mknetdir --net-directory=/var/lib/tftpboot

# Above command creates the following files:
$ ls -l /var/lib/tftpboot/boot/grub2/
total 16
drwxr-xr-x. 2 root root   25 Jun 14 04:57 fonts
drwxr-xr-x. 2 root root 8192 Jun 14 04:57 powerpc-ieee1275

Create grub.cfg

Need to create below config file to create the menu entry for iso and mac mapping

$ cat /var/lib/tftpboot/boot/grub2/grub.cfg
# Note this file mostly matches the grub.cfg file from within the
# efiboot.img on the Fedora Server DVD iso. Diff this file with that
# file in the future to pick up changes.
#
# One diff to note is we use linux and initrd instead of linuxefi and
# initrdefi. We do this because it works and allows us to use this same
# file on other architectures. https://github.com/coreos/fedora-coreos-config/issues/63
#
# This file is loaded directly when booting via El Torito, and indirectly
# from a stub config in efiboot.img when booting via the hybrid ESP.

set default="1"

function load_video {
  insmod efi_gop
  insmod efi_uga
  insmod video_bochs
  insmod video_cirrus
  insmod all_video
}

load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2

set timeout=5
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'RHEL CoreOS (Live)' --class fedora --class gnu-linux --class gnu --class os {
}

Install squid proxy

Squid proxy is needed to connect the ingress endpoints to workers from bastion

$ yum install -y squid
$ systemctl enable squid
$ cat /etc/squid/squid.conf
http_access allow all
http_port 2005
debug_options ALL,2
coredump_dir /var/spool/squid 
$ systemctl start squid

Copy scripts

Scripts which are required to setup the network boot from CI

$ git clone https://github.com/ppc64le-cloud/hypershift-agent-automation.git
$ mkdir /root/agent-ci
$ mkdir /root/agent-ci/scripts
$ cp hypershift-agent-automation/hack/upstream-ci-scripts/* /root/agent-ci/scripts

Boot the workers(only for manual installation)

This is required if you are booting the workers manually.

Below process we have to follow every time when we have different ISO to host in our DHCP TFTP server. Have automated the below process and scripts are here https://github.com/ppc64le-cloud/hypershift-agent-automation/

# Download and copy the ISO to the bastion node:
$ mount -o loop 307de63f-b667-4402-bc27-04b2742fe2c3.iso /mnt/iso/


# Copy the entire images folder into following dir
$ cp -r /mnt/iso/images /var/lib/tftpboot/

$ cp /mnt/iso/boot/grub/grub.cfg /var/lib/tftpboot/boot/grub2

# sample grub.cfg file looks like
[root@hypershift-bastion ~]# cat /mnt/iso/boot/grub/grub.cfg
# Note this file mostly matches the grub.cfg file from within the
# efiboot.img on the Fedora Server DVD iso. Diff this file with that
# file in the future to pick up changes.
#
# One diff to note is we use linux and initrd instead of linuxefi and
# initrdefi. We do this because it works and allows us to use this same
# file on other architectures. https://github.com/coreos/fedora-coreos-config/issues/63
#
# This file is loaded directly when booting via El Torito, and indirectly
# from a stub config in efiboot.img when booting via the hybrid ESP.

set default="1"

function load_video {
  insmod efi_gop
  insmod efi_uga
  insmod video_bochs
  insmod video_cirrus
  insmod all_video
}

load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2

set timeout=5
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'RHEL CoreOS (Live)' --class fedora --class gnu-linux --class gnu --class os {
	linux /images/pxeboot/vmlinuz ignition.firstboot ignition.platform.id=metal 'coreos.live.rootfs_url=https://assisted-image-service-multicluster-engine.rdr-pravin-jun12-c18b889c62f65204a6ec05dbac99a2d4-0000.jp-tok.containers.appdomain.cloud/boot-artifacts/rootfs?arch=ppc64le&version=4.13.0'
################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################ COREOS_KARG_EMBED_AREA
	initrd /images/pxeboot/initrd.img /images/ignition.img /images/assisted_installer_custom.img
}

Note:

  1. Find a way to place the images folder for every run in a different dir structure
  2. Modify the grub.cfg file to point to correct vmlinuz and initrd files
  • Create the worker node with the private network
  • Once MAC and IP address assigned, feed them into /etc/dhcp/dhcpd.conf and restart the dhcpd service
  • Modify the grub.cfg as well if required
⚠️ **GitHub.com Fallback** ⚠️