Installation - Mellanox/mlxsw GitHub Wiki

By default, the switch comes pre-installed with ONIE, which "defines an open 'install environment' for bare metal network switches" This document describes Fedora installation using ONIE and how to install ONIE.

1. Getting the Installation Files
2. Default Credentials
3. Installing with ONIE
   1. Preparing the Installation
   2. Performing the Installation
   3. Uninstalling
4. Customizing the Installation
5. Post Installation Updates
6. Installing ONIE
   1. Using PXE
   2. Using a USB Drive
   3. Using kexec
7. Querying ASIC Revision
   1. Using devlink
   2. Using mstflint

For Fedora, an installation tarball is available here. The following files are included in the provided tarball:

• The operating system ONIE installer - Fedora-ONIE-installer.bin
• Auto-install file (kickstart) for ONIE - install-onie.ks
• Recent Fedora ISO

Note that the Fedora ISO image is there to ease the installation process; It is an unmodified copy of the ISO that can be downloaded directly from Fedora.

User: root Password: switchdevftw

The root password can be easily changed by editing the rootpw command in the kickstart file. It is highly recommended to do so!

The ONIE project describes a variety of installation options. This section describes installation using an HTTP server accessible through the management port on the switch.

The extracted contents of the ISO file, the ONIE installer and the kickstart file (e.g. install-onie.ks), should all be accessible through an HTTP server.

To extract the contents from the ISO file, you must first mount it. Run:

$ mount -o loop,ro -t iso9660 /path/file.iso /path/iso/directory

Then copy the contents over to the HTTP server directory. Run:

$ cp -r /path/iso/directory /path/http/server

Finally, edit the auto-install file to point to the HTTP server. In the kickstart file, edit the url command to point to the extracted ISO directory.

Note: The ONIE installer makes a few assumptions about the location and name of the auto-install. In the provided package, it is assumed that the kickstart file is named install-onie.ks and that it is located in a directory called ks, placed next to the ONIE installer. For example:

├── ks
│   ├── install-onie.ks
├── Fedora-ONIE-installer.bin

Reboot the switch and select ONIE: Resuce in GRUB menu as the discovery phase is unnecessary:

 +----------------------------------------------------------------------------+
 | ONIE: Install OS                                                           |
 |*ONIE: Rescue                                                               |
 | ONIE: Uninstall OS                                                         |
 | ONIE: Update ONIE                                                          |
 | ONIE: Embed ONIE                                                           |
 |                                                                            |

In the ONIE prompt run the following:

ONIE:/ # onie-nos-install http://server/Fedora-ONIE-installer.bin

From this point on, the installation is completely automatic. Once it is complete, the switch reboots into the newly installed system.

Reboot the switch and select ONIE in the GRUB menu:

      Fedora (4.8.13-100.fc23.x86_64) 23 (Twenty Three)
      Fedora (0-rescue-851011758d7ea341a944f10920a2faa5) 23 (Twenty Three)
      ONIE

Then select ONIE: Uninstall OS. This removes the partitions used by the OS, but keeps ONIE installed, so that a new OS can be installed using it.

As explained above, the installation is automated using auto-install files. These files can be easily edited to provide different answers during the installation, and to select different packages to install by default. Please refer to the official RedHat kickstart documentation for more info.

Creating an ONIE-compatible installer from an ISO image is explained in ONIE's Github repository, in case one wants to create its own installer.

Once Fedora is installed, it is advised to upgrade all packages to the latest available by using:

$ dnf upgrade

Notice this would also upgrade the kernel image (assuming a newer one is available) and a reboot would be required following the command's completion.

You can find more information here or in the DNF man page.

At this point it is also possible to use DNF and install packages of interest that are not part of the ISO. Some notable packages worthy of mentioning are: iproute-tc gcc teamd bridge-utils

This can be achieved by using:

$ dnf install <package>

To install ONIE on the switch you might need to access the BIOS (Ctrl+b) and change the boot order. If the BIOS is password-protected, please contact Mellanox global support.

For both methods described below you need the ONIE installation files which are available here. Version r5.0.1410 is used as an example.

Further information regarding ONIE installation can be found in ONIE's official documentation located here.

Copy the following files to the appropriate place on the PXE server:

mlnx_x86-r5.0.1410.vmlinuz
recovery-x86_64-mlnx_x86-r5.0.1410.initrd

Create the following PXE label on the server:

LABEL ONIE
	kernel mlnx_x86-r5.0.1410.vmlinuz
	append initrd=recovery-x86_64-mlnx_x86-r5.0.1410.initrd console=ttyS0,115200n8 ramdisk_size=1048576 root=/dev/ram boot_reason=embed install_url=file:///lib/onie/onie-updater boot_env=recovery

Make sure the switch is configured to boot from PXE, reboot it and choose the ONIE label in the PXE menu.

Copy the ISO file (e.g. onie-recovery-x86_64-mlnx_x86-r5.0.1410.iso) to a USB drive. On Linux systems this can be done using the dd utility. On Windows, an application such as this one is needed.

Connect the USB drive to the switch, make sure the switch is configured to boot from the USB drive and reboot the switch. When the ONIE menu is presented choose the ONIE: Embed ONIE option to install ONIE on the switch.

If you are currently running Linux with kexec tools, booting into the recovery image directly from the current operating system might also be possible.

Download the ISO file to the switch, and issue the following commands:

mkdir -p /mnt/cdrom
mount -o loop onie-recovery-x86_64-mlnx_x86-r5.0.1410.iso /mnt/cdrom
kexec -l /mnt/cdrom/vmlinuz --initrd=/mnt/cdrom/initrd.xz --command-line "console=tty0 console=ttyS0,115200n8 boot_env=recovery boot_reason=embed install_url=file:///lib/onie/onie-updater"
kexec -e

This should reinstall ONIE.

It is possible to query the ASIC revision using mstflint or devlink.

To query the ASIC revision, run:

$ devlink dev info pci/0000:03:00.0
pci/0000:03:00.0:
  driver mlxsw_spectrum
  versions:
      fixed:
        hw.revision A0
        fw.psid MT_2750110033
      running:
        fw.version 13.2000.1122

Where pci/0000:03:00.0 is the PCI address of the device. In the above example the ASIC revision is A0.

This option is supported from kernel 5.1 and corresponding iproute2 version.

In Fedora it can be installed using dnf:

$ dnf install mstflint

To query the ASIC revision, run:

$ mstflint -d 01:00.0 hw query | grep HwRevId
  HwRevId               0x0

Where 01:00.0 is the PCI address of the device. In the above example the ASIC revision is A0. In case the ASIC revision is A1 the output will be:

  HwRevId               0x1