OSVDC Series: Storage Host Build with FreeNAS 9.3 - rharmonson/richtech GitHub Wiki

OSVDC Series: Storage Host Build with FreeNAS 9.3

Revised: July 27, 2016; correct formatting uses with '#' Revised: April 4, 2016

Article 6 of the Open Source Virtual Data Center Series

Prerequisites

An internet connection and connectivity from a workstation to the Storage host is necessary to complete the steps in this article.

Assemble Hardware

If you have no or little experience in assembling computer components, please review the following for help. Youtube is an excellent resource, too. Prevention of ESD is very important!

Update System Board BIOS (firmware)

Per Supermicro's website, flashing or upgrading the system board voids the warranty. Sounds like bull**** to me but consider yourself warned. If you do not have the most current BIOS, download the BIOS update here:

http://www.supermicro.com/products/motherboard/Xeon/C220/X10SL7-F.cfm

  • Select "Update Your BIOS"

After download, open the archive and read and follow the instructions carefully.

I updated with x10sl75.C17 which results with the BIOS displaying version 3.0 build date 04/24/2015.

Note

If using solid state drives, update the BIOS > Advance > SATA for each SSD from Hard Disk Drive to Solid State Drive. Is this important? No idea, but disclosing the settings I used.

Update "BMC/IPMI Firmware"

Update the BMC/IPMI firmware if you do not have the current by downloading the archive from here:

http://www.supermicro.com/products/motherboard/Xeon/C220/X10SL7-F.cfm

  • Select "BMC/IPMI Firmware"

After download, open the archive and read and follow the instructions carefully.

I am using IPMI version 1.92.

While you are waiting for the firmware to be updated, consider reading Best Practices BMC Security.

Update LSI (Avago) 2308 Storage Controller Firmware

With FreeNAS 9.3, you will want to flash the firmware from IR mode (RAID) to IT mode (JBOD). FreeNAS will complain if it is not version 20. Download the firmware and read the instructions within the archive found here:

ftp://ftp.supermicro.com/driver/SAS/LSI/2308/Firmware/IT

Before proceeding, review Supermicro's FAQ "How do I upgrade firmware for onboard LSI 2308 in X10SL7-F?" found here:

http://www.supermicro.com/support/faqs/faq.cfm?faq=19429

I am using Avago MPT2BIOS-7.39 which is dated 2014.09.18.

Update Chelsio

Drivers, ROMs, and documentation can be found at Chelsio Support website.

I am using the DOS download version 1.0.0.85, dated 07/07/2015, and with the file name Chelsio-Uboot-1.0.0.85.zip which uses the firmware file t5fw-1.13.32.0.bin. Results with boot ver. 1.0.0.85, firmware ver. 1.13.32.0, and config ver. 1.0.0.85.

Since the flash is more convoluted than most, I am providing the instructions as found in the doc/README.txt file. Please review before flashing to verify nothing has changed.

I. Preparing USB flash drive
-----------------------------

This document assumes that you are using an USB flash drive as a storage 
media for the necessary files. Follow the steps below to prepare the drive:

a. Create a DOS bootable USB flash drive.
b. Create a directory "CHELSIO" on USB flash drive.
c. If you haven't done already, download Chelsio-Uboot-x.x.x.xx.zip from Chelsio
   Download Center, service.chelsio.com
d. Unzip the downloaded package and change your working directory to "OptionROM"
   directory. 
e. Copy all the files and place them in the CHELSIO directory created on the USB 
   flash drive.
f. Plug-in the USB flash drive in the system on which the Chelsio CNA is 
   installed.
g. Reboot the system and enter the system BIOS setup.
h. Make the USB flash drive as the primary boot device. 
i. Save the changes and exit.
   
II. Legacy 
----------

a. Configure the system having Chelsio CNA to boot in Legacy mode. 

b. Once the system boots from the USB flash drive, change your working directory
   to CHELSIO directory:

   C:\>cd CHELSIO

c. Run the following command to list all Chelsio CNAs present in the 
   system. The list displays a unique index for each CNA found.

   C:\CHELSIO>cfut4 -l

d. Delete any previous version of Option ROM flashed onto the CNA:

   C:\CHELSIO>cfut4 -d <idx> -xb 
   
Here, idx is the CNA index found in step (c) 

e. Delete any previous firmware using the following command:

   C:\CHELSIO>cfut4 -d <idx> -xh -xf 
  
f. Delete any previous Option ROM settings:

   C:\CHELSIO>cfut4 -d <idx> -xc

g. Run the following command to flash the appropriate firmware 
   (t5fw-x.xx.xx.x.bin for T5 adapters;t4fw-x.xx.xx.x.bin for T4 adapters). 

   C:\CHELSIO>cfut4 -d <idx> -uf <firmware_file>.bin
   
h. Flash the Option ROM onto the Chelsio CNA using the following command:

   C:\CHELSIO>cfut4 -d <idx> -ub cubt4.bin  

Here, "cubt4.bin" is the unified option ROM image file present in the CHELSIO 
directory.   
 
i. Flash the default boot configuration file.  

   C:\CHELSIO>cfut4 -d <idx> -uc bootcfg

j. Reboot the system for changes to take effect.

Burn-in

It is a good idea to leave the assembled system running for a day or two to burn-in. You can, also, use software to stress test the system but I have not found a suitable solution that does not require installing an operating system first. At a later date I may explore Phoronix Test Suite. It looks to have potential.

Obtain FreeNAS

I am using the FreeNAS 9.3 Stable release dated 201602031011. The most recent installation packages and SHA256 checksums are found here:

http://download.freenas.org/latest/x64/

FreeNAS has some of the better documentation for an Open Source project. The FreeNAS 9.3 User Guide is found here:

http://doc.freenas.org/9.3/freenas.html

Install FreeNAS

Installation (and upgrade) instructions and details can be found here:

http://doc.freenas.org/9.3/freenas_install.html

You can install FreeNAS using a CD, USB flash drive, or Supermicro's BMC/IPMI.

BMC/IPMI

Using the Intelligent Platform Management Interface (IPMI) is not the easiest nor fastest method to install an operating system. However, once it is setup and working, IPMI, provides out-of-band management via redirected consoles (KVM) and devices even when the operating system faults or services like SSH or RDP are unavailable.

Configure IPMI using BIOS:

  1. Connect an ethernet cable to the dedicated network interface
  2. Power-on or reboot the system
  3. Select the DEL key at the Supermicro splash screen to enter the system BIOS
  4. Select the "IPMI" menu
  5. Select "BMC Network Configuration"
  6. Select "Update IPMI LAN Configuration" to toggle No to Yes
  7. Enter the IP address 192.168.101.1
  8. Enter the subnet mask 255.255.255.0
  9. Enter the default gateway 192.168.101.254
  10. Select F4 to save and reboot

IPMI User Manual

ftp://ftp.supermicro.com/utility/IPMIView/IPMIView20.pdf

Using IPMI's Remote Control requires the use of a share to mount an installation ISO.

To complete an operating system installation using IPMI:

  1. Login to IPMI; https://ip_address, e.g. 192.168.101.1, using user ADMIN, password ADMIN. Case sensitive!
  2. Mount installation image; select "Virtual Media," "CD-ROM Image" and specify host, path to image, credentials, then click "Mount" (See Example below)
  3. Open remote control; select "Remote Control," "Console Redirection", then "Launch Console" (Java)
  4. Restart server; select "Remote Control," "Power Control," then "Power Cycle Server"
  5. At the Supermicro splash screen, select F11 to display boot device menu"
  6. Select virtual CDROM (ATEN Virtual CDROM YS0J)
  7. Install operating system
  8. Umount installation image; select "Virtual Media," "CD-ROM Image," then click "Unmount"
  9. Complete!

Virtual Media Example

Assumption: You are using a windows workstation. If so, you are not permitted to browse to an ISO on the local file system but must use a CIFS or SMB share. Odd. Using windows, you can simply select a folder, right mouse click, and select "Share with." In the example below, I am using a CIFS share that permits guest read access. Otherwise, you will need to provide user credentials with the appropriate access.

  • Share host: 192.168.22.10
  • Path to image: \cifsshare\instmedia\isos\freenas\freenas93.iso
  • User: guest
  • Password: (blank)

Security Consideration

Access to BMC/IPMI like all privileged services should be carefully restricted. Firewall dependencies and best practices per Supermicro for BMC/IPMI are found here:

http://www.supermicro.com/products/nfo/files/IPMI/Best_Practices_BMC_Security.pdf

FreeNAS Configuration

Connectivity

Using a Single Management Interface

If using DHCP, the interface should be connected and ready. The IP address the web service is utilizing will be shown on the console after installation. If not, skip to the heading titled "Configure IPv4" below and use igb0 or your desired interface not the LAGG interface.

Using a LAGG Management Interface

To use two or more interfaces, use link aggregation (LAGG). The LAGG can be configured to use a number of protocols but I would advise using failover or LACP. My preference is LACP when using a switch but prior to connecting both Ethernet cables, you will need to configure LACP for both the Storage host and the network switch--Ubiquiti enables LACP by default for all ports. When in doubt, begin with only one Ethernet cable to the network switch.

Create LAGG

  1. Enter 2, "Configure Link Aggregation"
  2. Enter 1, "Create Link Aggregation"
  3. Enter 3, "lacp"
  4. Enter 1, interface "igb0"
  5. Enter 1, interface "igb1"
  6. Enter q to quit

Configure IPv4

  1. Enter 1, "Configure Network Interfaces"
  2. Enter the number for "lagg0"
  3. Enter n to "Delete interface?"
  4. Enter n to "Reset network configuration?"
  5. Enter n to "Configure interface to DHCP?"
  6. Enter y to "Configure IPv4?"
  7. Enter using the default interface name lagg0 or use "mgmt"
  8. Enter 192.168.101.10/24 for the IPv4 Address
  9. Enter n to "Configure IPV6?"

Configure Default Route

  1. Enter 4, "Configure Default Routes"
  2. Enter y for "Configure IPv4 Default Route?"
  3. Enter IP address for the default route, 192.168.1.254
  4. Enter n to "Configure IPV6 Default Route?"

Configure DNS

  1. Enter 6 to "Configure DNS"
  2. Accept the default 'local' domain or enter your own like 'mylab.net'
  3. Enter 8.8.8.8 for "DNS Nameserver 1"
  4. Enter 8.8.4.4 for "DNS Nameserver 2"
  5. Enter nothing or blank for "DNS Nameserver 3" to quit

Storage Interfaces

Unfortunately, LAGG requires 802.3ad support which implies the use of a switch. I am not using a 10 Gb switch for the storage networks. As a consequence, I will use a single physical interface for each storage network. For production environments, buy a 10 Gb switch to utilize redundant storage network links. Better yet, purchase two 10 Gb switches for an additional level of redundancy.

Configure IPv4

  1. Enter 1, "Configure Network Interfaces"
  2. Enter the number for "cxl0" (Chelsio 10 Gb interface)
  3. Enter n to "Delete interface?"
  4. Enter n to "Reset network configuration?"
  5. Enter n to "Configure interface to DHCP?"
  6. Enter y to "Configure IPv4?"
  7. Enter using the default interface name or use "stor1"
  8. Enter 192.168.111.1/30 for the IPv4 Address
  9. Enter n to "Configure IPV6?"
  10. Repeat the steps to configure cxl2 using name "stor2" and IP 192.168.112.1/30

Reboot

  1. Enter 13 then y to reboot.

Upgrade

Using a workstation and a browser

  1. Enter the IP address of the Storage host as shown on the console, e.g. http://192.168.101.10
  2. Enter root and the password you set during installation
  3. Close the configuration wizard if it appears
  4. Select "System"
  5. Select "Update"
  6. Select "Check Now"
  7. Select "OK" to download and apply updates
  8. Select "Reboot" on the left-side menu to restart
  9. Complete!

General Configuration

The headings below correlate to menu and submenu selections.

Account --> Group, User

Create a new group with GID 36 called kvm. Create users for administration with sudo access, then an oVirt management account with UID 36 called vdsm and "Primary Group" kvm.

System --> General

By default, the "WebGUI IPv4 Address" is 0.0.0.0 which denotes listening on all IP address. Update for only the management interface IP address. Also, verify the correct timezone is selected.

Network --> Global Configuration

Update with the desired host name and domain then select "Save."

Note:

"Host name data base" is where you enter an entry per row to add to /etc/hosts if name resolution is required without DNS.

SLOG Device Preparation

Using a SLOG device or resizing the SLOG device is optional. Search the FreeNAS forum if you are curious why resizing is a consideration.

Prior to creating our ZFS pool, we will prepare our SLOG device. The SLOG device size should be either one-half RAM or only as large as two workloads. We don't know the latter, thus we will use 16 GB which is one-half of the installed 32 GB RAM.

Using either the console at the Storage host or selecting "Shell" from FreeNAS' WebGUI, execute the following:

  1. Enter "camcontrol devlist" to identify the SLOG device. Mine is "ada0."
  2. Enter "camcontrol hpa ada0" to verify HPA - Security is no, otherwise, execute "camcontrol security ada0 -e" to disable.
  3. Enter "camcontrol hpa ada0 -P -s 31250000" to resize to 16 GB.
  4. Enter "camcontrol hap ada0" to verify the change to 31250000.

Exit the shell then reboot for the changes to be picked up by FreeNAS.

Results after reboot freenas-viewdisks

Create Volume

Time to create our array of mirrors (RAID10) with one log device. The general process is to create one mirror and save. Select the extent and add another mirror pair. Repeat until all disk pairs are added and the SLOG device.

  1. Select Storage, Volumes, then Volume Manager
  2. Enter the "Volume Name" of r10vol or whatever you like but keep it to alpha-numerical
  3. Select and drag the "Volume layout" slider for two disks, da0 and da1.
  4. Verify "Mirror"
  5. Select "Add Volume" and wait..
  6. Select "Volume Manager"
  7. Select "Volume to extend" and select r10vol
  8. Select and drag the "Volume layout" slider for two disks, da2 and da3, which should default to "Mirror"
  9. Select "Extend Volume"
  10. Select "Volume Manager"
  11. Select "Volume to extend" and select r10vol
  12. Select and drag the "Volume layout" slider for two disks, da4 and da5, which should default to "Mirror"
  13. Select "Extend Volume"
  14. Select "Volume Manager"
  15. Select "Volume to extend" and select r10vol
  16. Select and drag the "Volume layout" slider for two disks, da6 and da7, which should default to "Mirror"
  17. Select "Add Extra Device"
  18. Select and drag the second "Volume layout" for one disk, ada0, and change to "Log (ZIL)" aka SLOG.
  19. Select "Extend Volume"

Results with freenas-volume

Select the volume r10vol then in the lower-left of the Volumes pane select the icon that looks like a page that displays "Volume Status" on mouse-hover. The resulting display will show a more complete picture of the volume and disks.

Results freenas-volumestatus

Create Datasets

To create separate filesystems, FreeNAS uses datasets. There are two required datasets for oVirt, data and iso, and eng. The storage domain data is used to store virtual machines, the iso storage domain is where installation media resides, and the eng dataset is for oVirt Appliance.

  1. Select Storage, Volumes, View Volumes
  2. Select r10vol (second listed)
  3. Select the second icon from the right that displays "Create Dataset" on mouse-hover
  4. For "Dataset Name: enter "data'
  5. Select "Add Dataset"
  6. Select r10vol
  7. Select "Create Dataset"
  8. For "Dataset Name: enter "iso'
  9. Select "Add Dataset"
  10. Select r10vol
  11. Select "Create Dataset"
  12. For "Dataset Name: enter "eng'
  13. Select "Add Dataset"

Results with freenas-datasets

The temp dataset was created for an unrelated project. You may not have a jails dataset.

Note:

You may select "Advance" or "Edit Options" to enable quotas, reservations, and deduplication. There is a potential performance penalty with the last.

File System Permissions

oVirt user vdsm and group kvm but have permissions for the file system and shares. To grant privileges, use the console or shell from the WebGUI to execute the following:

  1. cd /mnt/r10vol
  2. Use "ls -l" to verify you see data, eng, and iso with owner root
  3. chown 36:36 data
  4. chown 36:36 eng
  5. chown 36:36 iso
  6. Use "ls -l" to verify owner and group now read vdsm and kvm
  7. chmod 0755 data eng iso

The last step should be unnecessary since it is the default.

Create NFS Shares

Time to create NFS shares by executing:

  1. Select "Sharing"
  2. Select "UNIX (NFS)"
  3. Select "Add Unix (NFS) Share"
  4. Select "Advanced Mode"
  5. Select "Browse"
  6. Select /mnt/r10vol/data
  7. For "Comment" enter "Data Storage Domain"
  8. For "Authorized networks: enter "192.168.111.0/30" then new line and "192.168.112.0/30"
  9. For "Mappall User" select user vdsm
  10. For "Mappall Group" select user kvm
  11. Select "OK"
  12. When promoted to "Enable service," select "No"
  13. Select "Add Unix (NFS) Share"
  14. Select "Advanced Mode"
  15. Select "Browse"
  16. Select /mnt/r10vol/eng
  17. For "Comment" enter "Engine Appliance"
  18. For "Authorized networks: enter "192.168.111.0/30" then new line and "192.168.112.0/30"
  19. For "Mappall User" select user vdsm
  20. For "Mappall Group" select user kvm
  21. Select "OK"
  22. When promoted to "Enable service," select "No"
  23. Select "Add Unix (NFS) Share"
  24. Select "Advanced Mode"
  25. Select "Browse"
  26. Select /mnt/r10vol/iso
  27. For "Comment" enter "ISO Storage Domain"
  28. For "Authorized networks: enter "192.168.111.0/30" then new line and "192.168.112.0/30"
  29. For "Mappall User" select user vdsm
  30. For "Mappall Group" select user kvm
  31. Select "OK"
  32. When promoted to "Enable service," select "No"

Results with freenas-nfsshares

NFS Service

NFS version 4 has performance benefits over version 3 but does add complexity. Either version of the protocol are supported by oVirt. I will be using NFSv4.

To configure and start the NFS service, complete the following tasks:

  1. Select "Services"
  2. Select "NFS"
  3. Select the "Bind IP Addresses" 192.168.111.1 and 192.168.112.1
  4. Select "Enable NFSv4"
  5. Select "OK"
  6. Start the NFS service by select the ON/OFF graphic
  7. Complete!

Reference:

An excellent resource for trouble shooting NFS shares used by oVirt can be found here:

http://www.ovirt.org/documentation/how-to/troubleshooting/troubleshooting-nfs-storage-issues/

Backup FreeNAS Configuration

Time to backup our current configuration.

  1. Select "System"
  2. Select "General"
  3. Select "Save Config"
  4. Provide a backup location
  5. Done!

Other Considerations

I have glossed over a number of useful FreeNAS settings and features that I would advise investigating. The FreeNAS forums and the Guide found in the WebGUI are excellent resources.

  • Use of HTTPS and redirecting HTTP: System --> General (See CAs and certificates)
  • Syslog: System --> General
  • Email: System --> Email
  • CAs (Certificate Authority): System --> CAs
  • Certificates: System --> Certificates
  • SMART tests: Tasks --> S.M.A.R.T. Tests
  • Scrub: Storage --> Scrubs
  • Directory Service --> Active Directory or LDAP or Kerberos*
  • Reporting

Next

Next article in the series is Compute Host Build with CentOS 7 1511.

⚠️ **GitHub.com Fallback** ⚠️