Clean Install Ubuntu 18.04 LTS on ZFS mirror as root filesystem - jbilander/HowTos GitHub Wiki
This guide will show you how to do a clean install of Ubuntu 18.04 LTS on ZFS as Root filesystem on a mirrored (Raid1) disk setup.
The system will boot from the mirror using Grub. The mirror will consist of two identical SAS-disks (same brand and size). We will also use two dedicated SSD drives for ZIL (ZFS Intent Log).
Why ZIL?
ZIL (ZFS Intent Log) drives can be added to a ZFS pool to speed up the write capabilities of any level of ZFS RAID. It writes the metadata for a file to a very fast SSD drive to increase the write throughput of the system. When the physical spindles have a moment, that data is then flushed to the spinning media and the process starts over. Significant performance increase can be achieved by adding ZIL drives to our ZFS configuration. One thing to keep in mind is that the ZIL should be mirrored to protect the speed of the ZFS system. If the ZIL is not mirrored, and the drive that is being used as the ZIL drive fails, the system will revert to writing the data directly to the disk, severely hampering performance.
As you might know Ubuntu 18.04 LTS dvd media does not come with support for installing your System directly on ZFS. As a workaround we will first install our system to a usb-stick using ext4 filesystem and then move it to the mirror using some tricks. All steps will be carefully explained.
This guide will require you to have the following:
* Server or Desktop Machine with controller in HBA/IT-mode (presents each drive individually to the host)
* Two blank disks (or SSD:s) to be used for the bootable ZFS-mirror.
* Two blank SSD for cache and log (mirrored)
* (optional) Two blank disks for backup purposes (mirrored)
* Ubuntu 18.04 LTS live-cd iso image burned to a dvd+r or dvd-r
* USB-stick with enough space for your base system installation
* Working internet connection (While installing ubuntu from live-cd)
My Setup for this guide:
* 1U SuperMicro Server SYS-1027R-WRF, 8 x 2,5" drive slots, 32GB RAM, Dual Intel Xeon E5-2600 V2 6-cores each.
* LSI SAS 9207-8i controller card (MPT2BIOS-7.39.02.00)
* Two Seagate Savvio 10k spins SAS disks
* Two Samsung 845DC Pro SSD SATA disks
* Two Seagate Constellation.2 SATA disks
* One Sandisk Ultra 32GB usb-stick
* One Samsung portable dvd-rw device
Plug your usb-stick and your usb-dvd-drive in the machine and boot from the Ubuntu media and go through the Ubuntu installation wizard...
Ok, installation done and system rebooted from the usb-stick!, now lets start installing ZFS and configure our system...
I will do this from a client using ssh, but you can do it from the console as well:
First get a sudo root shell:
jbilander@zeus:~$ sudo -s
[sudo] password for jbilander:
root@zeus:~#
Run update:
root@zeus:~# apt update
Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
151 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@zeus:~#
Run upgrade:
Please note that unattended-upgrade might be running, so let it finish or turn it off...
root@zeus:~# apt upgrade
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
Ok, lets turn unattended-upgr off,
root@zeus:~# vi /etc/apt/apt.conf.d/20auto-upgrades
Once you have the file opened, switch off the Unattended-Upgrade directive from 1 to 0 as shown below:
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "0";
And a reboot after that:
root@zeus:~# reboot
Now do the upgrade:
root@zeus:~# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
linux-headers-4.15.0-20 linux-headers-4.15.0-20-generic linux-image-4.15.0-20-generic linux-modules-4.15.0-20-generic
linux-modules-extra-4.15.0-20-generic
Use 'apt autoremove' to remove them.
The following NEW packages will be installed:
libuv1 python3-netifaces
The following packages will be upgraded:
apport apt apt-utils base-files bind9-host bsdutils cloud-init cloud-initramfs-copymods cloud-initramfs-dyn-netconf command-not-found
command-not-found-data console-setup console-setup-linux cryptsetup cryptsetup-bin dnsutils dpkg ebtables fdisk friendly-recovery
gcc-8-base grub-common grub-pc grub-pc-bin grub2-common initramfs-tools initramfs-tools-bin initramfs-tools-core keyboard-configuration
kmod landscape-common language-selector-common libapt-inst2.0 libapt-pkg5.0 libbind9-160 libblkid1 libcryptsetup12 libdns-export1100
libdns1100 libfdisk1 libgcc1 libglib2.0-0 libglib2.0-data libirs160 libisc-export169 libisc169 libisccc160 libisccfg160 libkmod2
libldap-2.4-2 libldap-common liblwres160 liblxc-common liblxc1 libmount1 libncurses5 libncursesw5 libparted2 libplymouth4
libpython3-stdlib libpython3.6 libpython3.6-minimal libpython3.6-stdlib libsmartcols1 libstdc++6 libtinfo5 libuuid1 lshw lxcfs lxd
lxd-client man-db mdadm mount ncurses-base ncurses-bin ncurses-term netcat-openbsd netplan.io networkd-dispatcher nplan open-iscsi
open-vm-tools overlayroot parted plymouth plymouth-theme-ubuntu-text pollinate python-apt-common python3 python3-apport python3-apt
python3-commandnotfound python3-distupgrade python3-gdbm python3-minimal python3-problem-report python3-software-properties
python3-update-manager python3.6 python3.6-minimal screen snapd software-properties-common sosreport squashfs-tools ssh-import-id tmux
ubuntu-keyring ubuntu-release-upgrader-core unattended-upgrades update-manager-core update-notifier-common util-linux uuid-runtime
wireless-regdb
116 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 49.2 MB of archives.
After this operation, 3,223 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
When asked about where to install Grub, choose the sandisk /dev/sdg in this case:
When asked about what to do with the 20auto-upgrades file choose "keep the local version currently installed"
Ok upgrade all done!, let's remove unneccesary packages:
root@zeus:~# apt autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
linux-headers-4.15.0-20 linux-headers-4.15.0-20-generic linux-image-4.15.0-20-generic linux-modules-4.15.0-20-generic
linux-modules-extra-4.15.0-20-generic
0 upgraded, 0 newly installed, 5 to remove and 0 not upgraded.
After this operation, 335 MB disk space will be freed.
Do you want to continue? [Y/n] Y
Ok, at last, Install ZFS support on this new system:
root@zeus:~# apt install zfs-initramfs
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libnvpair1linux libuutil1linux libzfs2linux libzpool2linux zfs-zed zfsutils-linux
Suggested packages:
nfs-kernel-server samba-common-bin
The following NEW packages will be installed:
libnvpair1linux libuutil1linux libzfs2linux libzpool2linux zfs-initramfs zfs-zed zfsutils-linux
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,190 kB of archives.
After this operation, 4,367 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Ok, we are now ready to take a backup of our system, but first let us remove the swap-file, it will only waste space on our backup archive file.
root@zeus:~# vi /etc/fstab
Comment out the swap entry:
#/swap.img none swap sw 0 0
and reboot
root@zeus:~# reboot
and delete the swap file:
root@zeus:~# cd /
root@zeus:/# rm swap.img
Ok, now boot on the Ubuntu 18.04 LTS DVD media with the usb stick still plugged in.
When presented with the following view:
Press Ctrl-Alt-F2 to get a shell:
Now, become root:
ubuntu-server@ubuntu-server:~$ sudo -s
root@ubuntu-server:~#
Create the folder sandisk and mount the usb-stick under /mnt/sandisk
root@ubuntu-server:~# cd /mnt
root@ubuntu-server:/mnt# mkdir sandisk
Check what device name the root partition got:
root@ubuntu-server:/mnt# lsblk
We can see that the root partition is named /dev/sdg2 on our system, so let's mount it:
root@ubuntu-server:/mnt# mount -t ext4 /dev/sdg2 /mnt/sandisk
Nice, we now have the root filesystem mounted under /mnt/sandisk, let's take a backup with the tar command, we will exclude the file itself from the archive:
root@ubuntu-server:/mnt# cd sandisk
root@ubuntu-server:/mnt/sandisk# tar -cvpzf backup.tar.gz --exclude=/mnt/sandisk/backup.tar.gz --one-file-system /mnt/sandisk
Now, this will take a while to finish so just wait until done...for me the created archive file became around 950 MB.
When done, let's boot up our system on the usb-stick once again.
We can now take a peek (show first few lines) in the archive file with this command:
root@zeus:~# cd /
root@zeus:/# tar -tf backup.tar.gz | head
mnt/sandisk/
mnt/sandisk/mnt/
mnt/sandisk/usr/
mnt/sandisk/usr/games/
mnt/sandisk/usr/bin/
mnt/sandisk/usr/bin/usbhid-dump
mnt/sandisk/usr/bin/calendar
mnt/sandisk/usr/bin/pslog
mnt/sandisk/usr/bin/skill
mnt/sandisk/usr/bin/grotty
We notice the mnt/sandisk folder structure so we'll have to account for that when extracting this file to disk later on.
First, find out the disk by id:
root@zeus:/# ls -al /dev/disk/by-id
total 0
drwxr-xr-x 2 root root 360 Dec 26 11:26 .
drwxr-xr-x 6 root root 120 Dec 26 11:26 ..
lrwxrwxrwx 1 root root 9 Dec 26 11:26 ata-Samsung_SSD_845DC_PRO_400GB_S1S2NWAF800045R -> ../../sdc
lrwxrwxrwx 1 root root 9 Dec 26 11:26 ata-Samsung_SSD_845DC_PRO_400GB_S1S2NWAG500511B -> ../../sdb
lrwxrwxrwx 1 root root 9 Dec 26 11:26 ata-ST9500620NS_9XF2KNT5 -> ../../sde
lrwxrwxrwx 1 root root 9 Dec 26 11:26 ata-ST9500620NS_9XF2KNWK -> ../../sdf
lrwxrwxrwx 1 root root 9 Dec 26 11:26 scsi-35000c50059545707 -> ../../sda
lrwxrwxrwx 1 root root 9 Dec 26 11:26 scsi-35000c50059d8b177 -> ../../sdd
lrwxrwxrwx 1 root root 9 Dec 26 11:26 usb-SanDisk_Ultra_20052444501D7241AE8F-0:0 -> ../../sdg
lrwxrwxrwx 1 root root 10 Dec 26 11:26 usb-SanDisk_Ultra_20052444501D7241AE8F-0:0-part1 -> ../../sdg1
lrwxrwxrwx 1 root root 10 Dec 26 11:26 usb-SanDisk_Ultra_20052444501D7241AE8F-0:0-part2 -> ../../sdg2
lrwxrwxrwx 1 root root 9 Dec 26 11:26 usb-TSSTcorp_CDDVDW_SE-S084C_SATASLIM00003000000-0:0 -> ../../sr0
lrwxrwxrwx 1 root root 9 Dec 26 11:26 wwn-0x5000c50059545707 -> ../../sda
lrwxrwxrwx 1 root root 9 Dec 26 11:26 wwn-0x5000c50059d8b177 -> ../../sdd
lrwxrwxrwx 1 root root 9 Dec 26 11:26 wwn-0x5000c500651acfcf -> ../../sdf
lrwxrwxrwx 1 root root 9 Dec 26 11:26 wwn-0x5000c500651ad32d -> ../../sde
lrwxrwxrwx 1 root root 9 Dec 26 11:26 wwn-0x5002538570000670 -> ../../sdc
lrwxrwxrwx 1 root root 9 Dec 26 11:26 wwn-0x5002538570004e63 -> ../../sdb
Okay, scsi-35000c50059545707 (/dev/sda) and scsi-35000c50059d8b177 (/dev/sdd) are the two Savvio disks.
Checking status with fdisk:
root@zeus:/dev/disk/by-id# fdisk -l /dev/sda
Disk /dev/sda: 558.9 GiB, 600127266816 bytes, 1172123568 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@zeus:/dev/disk/by-id# fdisk -l /dev/sdd
Disk /dev/sdd: 558.9 GiB, 600127266816 bytes, 1172123568 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Note if you have old partitions and want to zero out the disks you can use the dd-command like this:
root@zeus:~# dd if=/dev/zero of=/dev/sda bs=4M status=progress
root@zeus:~# dd if=/dev/zero of=/dev/sdd bs=4M status=progress
Create partitions on the first disk /dev/sda
root@zeus:/# fdisk /dev/sda
Welcome to fdisk (util-linux 2.31.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Please note! if you get something like this from fdisk:
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xe63083a7.
Just create a GPT instead:
Command (m for help): g
Created a new GPT disklabel (GUID: CF097B2B-D56F-F542-A9D1-2CFEF95602F3).
And start creating the new partitions:
Command (m for help): n
Partition number (1-128, default 1): <enter>
First sector (34-1172123534, default 2048): <enter>
Last sector, +sectors or +size{K,M,G,T,P} (2048-1172123534, default 1172123534): +2M
Created a new partition 1 of type 'Linux filesystem' and of size 2 MiB.
Command (m for help): t
Selected partition 1
Partition type (type L to list all types): 4
Changed type of partition 'Linux filesystem' to 'BIOS boot'.
Command (m for help): n
Partition number (2-128, default 2): <enter>
First sector (6144-1172123534, default 6144): <enter>
Last sector, +sectors or +size{K,M,G,T,P} (6144-1172123534, default 1172123534): <enter>
Created a new partition 2 of type 'Linux filesystem' and of size 558.9 GiB.
Command (m for help): t
Partition number (1,2, default 2): <enter>
Partition type (type L to list all types): 48
Changed type of partition 'Linux filesystem' to 'Solaris /usr & Apple ZFS'.
Command (m for help): p
Disk /dev/sda: 558.9 GiB, 600127266816 bytes, 1172123568 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: CF097B2B-D56F-F542-A9D1-2CFEF95602F3
Device Start End Sectors Size Type
/dev/sda1 2048 6143 4096 2M BIOS boot
/dev/sda2 6144 1172123534 1172117391 558.9G Solaris /usr & Apple ZFS
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
Create partitions on the second disk /dev/sdd
root@zeus:/# fdisk /dev/sdd
Welcome to fdisk (util-linux 2.31.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): n
Partition number (1-128, default 1): <enter>
First sector (34-1172123534, default 2048): <enter>
Last sector, +sectors or +size{K,M,G,T,P} (2048-1172123534, default 1172123534): +2M
Created a new partition 1 of type 'Linux filesystem' and of size 2 MiB.
Command (m for help): t
Selected partition 1
Partition type (type L to list all types): 4
Changed type of partition 'Linux filesystem' to 'BIOS boot'.
Command (m for help): n
Partition number (2-128, default 2): <enter>
First sector (6144-1172123534, default 6144): <enter>
Last sector, +sectors or +size{K,M,G,T,P} (6144-1172123534, default 1172123534): <enter>
Created a new partition 2 of type 'Linux filesystem' and of size 558.9 GiB.
Command (m for help): t
Partition number (1,2, default 2): <enter>
Partition type (type L to list all types): 48
Changed type of partition 'Linux filesystem' to 'Solaris /usr & Apple ZFS'.
Command (m for help): p
Disk /dev/sdd: 558.9 GiB, 600127266816 bytes, 1172123568 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 84326FC4-B36B-FB40-AB83-684C2089EED0
Device Start End Sectors Size Type
/dev/sdd1 2048 6143 4096 2M BIOS boot
/dev/sdd2 6144 1172123534 1172117391 558.9G Solaris /usr & Apple ZFS
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
Check the partition id's:
root@zeus:/# ls -al /dev/disk/by-id/scsi*
lrwxrwxrwx 1 root root 9 Dec 26 17:39 /dev/disk/by-id/scsi-35000c50059545707 -> ../../sda
lrwxrwxrwx 1 root root 10 Dec 26 17:39 /dev/disk/by-id/scsi-35000c50059545707-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Dec 26 17:39 /dev/disk/by-id/scsi-35000c50059545707-part2 -> ../../sda2
lrwxrwxrwx 1 root root 9 Dec 26 17:42 /dev/disk/by-id/scsi-35000c50059d8b177 -> ../../sdd
lrwxrwxrwx 1 root root 10 Dec 26 17:42 /dev/disk/by-id/scsi-35000c50059d8b177-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Dec 26 17:42 /dev/disk/by-id/scsi-35000c50059d8b177-part2 -> ../../sdd2
You can't boot from ZFS directly because your firmware doesn't know what ZFS is. So there needs to be a non-ZFS partition the firmware can boot from MBR + Boot partition (sda1/sdd1), and since this won't be protected by ZFS redundancy, it makes sense to have copies of it in many places, at least two, to provide redundancy in the event of hardware failure.
The BIOS Boot Partition is used by GRUB 2 when booting from a GUID Partition Table (GPT) disk on a BIOS-based computer. Complex boot loaders such as GRUB 2 cannot fit entirely within the confines of the MBR's 398 to 446 bytes of space, thus they need an ancillary storage space. On MBR disks, such boot loaders typically use the sectors immediately following the MBR for this storage; that space is usually known as the "MBR gap". No equivalent unused space exists on GPT disks, and the BIOS boot partition is a way to officially allocate such space for use by the boot loader.
Let's create the mirror:
root@zeus:/# zpool create -o ashift=12 rpool mirror /dev/disk/by-id/scsi-35000c50059545707-part2 /dev/disk/by-id/scsi-35000c50059d8b177-part2
And check status:
root@zeus:/# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-35000c50059545707-part2 ONLINE 0 0 0
scsi-35000c50059d8b177-part2 ONLINE 0 0 0
errors: No known data errors
Nice!
Now, Create a zfs os folder under our rpool:
root@zeus:/# zfs create rpool/os
Create a folder for our Ubuntu 18.04 LTS installation:
root@zeus:/# zfs create rpool/os/ubuntu-1804-lts
Check that everything looks alright:
root@zeus:/# df -h
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 1.7M 3.2G 1% /run
/dev/sdg2 30G 3.3G 25G 12% /
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/loop0 90M 90M 0 100% /snap/core/6130
/dev/loop1 87M 87M 0 100% /snap/core/4486
tmpfs 3.2G 0 3.2G 0% /run/user/1000
rpool 539G 128K 539G 1% /rpool
rpool/os 539G 128K 539G 1% /rpool/os
rpool/os/ubuntu-1804-lts 539G 128K 539G 1% /rpool/os/ubuntu-1804-lts
Nice, extract the tarball to the rpool/os/ubuntu-1804-lts folder, remember to strip the /mnt/sandisk structure like this:
root@zeus:/# tar xfz backup.tar.gz --strip-components=2 -C rpool/os/ubuntu-1804-lts
and verify it looks alright:
root@zeus:/# ls -al /rpool/os/ubuntu-1804-lts/
total 94
drwxr-xr-x 23 root root 27 Dec 26 19:01 .
drwxr-xr-x 3 root root 3 Dec 26 18:50 ..
drwxr-xr-x 2 root root 172 Dec 23 13:57 bin
drwxr-xr-x 3 root root 13 Dec 23 14:36 boot
drwxr-xr-x 4 root root 15 Apr 26 2018 dev
drwxr-xr-x 92 root root 178 Dec 24 13:24 etc
drwxr-xr-x 3 root root 3 Dec 16 17:16 home
lrwxrwxrwx 1 root root 33 Dec 23 13:37 initrd.img -> boot/initrd.img-4.15.0-43-generic
lrwxrwxrwx 1 root root 33 Dec 23 13:37 initrd.img.old -> boot/initrd.img-4.15.0-42-generic
drwxr-xr-x 22 root root 35 Dec 23 14:33 lib
drwxr-xr-x 2 root root 3 Apr 26 2018 lib64
drwx------ 2 root root 2 Dec 16 16:51 lost+found
drwxr-xr-x 2 root root 2 Apr 26 2018 media
drwxr-xr-x 2 root root 2 Apr 26 2018 mnt
drwxr-xr-x 2 root root 2 Apr 26 2018 opt
drwxr-xr-x 2 root root 2 Apr 24 2018 proc
drwx------ 3 root root 5 Dec 16 17:16 root
drwxr-xr-x 9 root root 12 Apr 26 2018 run
drwxr-xr-x 2 root root 234 Dec 23 14:33 sbin
drwxr-xr-x 4 root root 5 Dec 16 17:17 snap
drwxr-xr-x 2 root root 2 Apr 26 2018 srv
drwxr-xr-x 2 root root 2 Apr 24 2018 sys
drwxrwxrwt 7 root root 7 Dec 24 13:38 tmp
drwxr-xr-x 10 root root 10 Apr 26 2018 usr
drwxr-xr-x 13 root root 15 Apr 26 2018 var
lrwxrwxrwx 1 root root 30 Dec 23 13:37 vmlinuz -> boot/vmlinuz-4.15.0-43-generic
lrwxrwxrwx 1 root root 30 Dec 23 13:37 vmlinuz.old -> boot/vmlinuz-4.15.0-42-generic
root@zeus:/# cd /rpool/os/ubuntu-1804-lts/
root@zeus:/rpool/os/ubuntu-1804-lts# mount --bind /dev dev
root@zeus:/rpool/os/ubuntu-1804-lts# mount --bind /proc proc
root@zeus:/rpool/os/ubuntu-1804-lts# mount --bind /sys sys
root@zeus:/rpool/os/ubuntu-1804-lts# mount --bind /run run
root@zeus:/rpool/os/ubuntu-1804-lts# chroot .
You are now in the new system :) so lets update grub...
root@zeus:/# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.0-43-generic
Found initrd image: /boot/initrd.img-4.15.0-43-generic
Found linux image: /boot/vmlinuz-4.15.0-42-generic
Found initrd image: /boot/initrd.img-4.15.0-42-generic
done
Install grub on the /dev/sda and /dev/sdd disks using id:
root@zeus:/# grub-install /dev/disk/by-id/scsi-35000c50059545707
Installing for i386-pc platform.
Installation finished. No error reported.
root@zeus:/# grub-install /dev/disk/by-id/scsi-35000c50059d8b177
Installing for i386-pc platform.
Installation finished. No error reported.
now exit from the chroot:
root@zeus:/# exit
exit
root@zeus:/rpool/os/ubuntu-1804-lts# umount dev
root@zeus:/rpool/os/ubuntu-1804-lts# umount proc
root@zeus:/rpool/os/ubuntu-1804-lts# umount sys
root@zeus:/rpool/os/ubuntu-1804-lts# umount run
Edit fstab in rpool/os/ubuntu-1804-lts/etc/fstab, comment out the mapping of root entry
root@zeus:/rpool/os/ubuntu-1804-lts# vi etc/fstab
#UUID=e08eab7e-0152-11e9-be44-ac1f6b90eb0a / ext4 defaults 0 0
Check out the zfs mountpoints:
root@zeus:/rpool/os/ubuntu-1804-lts# cd /
root@zeus:/# zfs get all|grep mountpoint
rpool mountpoint /rpool default
rpool/os mountpoint /rpool/os default
rpool/os/ubuntu-1804-lts mountpoint /rpool/os/ubuntu-1804-lts default
Change mountpoint to /
root@zeus:/# zfs set mountpoint=/ rpool/os/ubuntu-1804-lts
cannot mount '/': directory is not empty
property may be set but unable to remount filesystem
and verify:
root@zeus:/# zfs get all|grep mountpoint
rpool mountpoint /rpool default
rpool/os mountpoint /rpool/os default
rpool/os/ubuntu-1804-lts mountpoint / local
Now export all:
root@zeus:/# zpool export -a
And shutdown the server:
root@zeus:/# shutdown -h now
Remove the usb-stick and boot from any of the Savvio disks, meaning do nothing, and it will boot from the first disk :)
Nice, the system boots fine, don't worry if you see some garbled text during the boot like below. Grub cannot reliably perform a directory listing of a zpool due to a bug, Grub can, however, read the files required to boot, so just wait...
Check the filesystems:
root@zeus:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 1.6M 3.2G 1% /run
rpool/os/ubuntu-1804-lts 539G 2.5G 537G 1% /
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/loop0 87M 87M 0 100% /snap/core/4486
/dev/loop1 90M 90M 0 100% /snap/core/6130
rpool 537G 128K 537G 1% /rpool
rpool/os 537G 128K 537G 1% /rpool/os
tmpfs 3.2G 0 3.2G 0% /run/user/1000
Check the zpool status:
root@zeus:/# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-35000c50059545707-part2 ONLINE 0 0 0
scsi-35000c50059d8b177-part2 ONLINE 0 0 0
errors: No known data errors
Some improvements/tweaks that can be made to increase ZFS performance
The atime property controls file access time tracking. This hurts performance and most applications don't need it. But it's on by default.
zfs set atime=off pool
this is said to make ZFS significantly faster:
zfs set xattr=sa pool
And compression can be added:
zfs set compression=on pool
By default, the ZIL is allocated from blocks within the main pool. However, better performance might be possible by using separate intent log devices, such as NVRAM or dedicated SSD drives.
Checking Id's for our Samsung SSD drives:
root@zeus:~# ls -al /dev/disk/by-id/ata-Sam*
lrwxrwxrwx 1 root root 9 Dec 27 14:24 /dev/disk/by-id/ata-Samsung_SSD_845DC_PRO_400GB_S1S2NWAF800045R -> ../../sdc
lrwxrwxrwx 1 root root 9 Dec 27 14:24 /dev/disk/by-id/ata-Samsung_SSD_845DC_PRO_400GB_S1S2NWAG500511B -> ../../sdb
You can set up a ZFS log device when the storage pool is created or after the pool is created. Since we already have created the main pool (rpool) we will add it afterwards like this:
root@zeus:~# zpool add rpool log mirror /dev/disk/by-id/ata-Samsung_SSD_845DC_PRO_400GB_S1S2NWAF800045R /dev/disk/by-id/ata-Samsung_SSD_845DC_PRO_400GB_S1S2NWAG500511B
Thats it! Now check the pool status:
root@zeus:~# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-35000c50059545707-part2 ONLINE 0 0 0
scsi-35000c50059d8b177-part2 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
ata-Samsung_SSD_845DC_PRO_400GB_S1S2NWAF800045R ONLINE 0 0 0
ata-Samsung_SSD_845DC_PRO_400GB_S1S2NWAG500511B ONLINE 0 0 0
errors: No known data errors
Nice!
Checking how the drives were partitioned:
root@zeus:~# fdisk -l /dev/sdb
Disk /dev/sdb: 372.6 GiB, 400088457216 bytes, 781422768 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: BD421E0A-4251-DF4C-8346-743AB1221C91
Device Start End Sectors Size Type
/dev/sdb1 2048 781406207 781404160 372.6G Solaris /usr & Apple ZFS
/dev/sdb9 781406208 781422591 16384 8M Solaris reserved 1
root@zeus:~# fdisk -l /dev/sdc
Disk /dev/sdc: 372.6 GiB, 400088457216 bytes, 781422768 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: E79703FE-12F0-994D-87AF-22B810EB2780
Device Start End Sectors Size Type
/dev/sdc1 2048 781406207 781404160 372.6G Solaris /usr & Apple ZFS
/dev/sdc9 781406208 781422591 16384 8M Solaris reserved 1
Checking id's of the disks for the backup mirror:
root@zeus:~# ls -al /dev/disk/by-id/ata-ST*
lrwxrwxrwx 1 root root 9 Dec 27 14:24 /dev/disk/by-id/ata-ST9500620NS_9XF2KNT5 -> ../../sde
lrwxrwxrwx 1 root root 9 Dec 27 14:24 /dev/disk/by-id/ata-ST9500620NS_9XF2KNWK -> ../../sdf
create the backup pool mirror, lets call it "backup"
root@zeus:/# zpool create -o ashift=12 backup mirror /dev/disk/by-id/ata-ST9500620NS_9XF2KNT5 /dev/disk/by-id/ata-ST9500620NS_9XF2KNWK
Done!, check status:
root@zeus:~# zpool status
pool: backup
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
backup ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ST9500620NS_9XF2KNT5 ONLINE 0 0 0
ata-ST9500620NS_9XF2KNWK ONLINE 0 0 0
errors: No known data errors
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-35000c50059545707-part2 ONLINE 0 0 0
scsi-35000c50059d8b177-part2 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
ata-Samsung_SSD_845DC_PRO_400GB_S1S2NWAF800045R ONLINE 0 0 0
ata-Samsung_SSD_845DC_PRO_400GB_S1S2NWAG500511B ONLINE 0 0 0
errors: No known data errors
Check how the disks were partitioned:
root@zeus:~# fdisk -l /dev/sde
Disk /dev/sde: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: BF54FB2C-2BC9-844D-9153-360ACDC75F46
Device Start End Sectors Size Type
/dev/sde1 2048 976756735 976754688 465.8G Solaris /usr & Apple ZFS
/dev/sde9 976756736 976773119 16384 8M Solaris reserved 1
root@zeus:~# fdisk -l /dev/sdf
Disk /dev/sdf: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: CB0489EF-D28D-8645-BAD8-D4AC8D587687
Device Start End Sectors Size Type
/dev/sdf1 2048 976756735 976754688 465.8G Solaris /usr & Apple ZFS
/dev/sdf9 976756736 976773119 16384 8M Solaris reserved 1
Check that our backup pool is mounted under /backup
root@zeus:/# zfs list
NAME USED AVAIL REFER MOUNTPOINT
backup 528K 449G 96K /backup
rpool 2.46G 536G 96K /rpool
rpool/os 2.46G 536G 96K /rpool/os
rpool/os/ubuntu-1804-lts 2.46G 536G 2.43G /
And that it contains nothing at the moment:
root@zeus:/# cd /backup
root@zeus:/backup# ls -al
total 9
drwxr-xr-x 2 root root 2 Dec 27 16:55 .
drwxr-xr-x 25 root root 29 Dec 27 16:44 ..
Nice!, now let's take a snapshot of our pool with the Ubuntu installation and send it to the backup:
ZFS take snapshot, we call it snap1:
root@zeus:~# zfs snapshot rpool/os/ubuntu-1804-lts@snap1
List snapshots:
root@zeus:~# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
rpool/os/ubuntu-1804-lts@snap1 2.89M - 2.42G -
Transfer it over...
root@zeus:/# zfs send -R rpool/os/ubuntu-1804-lts@snap1 | zfs receive -vF backup
receiving full stream of rpool/os/ubuntu-1804-lts@snap1 into backup@snap1
received 2.33G stream in 38 seconds (62.9M/sec)
Check the content in /backup:
root@zeus:/# cd /backup
root@zeus:/backup# ls -al
total 127
drwxr-xr-x 25 root root 29 Dec 27 16:16 .
drwxr-xr-x 25 root root 29 Dec 27 16:44 ..
drwxr-xr-x 2 root root 2 Dec 27 16:16 backup
drwxr-xr-x 2 root root 172 Dec 23 13:57 bin
drwxr-xr-x 3 root root 13 Dec 23 14:36 boot
drwxr-xr-x 4 root root 15 Apr 26 2018 dev
drwxr-xr-x 92 root root 178 Dec 26 21:43 etc
drwxr-xr-x 3 root root 3 Dec 16 17:16 home
lrwxrwxrwx 1 root root 33 Dec 23 13:37 initrd.img -> boot/initrd.img-4.15.0-43-generic
lrwxrwxrwx 1 root root 33 Dec 23 13:37 initrd.img.old -> boot/initrd.img-4.15.0-42-generic
drwxr-xr-x 22 root root 35 Dec 23 14:33 lib
drwxr-xr-x 2 root root 3 Apr 26 2018 lib64
drwx------ 2 root root 2 Dec 16 16:51 lost+found
drwxr-xr-x 2 root root 2 Apr 26 2018 media
drwxr-xr-x 2 root root 2 Apr 26 2018 mnt
drwxr-xr-x 2 root root 2 Apr 26 2018 opt
drwxr-xr-x 2 root root 2 Apr 24 2018 proc
drwx------ 3 root root 5 Dec 16 17:16 root
drwxr-xr-x 2 root root 2 Dec 26 20:16 rpool
drwxr-xr-x 9 root root 12 Apr 26 2018 run
drwxr-xr-x 2 root root 234 Dec 23 14:33 sbin
drwxr-xr-x 4 root root 5 Dec 16 17:17 snap
drwxr-xr-x 2 root root 2 Apr 26 2018 srv
drwxr-xr-x 2 root root 2 Apr 24 2018 sys
drwxrwxrwt 9 root root 9 Dec 27 14:54 tmp
drwxr-xr-x 10 root root 10 Apr 26 2018 usr
drwxr-xr-x 13 root root 15 Apr 26 2018 var
lrwxrwxrwx 1 root root 30 Dec 23 13:37 vmlinuz -> boot/vmlinuz-4.15.0-43-generic
lrwxrwxrwx 1 root root 30 Dec 23 13:37 vmlinuz.old -> boot/vmlinuz-4.15.0-42-generic
Nice! Looks like a backup of our system :) Now verify the used space matches:
Checking the filesystems we can see that we got a nice backup 2.5GB used (same as rpool/os/ubuntu-1804-lts) in the backup pool.
root@zeus:/backup# df -h
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 1.7M 3.2G 1% /run
rpool/os/ubuntu-1804-lts 539G 2.5G 537G 1% /
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/loop1 90M 90M 0 100% /snap/core/6130
/dev/loop0 87M 87M 0 100% /snap/core/4486
rpool 537G 128K 537G 1% /rpool
rpool/os 537G 128K 537G 1% /rpool/os
tmpfs 3.2G 0 3.2G 0% /run/user/1000
backup 450G 2.5G 448G 1% /backup
Looking at the snapshots we can see a backup@snap1 was created, we can delete all snapshots now if we like:
root@zeus:/backup# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
backup@snap1 336K - 2.42G -
rpool/os/ubuntu-1804-lts@snap1 22.1M - 2.42G -
root@zeus:/backup# zfs destroy backup@snap1
root@zeus:/backup# zfs destroy rpool/os/ubuntu-1804-lts@snap1
root@zeus:/backup# zfs list -t snapshot
no datasets available
Taking snapshots is a good way to handle system upgrades and/or release management:
1. During a maintenance window, shutdown services like apache, mysql etc..
2. Take a snapshot before altering anything on the system
3. Do system upgrade with "apt upgrade" and/or do other software upgrades
4. If anything went severely wrong do a rollback
root@zeus:/# zfs rollback rpool/os/ubuntu-1804-lts@snap1
5. If all went well, destroy the snapshot, we don't need it any more, it will only take up resources since the delta is stored.
root@zeus:/# zfs destroy rpool/os/ubuntu-1804-lts@snap1
Okay, That's it, Enjoy your system running on ZFS :)
Now it's time for configuring your system, let's start with setting the timezone :)
How-to-change-timezone-on-Ubuntu-18.04-LTS
Example Maintenance Window
Turn services off (mysql, apache, etc...) then:
root@zeus:~# zfs list -t snapshot
no datasets available
root@zeus:~# zfs snapshot rpool/os/ubuntu-1804-lts@snap1
root@zeus:~# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
rpool/os/ubuntu-1804-lts@snap1 0B - 2.47G -
root@zeus:~# apt update
root@zeus:~# apt upgrade
If new kernel was installed, make sure you make new grub-install to both MBRs om mirror:
root@zeus:~# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.0-43-generic
Found initrd image: /boot/initrd.img-4.15.0-43-generic
Found linux image: /boot/vmlinuz-4.15.0-42-generic
Found initrd image: /boot/initrd.img-4.15.0-42-generic
done
root@zeus:~# grub-install /dev/disk/by-id/scsi-35000c50059545707
Installing for i386-pc platform.
Installation finished. No error reported.
root@zeus:~# grub-install /dev/disk/by-id/scsi-35000c50059d8b177
Installing for i386-pc platform.
Installation finished. No error reported.
All went well, do a reboot:
root@zeus:~# reboot
root@zeus:~# uname -a
Linux zeus 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
destroy snapshot:
root@zeus:~# zfs destroy rpool/os/ubuntu-1804-lts@snap1