Ubuntu ZFS mountall FAQ and troubleshooting - atomjack/zfs GitHub Wiki

The Ubuntu PPA for ZoL has an enhanced /sbin/mountall that recognizes ZFS and can mount ZFS at system start.

1.0 Installation

1.1 Installing the mountall package

Enable the PPA and install the ubuntu-zfs package. Do not do anything else.

The ubuntu-zfs package installs a pin file that is required to keep the right mountall package installed.

1.2 Verifying the mountall package

Run apt-cache policy mountall to check that the PPA is the preferred archive for the mountall package. The installed package should have "zfs" in the version string and there should be a line that begins with 900 in the output.

$ apt-cache policy mountall
mountall:
  Installed: 2.36-zfs1
  Candidate: 2.36-zfs1
  Version table:
 *** 2.36-zfs1 0
        900 http://ppa.launchpad.net/zfs-native/daily/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status
     2.36 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

Also check that the /sbin/mountall binary has a parse_zfs_list function:

$ grep parse_zfs_list /sbin/mountall
Binary file /sbin/mountall matches

1.3 Intended mountall operation

How is ZoL supposed to work at system start?

  1. All storage pool devices come online and appear in /dev/disk/by-id.
  2. Upstart runs the /etc/init/mountall.conf part.
  3. The /sbin/mountall utility calls the /sbin/zfs list command.
  4. The /sbin/zfs utility inserts the zfs.ko kernel module.
  5. The ZFS driver reads the /etc/zfs/zpool.cache file, which contains a list of disks in each zpool. Any disk that is not ready during this step faults out of the pool.
  6. The /sbin/mountall utility mounts all ZFS filesystems and all /etc/fstab entries during the same pass.
  7. The /sbin/mountall utility issues the local-filesystems event for Upstart.

2.0 Usage

2.1 Mount Points

By default, the mountpoint filesystem property determines mount points. For example:

# zfs get mountpoint tank/media
NAME           PROPERTY    VALUE           SOURCE
tank/media     mountpoint  /tank/media     default

# zfs get mountpoint tank/home
NAME           PROPERTY    VALUE           SOURCE
tank/home      mountpoint  /home           local

Set the mountpoint property to change the mount point:

# zfs set mountpoint=/export/media tank/media

Changes happen at reboot if the mount point is busy.

2.2 /etc/fstab

Most systems should use the mountpoint filesystem property, but the /etc/fstab file can still be used as an alternative. The mountpoint property must be set to legacy on each filesystem before it is added to the /etc/fstab file like this:

# zfs set mountpoint=legacy tank/media

The filesystem name in the /etc/fstab must not have a leading slash character:

tank/media  /export/media  zfs  defaults  0  0

3.0 Frequent Mistakes

  • Do not put the ZFS driver in the initrd.
  • Do not export pools at reboot or shutdown.
  • Do not set ZFS_MOUNTALL=y in the /etc/default/zfs file.
  • Do not install the mountall package on a vanilla Debian system.
  • Do not put a zfs line in the /etc/modules file or otherwise load it earlier than upstart calls the mountall part.
  • Do not delete the /etc/hostid file. It must exist.

And

  • Never manually compile ZoL and run make install on a system that is using the PPA. A local installation is incompatible with a managed installation.

Things should just work if the mountall package for ZoL is installed from the PPA. Please send a bug report otherwise.

Most startup problems are caused by:

  • Storage devices coming online after the /etc/zfs/zpool.cache file is checked.
  • ZFS mounting after the local-filesystems upstart event is emitted.

4.0 Known Problems

4.1 /tmp mount point

The mountall utility fails if a mountpoint property for a ZFS filesystem is the same as any mount point described in the /lib/init/fstab override file. On a default system, /tmp is the only likely collision.

If /tmp must be a separate filesystem, then set mountpoint=legacy and put a line for it in the /etc/fstab file. This bug does not happen if /tmp is in a ZFS root filesystem.

Changing other mount points described in the /lib/init/fstab file are likely to cause general system failures, even on non-ZoL systems.

4.2 Read-only Mounts

The "ro" mount flag is not always honored.

4.3 JBOD mode on hardware RAID implementations

Some HBAs bring JBOD disks online asynchronously at power-on. Pool import fails if JBOD disks appear after the ZFS driver loads and checks the /etc/zfs/zpool.cache for the list of zpool members. The failure can be sporadic and happens most frequently on computers that use the mpt2sas driver.

Try these fixes in order:

  1. Upgrade the HBA firmware.
  2. Disable JBOD and export each disk as a single-element RAID-0 or RAID-1 volume.
  3. Put a sleep command in the /etc/init/mountall.conf file like this:
...
sleep 60
exec mountall --daemon $force_fsck $fsck_fix
...

This means "wait sixty seconds before running mountall". Tune the number up or down according to how much time it takes for the HBA to bring JBOD disks online.

4.4 eSATA, USB, Firewire, Thunderbolt, etc

ZoL does not automatically import pools on storage devices that are configured through hotplug events, particularly USB drives. All storage devices must be "fixed disks" or otherwise fully configured during system start before the ZFS driver is loaded.

4.5 VMware

  • VMware ESX 5 issues late hotplug events on its mpt2sas implementation, which is incompatible with ZoL. (Virtual Box had a nearly identical bug that was fixed upstream.)

  • If the /dev/disk/by-id aliases are missing in a VMware guest, then set this option in the VMX configuration file or at the vSphere configuration panel:

disk.EnableUUID = "TRUE"

Afterwards, export and import all pools to refresh the /etc/zfs/zpool.cache file.

5.0 Troubleshooting

First, refresh the /etc/zfs/zpool.cache file:

  • Export all pools.
  • Delete the /etc/zfs/zpool.cache. file.
  • Import all pools.

Second, double-check that:

  • All mount points are empty in the underlying filesystem. This requirement is inherited from Solaris.
  • If the system has selinux or sudo customizations, then double check that the /sbin/mountall and /sbin/zfs commands are always invoked with full root privileges.
  • All filesystems in the /etc/fstab file have mountpoint=legacy set.

Third, run mountall --verbose as the root user. If this command succeeds, but automatic mounting fails during system start, then check for incompatibilities described in section 4.3 or 4.4 of this document.

Also run mountall --debug as the root user.

6.0 Bug Reporting

Please include these things with a bug report:

  • The whole mountall --verbose transcript.
  • The unmodified /var/log/kern.log and /etc/fstab files.
  • The zpool status and zfs get all for affected filesystems.
  • The full raw output of the dmesg command.