Introduction: On good and and bad luck

... and likely some bad practise: Sometimes bad and good luck come together bringing both things together. I had my FreeNAS Mini box humming on 11.1-U1 and ... like I regularly do after waiting a couple of days after a release: Update the OS not expecting any major issues.

On major OS updates I exported the configs but on minor version bumps, and since I don't regularly change a lot of options on my FreeNAS I didn't always export the configuration. Luckily FreeNAS 11.x introduced an additional message if I wanted to export the config before updating which I accepted.

This time however the box didn't come up and after 10' I connected to the IP KVM of the AsRock board and saw the following message:

failed to read pad2 area of primary vdev
ZFS: i/o error -  all block copies copies unavailable
ZFS can't read MOS of pool tank
gptzfsboot: failed to mount default pool tank

A quick search revealed that in many cases who have encountered this message, they had to look for a restore. Lucky me: I had exported the configuration this time so I guess things should be easy and I didn't bother digging down the rabbit hole of a full restore since I also had a spare SSD.

Restoring - and falling into another trap

Since I didn't have the time to test if the Apacer SATA DOM was at actual fault and I wanted to have access to my data again, I first replaced it with a known-good used 120 GB SATA SSDs and booted the latest 11.2-U2 installer disc over KVM remote media and booted selecting UEFI mode.

It's been years since I last saw the actual FreeNAS installer but the setup is pretty straightforward: Select a disk, provide a root password and decide whether to boot configure UEFI boot or legacy but known MBR boot.

Since with FreeBSD 11 UEFI support has improved and since I have started to only use BIOS mode whenever a system doesn't support UEFI (or where early UEFI firmware were buggy) thus I selected UEFI because - why not?

Althought the motherboard wasn't configured with CSM only boot, it didn't want to boot from the new system drive after the otherwise successful install. Then it dawned me that the loader wasn't detected by the UEFI firmware since all I saw was either the legacy boot drives or the EFI-bootable install disc.

Booting the OS temporarily

Dropping into EFI shell showed that FS0:, the EFI service partition (ESP) was properly discovered but the Asrock UEFI firmware didn't like how the bootloader was named. FreeBSD and hence FreeNAS put the EFI loader into boot/efi/BOOTx64.efi

Manually selecting ESP and loading the bootloader started FreeNAS without any issue so I whent ahead and restored the configuration asking for another reboot (hence repeating the manual boot selection) doing the following in the EFI shell:

FS0:
cd boot\efi
BOOTx64.efi

The good part: After booting and waiting for 10 minutes the system configuration was restored and things I needed such as shares were accessible again. Definitely a moment of relief even thogh I'd have had backups.

However manually selecting the bootloader at a reboot or after updates wasn't going to cut it in the future.

Fixing bootup: Adding a boot variable using EFI shell

Unfortunately some UEFI vendors don't always search for all possible and legit bootloaders available. Some vendors only look for EFI\Microsoft\Boot\bootmgr.efi for internal drives (as seen in FreeNAS but #16280). I also remembered some similar issues with virtual machines using the (based on TianoCore) OVMF UEFI firmware for Virtual Machines on Linux KVM some years ago.

Unlike Debian 9 for what I can tell, the FreeNAS installer does not configure a EFI variable which it the UEFI firmware could use. The toold that would do this is on Linux and the same command is part of FreeBSD. efibootmgr is also part of FreeNAS 11.2, however as of writing efirt.ko isn't shipped with this release, hence adding a new boot variable wasn't going to work from the OS.

From what it seems FreeBSD 11.2 and 12.0 were the first releases to ship efirt.ko by default. FreeNAS 11.2 is based on FreeBSD 11.2-STABLE, therefore I imagine some configurations are currently not building the necessary kernel module on FreeNAS. UPDATE: efirt.ko should be available with something after FreeNAS 11.2-U2, it was simply not enabled in FreeNAS builds. Once that module is available, you should be able to avoid booting into an EFI shell.

No worries, EFI shell - which I have never used beyond manually executing an EFI loader - should allow me to do this with the bcfg command. (See: Arch Linux Wiki: Unified Extensible Firmware Interface)

However in its infinite wisdom Asrock's has decided to limit/castrate its EFI shell to not include this command. Thankfully the TianoCore edk2 repository contains pre-built EFI shell binaries for x86-64 to load separately from Github.

Since I didn't want to plug a USB drive on it I, temporarily mounted the ESP partition on the boot and dropped the edk2 EFI shell binary onto it:

# camcontrol devlist | grep INTEL
<INTEL SSDSC2CW120A3 400i>         at scbus14 target 0 lun 0 (ada4,pass5)

# gpart show /dev/ada4
=>       40  234441568  ada4  GPT  (112G)
         40     532480     1  efi  (260M)
     532520  233897984     2  freebsd-zfs  (112G)
  234430504      11104        - free -  (5.4M)

mkdir /mnt/efi
mount_msdofs /dev/ada4p1 /mnt/efi
cd /mnt/efi
fetch https://<url-to-latest-edk2-efi-shell-binary>

Afterwards I rebooted into the AsRock EFI shell and from there I started the edk2 EFI shell:

FS0:
cd boot\efi
Shell.efi

Now with the help of the Arch Linux Wiki I was able to construct the bcfg command required:

# Show current entries
bcfg boot dump

# Since no 3rd option is present, add one
bcfg boot add 3 FS0:\efi\boot\BOOTx64.efi "FreeNAS"

Afterward I rebooted into the UEFI configuration and switched the boot priority to the new Boot Variable "FreeNAS" to be the first. Now booting directly into FreeNAS works.

Conclusion

Once I had found the quirks (FreeNAS not providing efirt.ko, EFI shell of the motherboard being restrictive) I am happy to say that the system works as expected again. Maybe I can provide some feedback and input to provide a better integration of UEFI support into FreeNAS, we'll see.

Had I remained with CSM boot, the restore procedure would have been faster, however I'm happy that I attempted it during the restore. UEFI isn't per se "better" than legacy boot, but newer systems that I encounter start defaulting to UEFI and some even flat out start behaving buggy when using CSM/legacy boot these days. (i.e. AsRock's J3455B-ITX board)

I have to thank the crew at iXsystems and the community around FreeNAS for providing a fully restorable configuration for the NAS OS of my choice for a couple of years already, for exactly those thing that I normally don't want to mess around.

Notes

1. I'm looking into automating a config export for my backups, it might be possible via the API. However I'll definitely change my behaviour / use to exporting a config before updating.
2. Explicitely adding a NVRAM Boot Options to the UEFI wouldn't have been needed at all, if the UEFI on my particular system wouldn't only look for Windows-style UEFI loaders on ESP partitions.
3. I've learnt that writing Boot Options to the UEFI NVRAM during install isn't yet integrated supported on any given FreeBSD release as of 12.0-RELEASE. UPDATE: Thanks to code contributed by Rebecca Cran (bcran@), FreeBSD 13 installer will add boot entries. The respective commit was made in December 2019: Mostly in r342637. Since then Rebeccas has contributed other EFI-related improvements to FreeBSD, thanks Rebecca!
4. Unfortunately some UEFI implementations don't keep custom Boot Options after reboots. It seems some implementations even more are buggy, luckily AsRock's implementation didn't have this bug.
5. It should be noted that although FreeBSD's efibootmgr(8) is the same command as in most Linux distributions, comparing the manpages reveals that the CLI parameters DO differ.