This took a huge amount of fiddling and was very confusing, so documenting it here for my own benefit and others’.
I wanted to run Ubuntu Server ARM64 on a Raspberry Pi CM4 dev board using a NVMe SSD for at least the root filesystem (and ideally booting from it)
Step 1: Update the CM4’s bootloader firmware
The CM4 probably has an older bootloader firmware installed from the factory that doesn’t support NVMe. This step is best done from a Linux PC. First you need the latest version of the Raspberry Pi USB boot tool.
# if you didn't have these installed already... sudo apt install git sudo apt install build-essential sudo apt install libusb-1.0-0-dev git clone --depth=1 https://github.com/raspberrypi/usbboot cd usbboot make
Now you need to edit the boot order settings and get the latest EEPROM image. This part is adapted from this tutorial which doesn’t work verbatim any more (the “nvme” folder doesn’t exist in the usbboot repository, but substituting the “recovery” folder works)
cd recovery sed -i -e '/^BOOT_ORDER=/ s/=.*$/=0xf25416/' boot.conf
NVMe on Raspberry Pi is still kind of bleeding edge, so it’s best to use the newest versions of everything. pieeprom-2021-07-06.bin was still the latest image at the time of writing:
rm -f pieeprom.original.bin curl -L -o pieeprom.original.bin https://github.com/raspberrypi/rpi-eeprom/raw/master/firmware/beta/pieeprom-2021-07-06.bin ./update-pieeprom.sh
Finally it’s time to burn your changes to the CM4… The “disable eMMC Boot” jumper must be fitted on J2 for this to work (Unless you have a CM4 Lite I guess…)
cd .. sudo ./rpiboot -d recovery
Now connect the CM4 Dev board J11 “USB Slave” to a USB port of your Linux PC and apply power to the 12V DC input.
The rpiboot tool should boot up the CM4 over USB and flash the new EEPROM image and boot order settings. Once the command line spew stops and the power LED on the dev board starts to blink rapidly, the process is complete and you can disconnect power.
Step 2: Try to boot from NVMe, scratch head
At this point I already had an ARM64 Ubuntu 20.04 LTS image downloaded from here and uncompressed to my Gigabyte GP-GSM2NE3128GNTD SSD. So I removed the SD card from my CM4 dev board and tried to boot from the NVMe.
I got a surprisingly informative error message from the Raspberry Pi bootloader (on a monitor connected to HDMI0)
start4.elf is not compatible. nvme boot requires newer software
A bit of Googling got me to this forum thread:
So my first attempt at a fix was to update a 32-bit Raspbian SD card image with rpi-update and then copy the start4.elf and fixup4.dat files from that image to the NVMe boot partition. This got me to the next problem: The Ubuntu 20.04 image uses a version of U-Boot that doesn’t support NVMe, so it just sticks waiting for a boot device.
Next I tried downloading the Ubuntu Server 21.04 image and burning it to a SD card. I planned to go through the full process documented in the forum thread, starting with upgrading the Ubuntu image, so I inserted the SD card and powered up.
Step 3: Unexpected success
Imagine my surprise when it booted into Ubuntu 20.04 from the NVMe instead of 21.04 from the SD card as I was expecting.
I guess what must have happened is that the boot partition on the 21.04 SD card had a version of UBoot that recognised the NVMe and treated it as a higher priority than the SD.
The next gambit would be to copy the boot partition from the SD card to the NVMe and see if it can boot directly from NVMe. But it works well enough as is, I don’t want to break it! This is left as an exercise for the reader