Boot process - jvandewiel/no-alexa GitHub Wiki
Boot process dissected
BootROM
- The Boot ROM's code is stored in the SoC itself.
- Takes values from eFuse as input.
- Shows up as device
0e8d:0003over USB (LinuxidVendor=0e8d, idProduct=0003, WindowsUSB\VID_0E8D&PID_0003) - It loads the preloader.
Preloader
- ...is stored in partitions
brhgptpl_[0-3](combined BootRomHeader + GuidPartitionTable + PreLoader). - It has 4 nearly identical copies at fixed blocks 0, 16, 32, 48 (most likely the BROM has a way to verify them in order to avoid booting a corrupted copy stored in a bad block.)
- Obviously this bit of code is running while the device is in preloader mode: USB ID
0e8d:2000 - This has a short, sub-second timeout: unless a successful handshake happens quick enough the normal boot process proceeds.
- At start it can only use the tiny SRAM, so one of its main tasks is to initialize the DRAM.
- It also decides which slot to boot (A or B) based on the few bytes stored in the
miscpartition:
NAND INFO:nand_bread 245: blknr:0xA00, blks:0x1
(Note: blks here refer to 4KiB pages instead of whole NAND erase blocks.)
Little Kernel
- ...is stored in the
lk_[ab]partitions. - It loads and initializes the Trusted Execution Environment (TEE) first.
- This is where the device becomes 'self-aware': it reads its own details from the
idme_nandpartiton. - If the Action (Uber) button is pressed then it provides the interface for the
fastbootmode: USB ID0bb4:0c01 - Otherwise it continues to normal (or recovery) boot mode by loading the kernel.
- (Also contains cryptographic checks to prevent test devices -e.g. dvt, evt, hvt- mixed with the production code/environment.)
Trusted Execution Environment
- ...is stored in the
tee[12]partitions. - Provides the ARM TrustZone (ATZ).
- (As a side-effect it blocks direct access to certain resources for the 'regular' OS.)
Android kernel
- ...is stored in the
boot_[ab]partitons. - Starts the actual OS kernel, then control can optionally be passed to the contents of an initial ramdrive (initcpio).
- (The latter can be extracted from the single binary blob using the
unpackbootimgtool found here.) - The initrd seems to provide the
recoverymode for Android. - (It's unclear yet how that's triggered and also its usefulness - given the fact it's a headless device with no display to show graphics on.)
Root filesystem
- Finally the read-only root filesystem gets mounted through a few layers...
- Most of the NAND flash is allocated to the
userdata(GPT) partition. This partition contains a UBI overlay with multiple volumes in it (UBI is only used as a storage layer here: it provides wear-leveling + adds an extra layer of CRC protection). - Two of the dynamic volumes are
system_[ab](one copy for each slot). - The kernel is told to export this volume as a block device. That way the (partially LZ4 compressed) squashfs filesystem becomes mountable (readable).
- The device also runs an
android-veritysignature check on the image before starting/init. - All the above results in this part of the kernel command line seen in the logs:
androidboot.veritymode=enforcing androidboot.slot_suffix=_a rootwait ro init=/init ubi.mtd=persist,4096,4,0 ubi.mtd=userdata,4096,20,1 skip_initramfs ubi.block=1,0 root=/dev/dm-0 dm="system none ro,0 1 android-verity /dev/ubiblock1_0"
Miscellanous bits
- It seems like Amazon locked down the device quite well: the root filesystem's integrity is protected by a dm_verity checksum and the steps leading up to that point seem to be signed as well. (TBD: check if we can shorten this as AVB 1.0 is in effect.)
- There's a Bad Block Table (along with its backup copy) at the end of the image. Inside of the dump I checked all blocks were set as healthy though.
GPT partition layout
The LBA sector size is 4KiB, the GPT starts at LBA 2, but the protective MBR is incorrectly only 512 bytes in size. The latter makes validation/parsing fail in most standard partitioning tools. One exception is this tiny utility:
$ ./readgpt -f gpt.bin
0 | brhgptpl_0 | 0x00000000-0x0000003F | 256K
1 | reserve0 | 0x00000040-0x000000FF | 768K
2 | lk_a | 0x00000100-0x0000027F | 1536K
3 | lk_b | 0x00000280-0x000003FF | 1536K
4 | brhgptpl_1 | 0x00000400-0x0000043F | 256K
5 | reserve1 | 0x00000440-0x000005FF | 1792K
6 | idme_nand | 0x00000600-0x000007FF | 2048K
7 | brhgptpl_2 | 0x00000800-0x0000083F | 256K
8 | reserve2 | 0x00000840-0x000009FF | 1792K
9 | misc | 0x00000A00-0x00000BFF | 2048K
10 | brhgptpl_3 | 0x00000C00-0x00000C3F | 256K
11 | reserve3 | 0x00000C40-0x00000DFF | 1792K
12 | tee1 | 0x00000E00-0x000012FF | 5120K
13 | boot_a | 0x00001300-0x0000223F | 15616K
14 | tee2 | 0x00002240-0x0000273F | 5120K
15 | boot_b | 0x00002740-0x0000367F | 15616K
16 | persist | 0x00003680-0x00003E7F | 8192K
17 | userdata | 0x00003E80-0x0001FDFF | 458240K
(The tool being beta, got the UUIDs wrong. Found those irrelevant so cut them from the output for brevity.)
If lsblk would be able to handle MTD devices then its output would look something like the following:
NAME SIZE TYPE MOUNTPOINTS
mtd 512M mtd
├─mtd0 256K part
├─mtd1 768K part
...
├─mtd16 8M part
| └─ubi0 8M ubi
| └─persist 1.4M ubifs /persist
└─mtd17 449.5M part
└─ubi1 449.5M ubi
├─ubiblock1_0 130.3M ubiblock
| └─dm-0 130.3M android-verity
| └─system_a 130.3M squashfs /
├─ubiblock1_1 130.3M ubiblock
├─cache 3.7M ubifs /cache
└─userdata 142M ubifs /data
Exploring contents of the flash dump under Linux
Notes:
- DATA.bin in the example should be the ECC corrected dump, with OOB data removed already.
- UBI requires an MTD device having the same characteristics as the NAND chip (4K pages, 256K erase blocks, 512M capacity), we can simulate that using a RAM-backed
nandsimdevice. - Some commands of
mtd-utilsare known to fail on certain page+OOB size combinations on simulated devices... somtdblockmodule comes to the rescue: it makes easy to access effective page contents (so we can ignore the OOB area). - Only the last 2 GPT partitions contain mountable filesystems; in order to match the exact size of the last partition on the real device we are going to create them all, just for padding.
- On some systems Linux exports actual hardware MTD devices (system firmware), also the
ddcommands below are being ran with unrestricted privileges. Which means there's a tiny chance you can accidentally overwrite your system BIOS, which might render your computer unbootable. Please think twice, double-check the mtd indexes for your machine and only ever write to simulated MTD partitions.
$ sudo -s
# modprobe nandsim id_bytes=0xc2,0xdc,0x90,0xa6 parts=1,3,6,6,1,7,8,1,7,8,1,7,20,61,20,61,32
# cat /proc/mtd
...
mtd18: 00800000 00040000 "NAND simulator partition 16"
mtd19: 1c180000 00040000 "NAND simulator partition 17"
[ WARNING: your MTD indexes might be different! ]
# modprobe mtdblock
# dd if=DATA.bin bs=256k of=/dev/mtdblock18 skip=218 count=32
# dd if=DATA.bin bs=256k of=/dev/mtdblock19 skip=250
# modprobe ubi mtd=/dev/mtd18,4096,4,0 mtd=/dev/mtd19,4096,20,1 block=1,0 block=1,1
# modprobe ubifs
# mkdir -pv /mnt/{persist,userdata,cache,system_a,system_b}
# mount -t ubifs ubi0:persist /mnt/persist
# mount -t ubifs ubi1:userdata /mnt/userdata
# mount -t ubifs ubi1:cache /mnt/cache
# mount -t squashfs -o ro /dev/ubiblock1_0 /mnt/system_a
# mount -t squashfs -o ro /dev/ubiblock1_1 /mnt/system_b
Cleanup
If the above steps were carried out while booted into a Linux system temporarily (e.g. booted using a live-CD) it might be easiest to reboot. Otherwise here's how to revert them:
# umount /mnt/system_b
# umount /mnt/system_a
# umount /mnt/cache
# umount /mnt/userdata
# umount /mnt/persist
# modprobe -r ubifs
# modprobe -r ubi
# modprobe -r mtdblock
# modprobe -r nandsim
(Please note: your regular system might already use some of the modules so they might fail to unload. Also feel free to delete the mount points if necessary.)