The data on the NAND flash is scrambled by applying XOR masks to every quarter of a page (often referred to as chunk, 1024 bytes of data + 64 bytes of OOB). It's the same mask for all 4 chunks within a page and masks repeat after 64 pages (seems to restart with every erase block).

For the image at hand the XOR masks could be recovered using statistical analysis.

(For now we consider these masks as 'random' but their characteristics seem to suggest otherwise.)

A few bit errors are expected in some of the pages due to the nature of raw NANDs. These occasional bitflips get corrected by the hardware ECC controller of the SOC.

The MT8516 does it in 1024 byte chunks at a time for 4K page sizes. The BCH codec uses the following parameters: t=32, prim_poly=17475

The first 8 unused bytes of the OOB/spare area also get included in the calculation.

The integrated ECC controller seems to read bits in the 'opposite order'. That means each byte of the flash dump image has to be reversed temporarily for the error correction step, then back.

(The primitive polynomial thus turns out to be X^14 + X^10 + X^6 + X + 1, or 100010001000011 in binary, 0x4443, or 042103 in octal representation. As per the MATLAB Help Center that is the default for GF(2^14).)

---

(Left the notes here from earlier below. Please note those parameters apply to other SOCs using 512 byte chunks: t=12, prim_poly=8219)

The data on the NAND flash is encoded, likely using the hardware ECC flash interface of the CPU.

Assumption is ECC BCH encoding is used - see also page 1034 of this pdf and that parts of the MT CPUs are the same (e.g. messages (from the preloader here)[https://github.com/prshkr07/Thunder-Kernel/tree/master/mediatek/platform/mt6582/preloader/src] seems to be very similar to the UART logs of the dot, including spelling errors in the UART log [TOOL] <UART> receieved data: () and here)

The BCH codec module is implemented in GF(2^13) defined by primitive polynomial X^13 + X^4 + X^3 + X + 1.

GF = Galois Field/Finite field Wikipedia on BCH code

So given this, can we decode a set of bytes for which we know the output - the Android boot image should start with ANDROID!.

From the article here,

if g(x)=x8+x4+x3+x2+1 --> 0x11D in hex, 100011101 in binary then X^13 + X^4 + X^3 + X + 1 --> 10000000011011