src/brawl/recognise.ts is pure pixel code with no DOM and no Node APIs, so the same functions run in the browser (canvas) and in the fixture test (sharp). Layout constants are fractions of a 2560×1440 screen and were measured from screenshots; other sizes scale linearly.

The three card icons are centred at (712, 855), (1280, 527) and (1848, 855), 185 px square. Each icon is located by searching ±24 px and four scales around its anchor, box-filtered to 24×24 and compared by normalised cross-correlation (per channel, the tier-numeral corner masked) against brawl-icons.json, which holds the 173 draftable item icons flattened on the card colour, plus the 38 hero portraits (npm run icon-index).

The tier numeral (I–V) is read by counting dark strokes in the icon's top-right corner. It also separates items that ship the same icon file: the index records those as twins and resolveTwin picks the one whose tier matches. The current snapshot has no twins.

"RARE!" is teal text above the icon and "ENHANCED" a blue box under the name; both are detected by colour fraction in their region. A card counts as present when its icon scores ≥ 0.75, or ≥ 0.45 with a readable numeral. Scoreboard screenshots read as no cards.

"ROUND n" above the timer and "CHOICE n OF 3" under the title sit in fixed windows. Each is thresholded on its own text colour (light for the round, magenta for the choice), cropped to the digit's bounding box, resampled onto an 8×12 grid and matched against digit templates taken from the screenshots. Round 5 is hand-drawn since no screenshot of it exists yet; an unreadable digit leaves the tab's own counter alone.

The eight portraits in the top bar are matched with a circular mask against each hero's card art cropped to the head. The player's own portrait is the one drawn as a square-topped tile rather than a circle (readSelfSlot: two full-height vertical edges about a portrait apart, with the tile fill differing from the bar background beside it). The portrait is only a fallback for the hero (see below). While the "round starting" banner is up every slot is a tile, so those frames are skipped. The side holding the player's hero is the player's team and the other four are the enemies. A hovered portrait reads as unknown, so the enemy list fills in over a few frames.

1. A hero you pick by hand in the app always wins.
2. The loading screen. "Joining the fight as INFERNUS" is read while the loading screen is up (the name box is sent to the name reader every 2.5 s, up to 12 times). That hero holds for the whole match; no portrait read changes it. The next loading screen starts a new match with its own hero, so playing Kelvin and then Graves switches at the second loading screen.
3. The scoreboard portrait, only when no loading screen was read this match (the app was started mid-match, or the read failed). The app then says it is guessing: the status line reads Hero guessed from portraits: <name>. Pick the hero to correct., the hero pill carries a guessed tag, and the overlay shows Hero guessed: <name> for 5 s on the first draft. A loading read or a hand pick clears all three.
4. Nothing read: the hero picked in the window, with the line Hero not read. Using <name>.

A new match starts at a loading screen, or at round 1 choice 1 (or a round lower than one already seen) with no loading screen in the last 5 minutes. A loading read is never used for more than 60 minutes.

Why the loading screen first: the portrait reader can take an enemy's portrait for your own. In the 2026-10-15 recording, at 11:31, it read an enemy McGinnis as the player while the player was Infernus (the frame is pinned in scripts/fixtures/hero-misread/). Release 0.4.0 did this more often (it lacked a later fix to the own-tile test), which is how an Infernus game was advised as Viscous. scripts/__tests__/hero-replay.test.ts replays every labelled draft of both recordings: Infernus throughout, from the loading screen in every match that shows one.

The draft screen's bottom-left inventory grid is two rows of five 66 px icons on a 75 px pitch. A slot is skipped on contrast first: a flat tinted square is empty. The rest are matched against the icon index, and a new entry that was on offer is the card just taken, so the owned list needs no clicking. An empty slot with a background line through it can look like a dark streaky icon, so an item that was never offered in this game needs a near-perfect match (≥ 0.85) while offered or owned items need ≥ 0.6. The manual card buttons remain as a fallback. Nothing is sent to the game; only pixels are read.

Icon matching is coarse-to-fine: the nominal window is scored against all 173 icons and the position and scale search runs on the top 12 only. Hero portraits shortlist the top 8 from five windows (centre and the four corners of the search range, since portraits shift a little between screens). This gives the same accuracy as the exhaustive search at about a fifth of the time.

npm run brawl:see -- --fixtures            # 27 labelled card crops from 9 screenshots: item, tier, rare, enhanced
npm run brawl:see -- --screens             # 9 screen fixtures: round, choice, own slot and the eight portraits
npm run brawl:see -- screenshots/x.png     # name the three cards of a full screenshot
npm run brawl:see -- --save-fixture s7 screenshots/brawl/s7.png "Mystic Regeneration,Extended Magazine,Spirit Strike"

Current accuracy is 27/27 cards and 96/99 labels, both against a 95 % floor; either command exits non-zero below it. The three missed labels are one hovered portrait on two screens. The screenshots themselves stay out of git; only the card crops under scripts/fixtures/brawl-cards/ and the blanked screen fixtures under scripts/fixtures/brawl-screens/ are committed.