Model trials - diegoami/imperial_conquest_2 GitHub Wiki

Model trials

What each reviewer and implementer model actually caught, missed or failed on, in this build. The review tiers in build-process.md §3.4 cite this page when they substitute a model. It records the substitution's cause, the probes, the substitute, and what it caught or missed against the reviews before it. This page is append-only: a later entry corrects an earlier one, and nothing is rewritten.

1. Luna's terse approval of a six-task plan, and Sol's seven findings (2026-10-03)

Plan PR #632 (T114–T119) was reviewed by GPT-6 Luna (openai/gpt-6-luna, high). It approved with a two-line comment that named no findings. Its transcript showed 27 tool calls over the entries, reports and code symbols, so it had read them. The PR merged as 723fb9b. At the user's request, a post-merge review by GPT-6 Sol (openai/gpt-6-sol) found seven blocking defects, every one real:

  • an assertion that a session's active seat is the shuffled first nation (false whenever that nation is an AI);
  • an impossible Fisher–Yates count;
  • a seed-pinned test missing from Owns;
  • a two-command dialog that could commit half a transfer;
  • a headless check whose turn order T119 would change;
  • an untested port gate;
  • two Done-when lines that were not runnable.

These were fixed in #641. On #641, Sol also rejected the main session's first arithmetic for R8, correctly. Outcome: the user's review-tier rule of 2026-10-03 (Sol for complex reviews, plan PRs with contracts included).

2. Sol on its first code review (fix 594, 2026-10-03)

On PR #640, Sol (at high, before the user's effort rule) found and ran an overflow: disembark-army <army> -2147483648 3 threw OverflowException from LandingTile.ChebyshevDistance. The rework round fixed it, and Sol approved the re-review.

3. The OpenAI account out of quota; GLM-5.3 substitutes (2026-10-03)

  • Error: Sol's review of T27 (#643) failed at once: Error: The usage limit has been reached.
  • Probes (one-prompt opencode run, stdin closed):
    • openai/gpt-6-luna failed with the same error, so the whole OpenAI account was out;
    • opencode-go/glm-5.3 answered OK;
    • opencode-go/deepseek-v4-pro answered OK.
  • Substitutes:
    • T27 had a DeepSeek implementer, so its review went to GLM-5.3 (review). GLM reproduced the whole DoD itself, including the scrubbed, copied-away launch and a sidecar mutation, and named three real non-blocking points: the .cs.uid litter, the DOTNET_ROOT spellings, and the shipped .pdb files. Sol had not reviewed T27, so there was nothing to compare.
    • GLM-5.3 also reviewed plan PR #644. It caught a provenance error the main session had introduced: a global rename turned "renamed from post-v0.4.0" into "renamed from post-v1.0.0". It also confirmed #641's R8.
  • Side incident: GitHub returned 503s during the same hour. The review script's gh pr comment for #644 failed (unexpected EOF), yet the script printed posted: and exited 0. The main session recovered the review from the run's transcript export. Filed as bug #645, and fixed in the review-tiers PR: a failed post now exits 5 and saves the review.
  • Outcome: the user's substitute policy (GLM-5.3, then DeepSeek V4 Pro, then Luna; the family rule; Opus last unless Claude implemented), and Sol limited to low effort (medium at most, never high) once the credit returned.

2026-10-04/05: GLM moves to Z.AI, its 5-hour limit, and the quota rules (harness_imperial L50/L51)

  • GLM on the Z.AI Coding Plan (PR 692), after OpenCode Go's credit ran out:
    • It reviewed T124's rework, T132's rework and three passes of plan PR 695, and caught real defects each time. On 695 that was a check that could never fail: it searched the logs for a word T131's script never prints.
    • It then hit Z.AI's 5-hour window: Usage limit reached for 5 hour, exit 1, nothing posted. Sol at low took that re-check.
  • The quota rules (PR 700, CLAUDE.md rules 17–18, the user's instruction of 2026-10-05):
    • Every model choice is checked against quota-tracker (localhost:8765/avoid), and an exhausted provider is skipped and named.
    • luna is now GPT-5.6 Luna (openai/gpt-5.6-luna, effort high), on its own weekly pool. GPT-6 Luna is never the reviewer.
    • Heavy models run light: GLM-5.3 at low (it has no medium), Sol at low, and DeepSeek V4 Pro at high (the lightest it offers).
    • GPT-5.6 Luna's first review, of PR 700 itself, found four real stale claims. Its re-check approved.

2026-10-11: one blocker per round on T157's test exemption (PR 953, Sol)

  • Reviewer: Sol (openai/gpt-6.1-sol) on T157's PR 953, implementer Claude Sonnet 5.5.
  • Rounds: round 1 named one blocker (IsSuperseded found a decision by value with IndexOf, so a final duplicate could exempt itself). Round 2 named one blocker: the "destroyed twice" ambiguity guard in the turn-boundary reconstruction had no test (removing count > 1 kept all 4,723 tests green).
  • The class: gaps in the turn-boundary reconstruction helper that T157 added to T156's AiTargetTreeDecisionsAreCarriedOutTests (army lifetimes and untested guard branches). Each fix added a new branch, which the next round found untested.
  • The main session's sweep found one more of the same class: an army created (mobilised, RecruitMobilized with ArmyWasCreated) after the boundary, alive at the round's end and never moved, is placed in the boundary state at its end position. It could block a path it was not on at the boundary, and so widen the exemption. Reasoned from the code, not run. The implementer's claim that creation events are unavailable was wrong: RecruitMobilized carries ArmyId and ArmyWasCreated.
  • Contributing cause (main session): the main session asked the implementer to close the id-reuse edge in round 2 without asking for a test of every branch it added, and that branch is what round 2 found.

2026-10-15: fixtures provenance finding rated non-blocking (T129, Kimi K3 1M)

  • Reviewer: Kimi K3 1M (opencode-go/kimi-k3), user-selected independent reviewer for fixtures PR 1, full review.
  • Evidence: independently verified all 5,859 payload hashes, all 5,541 saves against upstream manifests, and all 318 hook logs record-for-record against upstream. It also proved that the fixture logs are CRLF while upstream Git blobs are LF, so README/PR wording “copied unchanged” overstates byte identity. The conversion direction is unproved.
  • Classification: reviewer approved and called this documentation-level N1 non-blocking. Main session required rework under the declared exact-byte/provenance boundary and build-process section4.2/run-task step3: a proven false provenance claim inside that boundary cannot be waived as wording. No gameplay or log-record mismatch was found.
  • Action: Sol medium corrects the README and PR wording, preserving all payload bytes and manifests; Kimi 1M rechecks the correction before merge. Main-session disposition.
⚠️ **GitHub.com Fallback** ⚠️