Testing and QA - Fighter90/career-ops-ui GitHub Wiki

Testing and QA

The test pyramid (v1.228.0 baseline)

Layer Count Command What it covers
Unit / integration 3210 npm test (node --test tests/*.test.mjs tests/acceptance/*.test.mjs) In-process createApp() hit with fetch on an ephemeral port; registry invariants; sanitizers; provider parity suites (stubbed transport, no network)
Playwright browser 101 npm run test:e2e:browser Narrow-viewport (320px overflow, v1.227.5), smoke, full-cycle, forms, locale sweep, theme toggle, persistent widgets (docs FAB + usage HUD)
Smoke E2E 21 npm run test:e2e (tests/e2e.mjs) Real server in a child process, every route walked
Comprehensive E2E 23 npm run test:e2e:full (tests/e2e-comprehensive.mjs) Long full-surface pass โ€” catches SPA regressions unit tests can't
  • Coverage floor: 80 % on non-trivial logic; the actual baseline is ~93 % line / ~83 % branch (npm run test:coverage).
  • Never hardcode port 4317 in tests โ€” always server.listen(0).
  • TDD when adding behavior (red โ†’ green โ†’ refactor); skipped only for pure refactors already under full coverage.
  • No mocks of internal collaborators โ€” fake the parent by pointing CAREER_OPS_ROOT at a mktemp -d with the minimal files the path under test needs.

CI gates

npm run test:ci = npm test + three deterministic gates:

  1. scripts/check-no-also-leftovers.mjs โ€” doc-hygiene check.
  2. scripts/check-changelog-parity.mjs โ€” all 16 translated CHANGELOGs at the current version.
  3. tools/i18n-audit.mjs โ€” locale dictionary integrity.

Plus in the workflow matrix: Node 18/20/22 runs, CodeQL, and the Playwright suites. Key i18n/docs parity tests: tests/i18n-locale-files.test.mjs, tests/i18n-coverage.test.mjs (snapshot at tests/fixtures/i18n-dict.snapshot.json), tests/canonical-docs-coverage.test.mjs, tests/help-ru-config-section.test.mjs, tests/help-ui.test.mjs, tests/manifesto-link.test.mjs (help bundle 32 H2 / 122 H3 structure gates), and โ€” since v1.227.5 โ€” tests/help-source-counts.test.mjs + tests/help-banner-strip.test.mjs, which guard the help bundles' CONTENT rather than their headings: the ยง17 source total and EN/RU breakdown are asserted against the live registry (they had silently contradicted each other for several releases), the paragraph may pin no version, and the GitHub-only provider banner must never survive the help ingress into the page or the docs-assistant corpus.

The hard gate is ci.yml, not the pre-commit hook. The pre-commit AI review is advisory; a green pre-commit with a red CI is possible โ€” watch the CI run.

CI-isolation rules (the traps)

Tests cannot assume the parent career-ops project exists. The two rules that cost releases when violated:

  1. PATHS resolves once per process. server/lib/paths.mjs computes PROJECT_ROOT at import time. A test that sets CAREER_OPS_ROOT in before() must load every paths.mjs carrier (server/index.mjs, prompts.mjs, store.mjs, en-scanner.mjs, ru-scanner.mjs, paths.mjs) via dynamic import() inside before(). A top-level static import runs before the env is set, pins the REAL parent, and leaks writes (e.g. PUT /api/profile) into the user's real files. Guards: tests/paths-once.test.mjs, tests/test-root-isolation.test.mjs.
  2. Bootstrap the minimal layout. CAREER_OPS_ROOT=$(mktemp -d) + write only what the test needs (cv.md, portals.yml, โ€ฆ). Fixtures live under tests/fixtures/ โ€” never real user data.
  3. New FS-write helpers take an explicit path param โ€” otherwise the once-per-process resolution leaks writes into the real parent.

Two dedicated reviewer subagents enforce conventions: web-ui-route-reviewer (routes/security envelope) and test-isolation-reviewer (CI isolation, no live network, no port collisions).

QA prompt methodology (qa/)

Every release ships a QA regression prompt โ€” qa/QA-REGRESSION-PROMPT-v<version>.md โ€” a delta driver covering only what that release added. Exactly one lives at the top of qa/: archiving the previous one into qa/archive/superseded-prompts/ is part of shipping a release, so "which prompt is current" never needs thought. That rule was written at v1.137.0 and then not applied for 95 releases โ€” by v1.231.1 the top level held 98 stale prompts, archived in v1.231.2.

Three perennial prompts sit beside it and read package.json::version rather than pinning one: FUNCTIONALITY-CHECK.md (does it actually work), UX-AUDIT-PROMPT.md (is it good UX) and DESIGNER-EXPORT-PROMPT.md (design-system + key-flow export). v1.231.2 retired five more drivers that called themselves current and were not โ€” REGRESSION-FINAL.md chief among them, whose ยงยง11โ€“15 are five closed cycle ledgers (v1.55.x โ†’ v1.59.7) and whose stated help baseline was 28 H2 / 103 H3 across 16 locales against 32 / 122 and 17 today.

Structure of a delta driver (e.g. v1.118.0):

  • A header pinning the version under test, parentVersion, route-module count, adapter count, and the unit-test baseline.
  • One ยง per shipped feature with concrete, executable checks: exact endpoints to hit, exact test files to run, exact UI states to verify (e.g. "GET /api/scan/sources โ†’ the EN list contains all 9 new values", "POST /api/tracker {status:'Hired'} โ†’ row lands; unknown status degrades to Evaluated").
  • A docs & i18n fan-out ยง (parity scripts + locale-file tests + help H2/H3 counts).
  • A sign-off ยง: npm test green (โ‰ฅ baseline), npm run test:ci green, Playwright green, both E2E suites green, CI matrix (Node 18/20/22 + CodeQL) green โ€” with the known CodeQL false-positive dismissal rationale pre-stated.

Operational rules learned the hard way (see also Troubleshooting & FAQ):

  • Never npm test 2>&1 | grep โ€ฆ โ€” grep masks the exit code; two releases shipped failing tests this way. Run the suite, capture $?, grep separately.
  • GET-only when smoke-testing a live deployed server โ€” no write-API calls against a real parent.
  • Baselines only ratchet up: the next ship must keep all four suite counts โ‰ฅ the previous floor.
โš ๏ธ **GitHub.com Fallback** โš ๏ธ