Benchmarks - sameernagar-hub/regen-engine GitHub Wiki
python scripts/bench_system.py runs every benchmark and prints one JSON line per benchmark. Each one compares a component
against a baseline or a real corpus, so a gap shows up as a number. Rule: when a new failure mode is found,
add a benchmark for it. The list only grows.
| # | Benchmark | What it measures | Baseline | Result (2026-10-10) |
|---|---|---|---|---|
| B1 | select |
top-k pick with freshness bands: lazy heap vs full sort, 50k synthetic jobs, k=100 (same picks asserted) |
sorted() 31.6 ms |
heap 5.5 ms, ×5.8 |
| B2 | answers |
share of every question the engine ever stopped on (297, from events.jsonl) that rules + saved answers answer now
|
0 | 67.5% of non-legal (158 of 234); 63 are legal/EEO and stay yours |
| B3 | claims |
8 threads race for 200 inbox batches; each must be claimed exactly once | — | rename-only claim: 233 claims for 200 (bug); lock-file claim: 200/200, 3 of 3 runs |
| B4 | coverage |
your JD terms that reach the resume, last 200 resumes | 88% median before the coverage fill | median 100%, 64.5% of resumes at full coverage, ATS read-back median 100 |
| S1 | scripts/bench_scheduler.py |
round-robin tabs vs sequential, seeded | sequential | ×2.05 throughput |
| S2 | scripts/bench_tailor.py |
tailoring time per resume | v0.7 | ×3 faster (v0.8) |
-
B3 caught a real race: on Windows,
os.renamelet two workers claim the same batch. The claim is now an exclusive lock file (O_CREAT|O_EXCL), atomic on every filesystem. -
B2 lists the unanswered questions (
missing_sample). Each is a missing rule or preset: for example, "currently based in " (needs the city-match rule) and "transitioning service member" (needs a preset).
- B5: end-to-end submissions per hour per worker count (1, 3, 5), from
pipelinereport events. - B6: code latency, from email
issuedtime tocodes/*.txtwritten (relay log). - B7: fit-gate precision, the share of rejections whose JD had a blocker the gate missed (from
outcomeevents).