Status Report 2026 10 20 - MatthewFeroz/openeval GitHub Wiki
Status report: October 20, 2026
Sprint 2 (October 14 through October 27), week 1 of 2. Reporting period: October 14 through October 20. Author: Matthew Feroz. Prepared October 20 from the repository, issue and CI records. This is a new snapshot; the October 6 and October 13 reports are unchanged.
Overall status: AMBER
Development continued, but none of the five Sprint 2 issues has completion evidence yet. The plan put fresh-account onboarding (#16) and report-access acceptance (#17) in week 1. Both depend on the hosted app running Clerk sign-in. As of October 14, production was still serving the October 2 WorkOS build: the October 8 deployment failed because production lacked its Clerk settings. Clerk's domain and email checks now pass, but the app has not been redeployed.
Work this week went into the pieces those checks need: embedded sign-in, the Oracle example and release checklist, and consolidating private development into one branch. The release-check workflow passed on public main on October 14. With one week left, the immediate risk is schedule; real-account acceptance and upgrade checks may still uncover implementation defects. Anything without evidence by October 27 carries into Sprint 3 rather than being marked done.
Feature status
| Major feature | Started | Design | Development | Testing | Integrated |
|---|---|---|---|---|---|
| Worker setup, pairing and no-model check | Yes | Complete | Complete | Automated checks and the Oct 14 installed-setup release check pass; fresh-account acceptance (#16) not yet run | Public source |
| Evaluation configuration and monitoring | Yes | Complete | Complete | UI/connected checks pass; hosted first-run acceptance pending | Public source |
| Reconnect, cancellation and partial results | Yes | Complete | Complete | Complete (recovery validation) | Public source; #10 closed |
| Private reports, Studio drafts and published revisions | Yes | Complete | Complete; combined-poster editing and PNG export added this week | Backend/export and fixture browser checks pass; real-role acceptance (#17) not yet run | Public source; poster editing in private development only |
| Hosted sign-in through Clerk | Yes | Complete | Complete; sign-in now opens over the current page | Contract and browser tests pass; isolated check against the Clerk development instance; no live sign-in yet | Public source; embedded dialog in private development only; production cutover pending |
| Turn/request reporting and evaluation proxy | Yes | Complete | Complete | Local regressions and public CI pass | Public source |
| Reproducible Oracle example and release checklist (#18) | Yes | Complete | Written | Profile preparation and synthetic normalization pass; no real Harbor/Docker run | Private development docs; not yet public |
| Installed-worker upgrades against Harbor 0.24 (#19) | Yes | Complete | Harbor 0.24 pin complete (Sprint 1) | Release checks pass on x64 and ARM Linux; upgrade/rollback evidence not recorded | Public source |
| Public prototypes and architecture refresh (#20) | No | Not started | Not started | Not started | Not started |
Resource utilization
Matthew Feroz is the sole project owner and developer. Coding agents assisted; each pull request records the model and harness used. No actual-hours log is kept, so this section describes deliverables, not time spent.
Completed October 14-20:
- Sign-in opens as a dialog over the current page instead of navigating away, and keeps report links intact. The Clerk development and production instances were set to email/password with verification, with passkeys and the unconfigured Google option turned off. (Private development PR #39, merged October 15.)
- Wrote the Oracle evaluation example, the remaining acceptance checklist, and worker upgrade/rollback guidance. A read-only check found the production deployment gap described above. (Private PR #38, merged October 15.)
- Consolidated pending branches into private main, including combined-poster editing/export and the archived SWE-bench study results. Recorded a proposal for a public task and run gallery. (Private PR #40, merged October 15.)
- Ran the release-check workflow on public main (October 14): deployment builds and installed setup on x64 and ARM Linux workers passed.
No repository changes were recorded October 16-20.
Planned October 21-27 (Sprint 2 week 2):
- Decide whether to configure production Clerk settings and deploy the current public release. Deployment was outside the original Sprint 2 commitment. If approved, run fresh-account onboarding (#16) and owner/editor/viewer/anonymous report access (#17) against it.
- Record installed-worker upgrade and rollback evidence against Harbor 0.24 (#19).
- Transfer the Oracle example and release checklist to the public repository with a real run (#18).
- Refresh public prototypes and architecture pages (#20). This is still committed work, but it is last in sequence and at risk if the earlier items slip.
- Sprint 2 closeout and Sprint 3 planning on October 27.
Team activities
No human team meetings took place; this is a one-person project. The working sessions below are recorded in pull requests and CI runs.
| Date | Purpose and participants | Outcome and follow-up |
|---|---|---|
| Oct 14 | Sprint 1 closeout and Sprint 2 planning; Matthew Feroz with Codex assistance | Sprint 2 issues #16-#20 created; #10 closed; October 13 report posted. |
| Oct 14 | Release-check run on public main; Matthew Feroz | Deployment builds and installed Linux worker setup passed on x64 and ARM. |
| Oct 14-15 | Sign-in, acceptance documentation and branch consolidation; Matthew Feroz with Codex assistance | Private PRs #38-#40 merged with CI passing. Production Clerk configuration found to be a prerequisite for #16/#17. |
| Oct 20 | Discussion of the public run gallery proposal (recorded October 15) and preparation of this report; Matthew Feroz with Claude assistance | No change to Sprint 2 scope. |
Changes and major decisions
| Change | Reason | Impact |
|---|---|---|
| Sign-in moved into a dialog over the current page | Sign-in sent users away from the page they were on, and the account button stayed disabled after cancelling | Report links and their parameters are kept through sign-in. Takes effect only after deployment. |
| Email/password with verification as the only sign-in method; passkeys and Google turned off | Match the documented sign-in flow; the Google option had no provider configured | One sign-in path to accept. Social providers can be configured separately later. |
| Production Clerk configuration and deployment identified as a prerequisite for #16/#17 | As of October 14, production served the October 2 WorkOS build | #16/#17 did not run in week 1. Deploying needs an explicit decision because it was outside the sprint commitment; this is the main schedule risk. |
| Pending private branches consolidated into one main branch | Work was split across branches, so main was incomplete | One source for the next public release; no public changes this week. |
| Public task and run gallery proposed, not scheduled | A visual way to compare harness/model outputs on the same task | Scope unchanged for Sprint 2; considered for a later sprint. |