Status Report 2026 10 20 - MatthewFeroz/openeval GitHub Wiki

Status report: October 20, 2026

Sprint 2 (October 14 through October 27), week 1 of 2. Reporting period: October 14 through October 20. Author: Matthew Feroz. Prepared October 20 from the repository, issue and CI records. This is a new snapshot; the October 6 and October 13 reports are unchanged.

Overall status: AMBER

Development continued, but none of the five Sprint 2 issues has completion evidence yet. The plan put fresh-account onboarding (#16) and report-access acceptance (#17) in week 1. Both depend on the hosted app running Clerk sign-in. As of October 14, production was still serving the October 2 WorkOS build: the October 8 deployment failed because production lacked its Clerk settings. Clerk's domain and email checks now pass, but the app has not been redeployed.

Work this week went into the pieces those checks need: embedded sign-in, the Oracle example and release checklist, and consolidating private development into one branch. The release-check workflow passed on public main on October 14. With one week left, the immediate risk is schedule; real-account acceptance and upgrade checks may still uncover implementation defects. Anything without evidence by October 27 carries into Sprint 3 rather than being marked done.

Feature status

Major feature Started Design Development Testing Integrated
Worker setup, pairing and no-model check Yes Complete Complete Automated checks and the Oct 14 installed-setup release check pass; fresh-account acceptance (#16) not yet run Public source
Evaluation configuration and monitoring Yes Complete Complete UI/connected checks pass; hosted first-run acceptance pending Public source
Reconnect, cancellation and partial results Yes Complete Complete Complete (recovery validation) Public source; #10 closed
Private reports, Studio drafts and published revisions Yes Complete Complete; combined-poster editing and PNG export added this week Backend/export and fixture browser checks pass; real-role acceptance (#17) not yet run Public source; poster editing in private development only
Hosted sign-in through Clerk Yes Complete Complete; sign-in now opens over the current page Contract and browser tests pass; isolated check against the Clerk development instance; no live sign-in yet Public source; embedded dialog in private development only; production cutover pending
Turn/request reporting and evaluation proxy Yes Complete Complete Local regressions and public CI pass Public source
Reproducible Oracle example and release checklist (#18) Yes Complete Written Profile preparation and synthetic normalization pass; no real Harbor/Docker run Private development docs; not yet public
Installed-worker upgrades against Harbor 0.24 (#19) Yes Complete Harbor 0.24 pin complete (Sprint 1) Release checks pass on x64 and ARM Linux; upgrade/rollback evidence not recorded Public source
Public prototypes and architecture refresh (#20) No Not started Not started Not started Not started

Resource utilization

Matthew Feroz is the sole project owner and developer. Coding agents assisted; each pull request records the model and harness used. No actual-hours log is kept, so this section describes deliverables, not time spent.

Completed October 14-20:

  • Sign-in opens as a dialog over the current page instead of navigating away, and keeps report links intact. The Clerk development and production instances were set to email/password with verification, with passkeys and the unconfigured Google option turned off. (Private development PR #39, merged October 15.)
  • Wrote the Oracle evaluation example, the remaining acceptance checklist, and worker upgrade/rollback guidance. A read-only check found the production deployment gap described above. (Private PR #38, merged October 15.)
  • Consolidated pending branches into private main, including combined-poster editing/export and the archived SWE-bench study results. Recorded a proposal for a public task and run gallery. (Private PR #40, merged October 15.)
  • Ran the release-check workflow on public main (October 14): deployment builds and installed setup on x64 and ARM Linux workers passed.

No repository changes were recorded October 16-20.

Planned October 21-27 (Sprint 2 week 2):

  1. Decide whether to configure production Clerk settings and deploy the current public release. Deployment was outside the original Sprint 2 commitment. If approved, run fresh-account onboarding (#16) and owner/editor/viewer/anonymous report access (#17) against it.
  2. Record installed-worker upgrade and rollback evidence against Harbor 0.24 (#19).
  3. Transfer the Oracle example and release checklist to the public repository with a real run (#18).
  4. Refresh public prototypes and architecture pages (#20). This is still committed work, but it is last in sequence and at risk if the earlier items slip.
  5. Sprint 2 closeout and Sprint 3 planning on October 27.

Team activities

No human team meetings took place; this is a one-person project. The working sessions below are recorded in pull requests and CI runs.

Date Purpose and participants Outcome and follow-up
Oct 14 Sprint 1 closeout and Sprint 2 planning; Matthew Feroz with Codex assistance Sprint 2 issues #16-#20 created; #10 closed; October 13 report posted.
Oct 14 Release-check run on public main; Matthew Feroz Deployment builds and installed Linux worker setup passed on x64 and ARM.
Oct 14-15 Sign-in, acceptance documentation and branch consolidation; Matthew Feroz with Codex assistance Private PRs #38-#40 merged with CI passing. Production Clerk configuration found to be a prerequisite for #16/#17.
Oct 20 Discussion of the public run gallery proposal (recorded October 15) and preparation of this report; Matthew Feroz with Claude assistance No change to Sprint 2 scope.

Changes and major decisions

Change Reason Impact
Sign-in moved into a dialog over the current page Sign-in sent users away from the page they were on, and the account button stayed disabled after cancelling Report links and their parameters are kept through sign-in. Takes effect only after deployment.
Email/password with verification as the only sign-in method; passkeys and Google turned off Match the documented sign-in flow; the Google option had no provider configured One sign-in path to accept. Social providers can be configured separately later.
Production Clerk configuration and deployment identified as a prerequisite for #16/#17 As of October 14, production served the October 2 WorkOS build #16/#17 did not run in week 1. Deploying needs an explicit decision because it was outside the sprint commitment; this is the main schedule risk.
Pending private branches consolidated into one main branch Work was split across branches, so main was incomplete One source for the next public release; no public changes this week.
Public task and run gallery proposed, not scheduled A visual way to compare harness/model outputs on the same task Scope unchanged for Sprint 2; considered for a later sprint.