How an evaluation works

This page addresses the week 4 feedback to explain the system's operation more clearly. It follows the implemented public source at dd69e2b, not a private developer deployment.

A concrete example

An evaluator selects an online Linux worker with approved profiles for Codex, Claude Code, and Pi using the same model/vendor and the same two reviewed tasks. With one attempt per task, the experiment requests six trials: three harnesses x two tasks x one attempt. These are example selections, not reported benchmark evidence.

A task includes the problem, environment, and verifier. A trial is one attempt by one harness/model on one task. A child run is the work for one selected profile. The experiment groups those child runs into one comparison report.

Numbered lifecycle

Step	What the user sees	Component responsible and saved state
1. Prepare	Machines guides pairing and the no-model Oracle check	Operator installs Linux prerequisites and approves local profiles; daemon advertises public metadata; secrets stay on the worker
2. Configure	Evaluations offers compatible tasks, harnesses, models, vendor, attempts, and advanced settings	Browser constructs a selection from advertised profiles; task/profile digests identify what was approved
3. Submit	Start experiment creates saved work	Convex verifies identity, worker readiness, compatibility, attempt approval, and run settings; an idempotent request ID prevents a duplicate submission
4. Claim	A queued child run becomes active	Daemon polls outbound, claims owned work, and starts the detached supervisor; local state supports reconnect recovery
5. Execute	Monitoring shows task starts and provisional outcomes	Supervisor snapshots tasks and invokes Harbor; Docker supplies environments; harness requests inference; verifier checks candidate code
6. Observe	Counts and an expandable task activity log update	Supervisor collects bounded allowlisted telemetry; daemon uploads snapshots; Convex validates credentials/claim/sequence; browser subscribes
7. Save results	A child diagnostic report appears	Supervisor normalizes and validates the report from raw local artifacts and saves a durable outcome; daemon uploads that sanitized result
8. Combine	The experiment report becomes available after all cells are terminal	Convex combines available child trial rows, including partial outcomes when other runs fail, cancel, or interrupt
9. Present	Report opens Studio, exports, or a public revision	Shared chart recipes consume recorded rows; presentation changes do not execute another trial; owner publication is separate from saving a draft

Monitoring reports task lifecycle events, not terminal output, model messages, file content, or reasoning. The latest 200 lifecycle events are a bounded window, not a full audit log. Final reports remain the source for recorded metrics and outcomes. Unknown cost remains unknown; successful-task median time is not total end-to-end duration.

Execution sequence

sequenceDiagram
    actor User as Evaluator
    participant Web as Browser workspace
    participant API as Convex backend and database
    participant Daemon as Linux daemon
    participant Supervisor as Detached supervisor
    participant Harbor as Harbor and Docker
    participant Model as Model provider / Merge Gateway
    User->>Web: Select approved profiles and start
    Web->>API: Authenticated selection and request ID
    API->>API: Validate and persist experiment / queued runs
    API-->>Web: Saved experiment ID
    Daemon->>API: Poll and claim owned queued run
    API-->>Daemon: Claimed profile and run settings
    Daemon->>Supervisor: Launch with durable local state
    Supervisor->>Harbor: Snapshot tasks and execute profile
    Harbor->>Model: Harness inference requests
    Model-->>Harbor: Model responses
    Harbor->>Harbor: Run task verifier
    loop While executing
        Supervisor-->>Daemon: Bounded lifecycle snapshot
        Daemon->>API: Status and validated telemetry
        API-->>Web: Reactive progress update
    end
    Harbor-->>Supervisor: Outcome and raw local artifacts
    Supervisor->>Supervisor: Normalize and validate report rows
    Supervisor-->>Daemon: Durable completion state and report
    Daemon->>API: Normalized sanitized child report
    API->>API: Combine available results when all cells terminal
    API-->>Web: Private experiment report
    User->>Web: Review and explicitly publish revision
    Web->>API: Owner-authorized publication
    API-->>Web: Revocable public link

What happens when something goes wrong?

Event	Observable behavior	Recovery or limit
Browser closes	Execution continues; saved experiment remains	Reopen Evaluations and the experiment
Daemon disconnects/restarts	Supervisor can continue; cloud heartbeat becomes stale	Reconnect to recover local state and latest monitoring snapshot; no automatic duplicate paid run
Worker heartbeat stops	UI says worker offline; running work may still execute	Restore worker connection; do not infer the harness stopped
Telemetry is stale with a live heartbeat	UI says monitoring delayed; timers freeze at last sample	Final report/cancellation are not blocked by telemetry upload failure
User cancels queued work	Backend marks it cancelled before execution	No task is launched for that queued run
User cancels active work	UI shows cancellation pending	Worker must stop/clean up and acknowledge; offline revocation cannot guarantee immediate cleanup
Host reboots or is lost	A supervisor may be interrupted	Claimed work is not silently rerun elsewhere; preserve local artifacts and inspect the terminal outcome
Approved tasks/profile changes	Old selection becomes stale	Review advertised options and resubmit a new selection
A child fails, cancels, or interrupts	Error/unfinished state remains visible	Combine available results after all child cells are terminal; no false verifier pass
Public link is revoked	Future reads through that link fail	Private report remains; existing downloaded copies cannot be recalled

Check the explanation against source

Submission and combination, claims and telemetry, supervisor lifecycle, monitoring contract, connected-runner recovery, and report sharing.

Recovery validation

Reconnect and cancellation validation records the verified scenarios, source/CI evidence and limits. Cancellation retains completed verifier outcomes and excludes unfinished trials. Recovery does not silently start another claimed run.