Testing - joeylking/agent-runtime GitHub Wiki
Testing
GOWORK=off go vet ./...
GOWORK=off go test -race -count=1 ./...
GOWORK=off keeps a workspace, if you made one, out of the core, so the core is exercised with only its own two dependencies, which is how CI sees it. Everything runs in memory or against temporary SQLite files, with no network and no model. The suite is organized by concern:
| File | Covers |
|---|---|
driver_test.go |
the loop, verbatim decisions, schema validation, policy outcomes, terminal tools, ErrAbortRun, timeouts, panics, step and failure limits, loop detection, persistence and reopen |
approval_test.go |
hash-bound approvals, approve then resume, reject, tampering refused, re-pausing when policy changes its answer, expiry |
interrupted_test.go |
interrupted steps, reconciliation outcomes, the interrupted_side_effect pause, side effects never re-executed by the runtime |
model_test.go |
accounting rows, usage and cost, call limits before dispatch, projected token and cost limits, transient retries, exhaustion, ambiguous attempts, record and replay |
accounting_test.go |
ServedError charging, concurrent calls reserving against the limits, uncapped requests refused under a projected limit, a cancelled context still recording the call |
retry_test.go |
Retry-After honoured, and a wait that cannot fit failing as model_unavailable |
limits_test.go |
active time excludes approval waits, elapsed time includes them, both end runs |
policy_test.go |
the require_approval constructor and the typed presentation decoder |
integrity_test.go |
the audit fixes: concurrent resumes executing once, a cancel stopping a live loop, one policy dispatch, unusable JSON recorded as invalid, a step committing with its run, the driver's clock deciding expiry |
security_test.go |
the security sweep: deep JSON, hashes of encoding errors, undecodable rows, latest-approval-only grants, unusable usage, saturating totals, lease ownership and the starved heartbeat |
lease_test.go |
the run lease: refusal of a live run, takeover after expiry, a loop losing its lease, long calls, Cancel without a lease, the v0.2.1 migration |
cache_test.go |
the step cache equals a fresh reload, after every write, through failure, and against consumers that write through what they were handed |
hash_test.go |
the canonical encoder and content hash pinned byte for byte (golden), the boundary check on consumer JSON |
clock_test.go |
the clock-reading contract (below) |
store_internal_test.go, store_open_test.go, store_mode_test.go |
panic rollback, concurrent first opens, newer schemas refused, insertion order, paths with URI characters, read-only opens, file modes under any umask and in a child process, symlinks, schema references |
api_test.go |
the v0.3.1 core additions: Store.Lease, Store.ListEventsAfter across runs, an expired approval as ErrApprovalExpired, Operator on a chosen clock, Store.ApprovalReason, and CheckArgs as the driver's acceptance |
schema_internal_test.go |
CheckArgs compiling a spec once |
gate_test.go |
the gate: a loop outside the runtime leaving the Driver's record, misuse refused, limits through Step, abandoned sessions at each point, the GrantTTL re-check at Execute, Session.Input, and no goroutine left behind |
canonical_test.go |
agentrt.CanonicalJSON as the form hashes are computed over |
example_test.go |
runnable Example functions whose output go test checks |
bench_test.go |
the driver benchmarks behind Performance |
The packages inside the core module (providers, render, trace, view, replay, cmd/agentrt, and since v0.3.1 testkit, export, approver, approver/webhook, and examples/approver) are covered by their own tests in the same run, render.Messages against the previous implementation over 3,000 random runs among them. The nested modules are not: ./... does not cross a go.mod.
For a consumer: the test kit
This page is about testing the runtime. A consumer testing its own tools, policy, and agent uses the testkit package, which crashes the loop at a chosen point and resumes it in a fresh driver, checks a table of policy decisions, fuzzes tool arguments against their schema, and checks that an agent renders the same model request every time, all without a model. See Testing a Consumer. The kit's own tests are testkit/testkit_test.go and testkit/check_test.go, and its Example functions are checked by go test.
Running the nested modules
Each nested module pins a released core, so testing one against the core in the tree needs a workspace:
go work init . ./providers/ollama ./providers/anthropic ./providers/openai ./mcp ./bench ./export/otel ./proxy ./examples/live ./examples/mcp
for m in providers/ollama providers/anthropic providers/openai mcp bench export/otel proxy examples/live examples/mcp; do
(cd "$m" && go vet ./... && go test -race -count=1 ./...)
done
go.work is not committed. Each local provider adapter has one round trip behind the live build tag, which needs a local Ollama server holding qwen3:30b-a3b, skips when nothing answers, and is excluded from CI:
(cd providers/ollama && go test -tags live -count=1 ./...)
(cd providers/openai && go test -tags live -count=1 ./...)
bench has no dependencies, so its tests need nothing but the module. export/otel is tested against the in-tree core through the workspace, with fixtures that run real drivers (paused and resumed, interrupted and taken over, cancelled with a late tool outcome, model-backed with a retry) and check that every event type the runtime emits is mapped to a span, a span event, or a recorded unknown.
The proxy tests need neither a model nor Node: they run its upstream and the proxy itself as copies of the test binary, over pipes, and cover the rules across calls, crashes and restarts, and the two-process cases. go run ./proxy/example is a demonstration with no model that TestDemo_RunsEveryStep also runs.
The mcp tests run a fake server in this process over the SDK's in-memory transports, with no subprocess and no port, except the process-group tests, which start a real child on Unix.
Lint
CI's lint job runs on the newest supported Go. Install the tools at the versions CI pins (.github/workflows/ci.yml names the current ones):
go install honnef.co/go/tools/cmd/[email protected]
go install golang.org/x/vuln/cmd/[email protected]
The core:
gofmt -l . # must print nothing
GOWORK=off go mod tidy -diff # must print nothing
GOWORK=off staticcheck ./...
GOWORK=off govulncheck ./...
Each nested module, through the workspace, runs staticcheck ./... and govulncheck ./.... Its tidy check against its pinned core belongs to the release check, because the workspace overrides that pin.
The release check
.github/workflows/release-check.yml builds each nested module against the core version it pins, with GOWORK=off: go mod verify, go mod tidy -diff, build, vet, go test -race, and govulncheck. It runs on a nested-module tag, weekly, and on demand, and it is the gate before a nested module is tagged. Run it for a commit with:
gh workflow run release-check.yml -f examples=false
Releasing explains why ordinary CI cannot do this job.
The clock-reading contract
The runtime takes exactly one reading of Config.Now per recorded state change, in an order that does not change between releases. TestDriver_ClockReadingsAreStable pins the count on every path: an allowed call, a denial, an invalid decision, completion, a terminal tool, a tool abort, a pause, a resume, and more, against a clock that hands out one second per reading.
That is a contract with consumers, and the reason is a consumer's deterministic clock. A consumer that drives the runtime in tests with a scripted Now, and stamps its own records from the same clock, keys those records on the sequence of readings: reading n is "the step was decided", reading n+1 is "the tool started". If a release took one more reading, or reordered two, every timestamp after it would shift, and the consumer's recorded runs would stop matching. So a change that merges writes, as the v0.3 audit did, must keep the readings it replaced, and settle deliberately reads the clock once for the step and once for the run even though they now commit together.
Lease times come from the wall clock precisely so that leases do not disturb this sequence, and so that a deterministic clock that never advances does not make every lease immortal.
The lease tests' renewal hook
The lease heartbeat renews every third of the TTL, and its renewal timeout is also a third. A test that is about renewal rather than expiry, with a short lease such as 300 ms, still lapsed on a shared CI runner, because one stall of a lease length (slow disk syncs, a starved goroutine) was enough. So those tests hold a lease of one to five seconds and renew every 25 to 50 ms through a test-only interval on the driver (renewEvery). A test that is about expiry switches the heartbeat off (noHeartbeat) and moves a fake wall clock instead. Production behaviour is unchanged: the default lease is 30 seconds, renewed every ten.
Patterns
Scripted agent, in-memory tools. Most tests build a driver with a scripted.Agent and tools that close over a counter or a slice, then assert on the run and the store.
Events as assertions. Tests read the event log and check that the expected sequence was written, which also verifies that events and state are committed together.
Two processes, one file. The lease and concurrency tests open the same database file from two drivers and two stores, as two processes would, and race them.
Fault injection lives in consumers. The runtime has no fault points; consumers inject at their journaled writes and drive Resume.
What a change should come with
A control gets a named test, and docs/status.md points at it. A change to what is recorded gets a test that reads it back after close and reopen. A new limit gets a test that crosses it and checks the reason. A change to a write path keeps TestDriver_ClockReadingsAreStable and TestDriver_CachedViewEqualsReloadedView passing unchanged. Tests are named TestX_Behaviour.