Safety Properties - joeylking/agent-runtime GitHub Wiki
Safety Properties
These are the properties a consumer can rely on as of v0.3.0, which v0.3.1 through v0.5.0 only add to, each with the test that verifies it in docs/status.md. They are stated as what the runtime guarantees and what it leaves to the consumer. Security Model covers the same ground organized by attacker.
Guaranteed by the runtime
No tool runs without passing its schema and the policy. Arguments are validated against the tool's JSON schema; the policy sees every validated request; the agent has no handle to either. An outcome the runtime does not know fails the run, in the loop and on resume alike. (TestDriver_InvalidArgumentsBecomeObservation, TestPolicy_UnknownOutcomeFailsLoopAndResume)
No partial tool call executes. A reply cut off by the output cap or refused reaches the runtime with no tool uses, and render.Decide records it as an invalid decision kind the driver will not act on. (TestDriver_PseudoDecisionKindsAreInvalid)
The policy sees the truth, and only a copy of it. It receives the run, every step, and every approval as recorded, in a copy no consumer can alter for the driver or for the other consumer. Facts it derives from the view cannot drift from the log. (TestDriver_ConsumersCannotCorruptEachOther, TestDriver_CachedViewEqualsReloadedView)
Every decision is recorded before it is validated. An invalid or denied request appears in the log with the agent's stated reason; arguments that were not JSON are kept verbatim out of band, never stored as if they were arguments. (TestDriver_InvalidArgumentsAreRecordedOutOfBand)
What is stored or hashed is never a stand-in. Consumer JSON that is malformed, invalid UTF-8, nested deeper than 256 levels, or carries a repeated key or an unpaired surrogate is refused where it enters; a value that will not encode fails the write that needed it and the run fails as internal_error with nothing more executed. The canonical encoder is pinned byte for byte to the hashes already stored. (TestCheckJSON_RejectsWhatWouldHashLossily, TestDriver_DeepJSONIsRefusedWhereItEnters, TestApprovalHash_ErrorsRatherThanSubstituting, TestContentHash_Golden)
State and audit are written together. Every transition and its event commit in one transaction; an observer sees an event only after commit. A step and the run it ends commit together, as do a resume, its policy decision, and the tool start it approved. (TestDriver_CompletesAndPersists, TestDriver_TerminalToolCommitsWithTheRun, TestResume_CrashAfterResumeKeepsTheApprovedRequest)
Transitions are compare-and-set. Of two processes racing to resume one approved run, one executes its side effect and the other gets ErrRunState; an operator's cancel is never overwritten by a live loop. (TestResume_ConcurrentResumeExecutesOnce, TestCancel_LiveLoopStopsAtItsNextWrite)
A run in progress belongs to one process. A RUNNING run is leased to the driver executing it; another process's Resume is refused with ErrRunLeased while the lease is live; a loop that loses its lease, or lets it lapse, starts no further tool call or model request. (TestLease_ResumeOfALiveRunIsRefused, TestLease_LoopThatLosesItsLeaseStopsBeforeTheNextToolCall, TestLease_LapsedLeaseStartsNoSideEffect, TestLease_SameLeaseOwnerNameIsTwoOwners)
A pause is bound to the request, and the decision to what was shown. An approval hashes its kind, capability, presentation, and the exact request. A tampered row is refused. On resume, only the step's latest approval can be a grant, the recorded arguments must still pass the tool's current schema, and the recorded request executes only if policy still allows it or asks for precisely the granted approval. A decision made through ApproveShown or RejectShown, as cmd/agentrt makes it, applies only if the stored approval is still the one the operator read. (TestApproval_TamperedRowRefused, TestResume_OnlyTheLatestApprovalOfAStepGrants, TestResume_ApprovedArgumentsMeetTheCurrentSchema, TestApproval_ShownDecisionIsAtomic)
Limits are enforced by the driver. Steps, consecutive failures, loops, model calls, tokens, cost, active time, elapsed time, approval expiry, and grant expiry end the run with a named reason. Token and cost limits stop before the call that would cross them, a request that cannot be projected is refused under them, and a retry never sleeps past a time limit. (TestModel_TokenAndCostLimitsProjected, TestModel_UncappedRequestRefusedUnderAProjectedLimit, TestModel_RetryWaitThatCannotFitIsUnavailable, TestResume_GrantNotResumedWithinTheTTLExpires)
Every model attempt is accounted, and usage is not believed blindly. A row is reserved before dispatch; retries, ambiguous attempts, and served-but-unusable replies are counted and charged; implausible usage is charged the conservative estimate; totals saturate rather than wrap. (TestModel_AmbiguousAttemptChargedConservatively, TestModel_ServedErrorIsChargedAndNotRetried, TestModel_UnusableUsageIsChargedTheEstimate, TestModel_TotalsSaturateRatherThanWrap)
A side effect is never re-executed by the runtime, nor re-requested without someone deciding. An interrupted step is marked; the consumer reconciles; without a reconciliation, a call that was executing a tool that is not read-only waits for an operator, and is run again only if the operator approves, once, on the same step. (TestResume_InterruptedStepIsReconciledAndContinued, TestResume_InterruptedSideEffectWaitsForAnOperator)
What a side effect did is recorded even when the caller gave up. A tool's outcome and a dispatched model call are recorded on a context that outlives the caller's cancellation; a tool that finishes after an operator's cancel is appended as a late event. (TestDriver_CancelledContextAfterToolKeepsTheObservation, TestCancel_ToolOutcomeIsAppendedLate)
A terminal run stays terminal. Resume refuses completed, failed, and cancelled runs.
A damaged record does not wedge its run or get continued from. A row that will not decode is listed with DecodeError; the run can be shown and cancelled; a driver fails it rather than continuing. (TestStore_UndecodableRowsDoNotWedgeTheRun)
The database is the operator's. It is created owner-only; one others can write is refused; a trigger or a view is refused; every connection runs with trusted_schema off and defensive mode on; every commit is synced. (TestStore_FilesAreOwnerOnlyUnderAnyUmask, TestStore_RefusesADatabaseOthersCanWrite, TestStore_RefusesTriggersAndViewsAndHardensItsConnections)
What the operator reads cannot be rewritten by the model. The operator command sanitises model- and server-chosen text, bounds each field, reads only the page it prints, and ends each approval with a line of its own. (TestShow_BidiAndCombiningMarksInArgumentsEscaped, TestShow_LongArgumentTruncatedHeadAndTail, TestRead_OnlyThePageIsReadFromAHugeDatabase)
Keys go where they were meant to. No adapter follows a redirect; a key goes over plain http only to loopback; keys are redacted from every encoding of a config or model. (TestGenerate_RedirectIsRefusedAndTheKeyStaysHome, TestLoopbackOnly_RefusesAConnectionThatLeavesTheMachine, TestConfigAndModel_EncodeWithoutTheKey)
An MCP server classifies nothing and cannot reach outside its schema. Tools need an operator rule; the pin covers schema and description from the raw bytes; a schema that refers outside itself is refused. (TestLoad_UnclassifiedToolIsNotRegistered, TestLoad_DescriptionChangeRefusesTheTool, TestLoad_RefusesASchemaThatReachesOutsideItself)
Left to the consumer
- Containment inside tools: paths, hosts, processes. The runtime bounds which tool runs with which arguments; the tool bounds what those arguments can reach.
- Domain policy: which paths are protected, what scope means, which targets are eligible.
- Journals for side effects, and the reconciliation that reads them; or accepting that an operator decides an interrupted side effect.
- Keeping untrusted content out of the system prompt and labelled as data in tool results.
- Sandboxing anything that executes untrusted code, including MCP stdio servers.
- With a gate, the loop itself: the gate bounds what a caller's loop can make happen, not what it proposes, and a tool the caller runs outside the gate is outside these properties. With the MCP proxy, what the host gives the model besides the proxy.
- Secrets: the runtime stores decisions, arguments, observations, and presentations verbatim and redacts nothing in them.
Not claimed
Protection against a compromised host, root, or the database owner's own account. A database on a network filesystem, or shared between hosts whose clocks disagree. An older binary that ignores leases. A provider that misreports usage within plausible numbers. What an MCP server does when called. Any property of the model's output beyond what the policy checks.