The benchmark record.
Measured outcomes and execution status for the Soterias SDK. AgentDojo runs across workspace, Slack, travel and banking, with 97 selectable tasks. The snapshot below shows the domains actually measured. Six additional suite boundaries remain experimental and have no scored runs.
Executions
Job steps refresh after the publisher uploads a new snapshot, while a benchmark is running. Open an execution for request and quota logs. Detailed logs require repository access.
Measured outcomes
Request accounting
Scored outcomes
| Condition | Soterias | Scored cases | Task completion | Attack success | Mean tokens | Agent latency |
|---|
A failed attacker objective can result from model behaviour. DENY and ESCALATE counts describe enforcement decisions. Group means are descriptive; incomplete pairs cannot establish efficacy.
Suite readiness
AgentDojo: workspace (40 tasks), Slack (21), travel (20) and banking (16). Routine CI uses a balanced 12-task sample (48 cases), plus specifically affected tasks. Nightly runs add rotating tasks (64 cases); weekly coverage includes all 97 tasks (388 cases). The full 2,092-case attack grid is available for manual or release runs. These are unchanged public AgentDojo tasks and graders; sampled results are not full-benchmark scores. Other benchmark suites remain experimental. Policies allow tenant reads and reviewed explicit operations; unresolved writes require approval. Approval-dependent tasks can lose utility; no automatic approval is fabricated.