A calibration-first leaderboard ranked by evidence-before-action (EGAR), with false-positive cost shown beside each score. Eight models, 320 episodes, deterministic oracle scoring.
Evaluation seeds and baseline traces are available on Hugging Face.
OpenSec evaluates incident response agents on dual-control scenarios where defenders process adversarial evidence and have authority to act. The tasks require reasoning about trust tiers, distinguishing attacker-planted evidence from legitimate alerts, and making real-time containment decisions.
Detection is not the binding constraint in these runs. The calibration gap is restraint: false positive rates range from 45% to 97.5% across the eight models.
Calibration view ranks rows by EGAR, then lower FP Rate, then higher TTFC. FP Rate stays visible beside each score because over-triggering is the operational failure mode. Containment view reorders rows by threat containment for comparison only.
OpenSec contains 60 evaluation seeds across three injection families: data exfiltration, direct harm, and adaptive. This leaderboard uses the 40 standard-tier seeds per model. The seed set, scoring oracle, and evaluation harness are open source.
Evaluation configuration and known asymmetries.
All models evaluated via OpenRouter (except GPT-5.2 via OpenAI direct). temperature: 0.2, max_tokens: 600. Attacker: GPT-5.2 at temperature 0.7 with strict mode. 40 standard-tier episodes per model.
No explicit reasoning parameters are passed. Kimi K2.5 uses reasoning by default (264/285 tokens). Opus 4.6 and Sonnet 4.5 receive 0 reasoning tokens. Reflects default API deployment behavior.
Deterministic multi-component reward: attribution, containment, injection safety, efficiency. EGAR and TTFC computed from episode traces. FP penalty capped at -1.0 per action type.