Rethinking Evaluation for Agents That Never Stop Learning
Online evaluations turn production traces into verified frontier tasks with calibrated difficulty, while anchor sets keep progress comparable as agents improve.
Writing
Research artifacts
Mechanistic Interpretability for Multilingual LLMs
Tiny Aya probes show how multilingual models route language into target-script, format, entity, and stopping behavior before decoding.
Self-Improving Pretraining as a Substrate for Agentic Post-Training
I adapted self-improving pretraining, interleaved-thought SFT, and RL mid-training to Qwen3-0.6B-Base.
Do Language Models Know When to Change Their Mind?
Signal detection framework for LLM belief revision across 6 open-weight models.
Where should test-time compute go? Surprisal-guided selection in verifiable environments
Surprisal-guided selection routes test-time compute toward high-value candidates, matching oracle selection with far fewer evaluated samples.
Frontier Security Agents Lack Restraint
Frontier security agents detect real threats but over-trigger containment, exposing a calibration gap between detection and restraint.
When Sampling Beats Training: Multi-Turn RL's Cost-Benefit Problem
Verifier-based sampling can beat a narrow RL gain when deployment can afford multiple attempts, changing the cost-benefit case for multi-turn training.
Essays and working notes
A Theory on Becoming an Expert
Becoming an expert means deliberately building the mental architecture to judge, question, and understand what models generate.
Building a World Model of Consequence
Browser agents need explicit consequence models that predict how actions change state, enabling planning, credit assignment, and transferable learning.
My Agents Keep Failing. Yours Will Too.
Production agents need shared learning loops where failures become reusable experience across the network instead of one-off human patches.
Everything is Changing...Again
Institutional knowledge becomes dynamic when every diff, decision, and correction is searchable, reviewable, and available at the moment of use.