Latent Mining - benchmark construction from public model uncertainty

Leaderboard

Six evaluated agents solve 9.1–14.5% of the tasks.

Each task asks an agent to choose one regulatory-DNA edit for an assay and cell context. Strict solve means clearing a hidden, cached model-backed threshold, not a wet-lab measurement. Solve-plus-near-miss also counts choices close to that target.

May 31–June 4, 2026 · 165 tasks per agent · 10 candidates per task

#	Agent	Solve	Near	Fail	Solve rate	Outcome distribution

Solve Near-miss Failure 990 retained runs · model-backed grading

Interpreting the rate. No random-choice baseline is reported here. Ten candidates alone do not establish a 10% strict-solve baseline: that depends on how many clear each hidden threshold.

Harness conditions. API agents had a 131,072-token output cap and 64 tool calls per task; coding CLIs used their native harness budgets. Gemini and GLM used temperature 0; the other sampling defaults were not recorded. Kimi combines native API and Router runs. Appendix G documents the protocol.

Retained attempts. Timed-out or interface-invalid attempts were manually rerun; the last valid attempt was retained per task. All 990 retained runs passed the interfaces, with no timeouts or nonzero exits in that final matrix.

One public task

Choose an edit at the HBB/HBE locus.

In task apsci_triage_672fd0911af7, the agent compares 10 candidate edits for a ChIP objective in erythroblasts. The decision concerns a TF-bound locus-control-region enhancer on chromosome 11.

Inspect

Read candidate edits, local sequence context, sequence-model signals, and public motif and regulatory features. The task explicitly warns that the largest Carbon perturbation need not be the target.

Commit

Select one candidate, cite at least two public evidence channels, reject at least two alternatives, and give a rationale.

Grade

The public validator checks answer structure. A separate cached AlphaGenome objective grades the choice without exposing its scores to the agent.

Read the public task specification · Explore the task atlas and full release bundle

Method

Verifier-Anchored Task Discovery

Latent Mining is a data-generation and benchmark-construction method organized around one question. Where do model-internal or model-disagreement signals reveal an unresolved scientific transition that can be packaged as a task? It starts from a workflow decision a scientist would face, reads the release-safe evidence that decision would draw on, and mines it for the cases where public signals point somewhere but do not settle the answer.

Only after mining does a hidden verifier enter and grade each candidate privately. A task is admitted when the hidden answer is well separated from the attractive public alternatives, and when no single public score, leaked field, or metadata artifact can recover it. Leakage scans and shortcut probes reject or repair any task that an agent could solve by cheating rather than by judgment. Plausibility alone never admits a task.

For this release, we applied Latent Mining to scientific workflows within computational biology, starting with regulatory-intervention triage. The agent compares candidate noncoding edits for one assay and cell context, weighing sequence-model scores, motif and chromatin annotations, locus context, and disagreement across public signals. Each signal carries information about the hidden regulatory effect and none recovers it, which makes the decision a calibration problem rather than a lookup.

Every admitted task answers one design question. Can we name the lab workflow decision, the public evidence, the hidden verifier, the failure label, and the replayable trajectory? A panel enters the benchmark only when all five are present, which keeps task construction from collapsing into generic question answering.

Latent Mining loop diagram with three stages. Stage one mines visible evidence from a workflow decision. Stage two admits and writes tasks through hidden-verifier and shortcut checks. Stage three evaluates agent choices and traces under private grading, with a diagnostic-replay feedback arrow. — The method mines visible evidence from a workflow decision, admits and writes tasks through hidden-verifier checks and public shortcut repair, then evaluates agent choices and traces under private grading. The withheld outcome and failure label support diagnostic replay without exposing hidden verifier values. Open the full-resolution diagram.

01

Define the workflow decision

In this biology release, the agent must choose the candidate regulatory edit most likely to improve a hidden regulatory outcome for a specified assay and cell context. That keeps the task focused on regulatory-genomics triage.

decision · constrained action

02

Generate candidate interventions

We use domain models and retrieval to produce plausible candidate edits, drawing each from a different view of the intervention space. The output is the comparison set for one task, several edits that all look defensible on some public signal. No source is trusted as the answer. They exist to populate a slate of credible alternatives.

generate · candidate space

03

Expose imperfect public evidence

Each task shows public evidence from sequence-model likelihood summaries, embedding-distance features, motif and PWM changes, edit geometry, locus metadata, assay family, chromatin context, and disagreement across public signals.

Difficulty comes from forcing the agent to weigh signals against each other. No single public score, motif heuristic, source model, or generic mechanism story is enough to solve an admitted task.

expose · imperfect evidence

04

Hold out verifier-backed outcomes

Candidate edits are scored by a private, model-backed objective the agent never sees. In this release it is a hidden regulatory-effect estimate for the requested assay and cell context, fixed during construction before any agent runs.

We treat this as a verifier for building and grading tasks, not as experimental truth. A selected edit still needs lab validation, and a grading disagreement can reflect the verifier rather than the agent.

hold out · private objective

05

Build contrastive panels

Each task is a panel of candidate edits the agent compares directly. The distractors are deliberate, an outcome-proximal near-miss, an edit that a single public scalar makes look strongest, and same-context variants.

The panel is built so the attractive public proxy and the hidden answer point to different edits, which is the exact commitment where agents fail.

build · contrastive panels

06

Audit and calibrate

Every task is checked for leakage, shortcut features, public-interface validity, grading-interface validity, and calibration behavior across frontier agents.

The benchmark studies the gap between noticing public regulatory evidence and using it to make the right intervention decision.

audit · calibration + traces

What latent means

The latent object is the boundary between public biological priors and withheld verifier-backed outcomes. We mine that boundary because it reveals where current scientific models produce signals that are useful enough to reason with, but incomplete enough that agents still need calibrated domain judgment.

Public evidence the agent sees

sequence-model likelihoodembedding distancemotif / PWMedit geometry locus metadataassay familychromatin context public-signal disagreement hidden objective · model-backed regulatory verifier

Agent trace behavior

Noticing without control.

Frontier agents often identify relevant regulatory evidence, including motif shifts, sequence-model priors, locus context, assay constraints, chromatin evidence, or model disagreement. The failure happens when those observations do not control the final intervention choice. 640 of 878 non-solves were labeled as public scalar collapse, the largest diagnostic failure category. These annotations describe recorded decisions; they do not establish a causal mechanism.

Noticing without control

The agent mentions a relevant public signal, but the final answer is driven by a more salient proxy, such as a motif change, a single model score, or the source that made one candidate look strongest.

Explanation instead of triage

The agent writes a plausible regulatory story for one candidate, but does not compare that candidate against the alternatives under the requested assay and cell context.

Qualitative audit

Aggregate solve rate tells us how often agents choose the hidden-verified target. Trace analysis distinguishes annotated patterns of missing evidence, proxy reliance, and weak comparative decision control. Low solve rates alone cannot distinguish reasoning failure from insufficient public information about the private objective.

Performance and behavior by workflow →

Links