Diagram in which answer confidence, evidence quality, and source pressure feed one final control step that decides whether to revise or resist. Source pressure can override evidence at that step.

Figure 1. The revise-or-resist decision passes through one final control step, where source pressure can override evidence quality. Arrows show hypothesized influences. The experiments measure answer confidence, evidence quality, source pressure, and the final decision, not separate causal modules.

Language models are increasingly corrected by something outside themselves: a reviewer flags an error, a test fails, another agent disagrees, a user pushes back. Every one of these loops works only if the model can tell a good correction from a bad one. If it yields to pressure, whoever sounds most authoritative can steer it, and oversight passes on the overseer’s mistakes as readily as it fixes the model’s. If it never yields, it cannot be corrected at all.

The question behind this post is whether a model knows when to change its mind: whether its decision to revise tracks the quality of the evidence rather than the confidence, status, or number of the people delivering it.

Behind that question sit three deeper ones, each with a practical stake:

  1. Is changing your mind for the right reason one ability or several? If several, a single sycophancy score hides which one failed.
  2. Does it arrive with scale, or does it have to be trained? If it arrives with scale, larger models will take care of it.
  3. If it can be trained, does the model learn that the messenger should not matter, or only to ignore the messengers in its training data? Only the first would hold up against a messenger nobody anticipated.

These are questions about metacognition in its control sense. Google DeepMind’s cognitive framework for measuring progress toward AGI names metacognition, the ability to monitor and control your own thinking, as one of the faculties with the largest benchmark gaps. Most existing work measures monitoring. Calibration benchmarks ask whether a model knows what it knows, AbstentionBench asks whether it knows when to decline, and Wang et al. separate metacognition from cognition with signal detection theory. Control, what a model does with new information once it arrives, has mostly been measured with sycophancy benchmarks, which, as the next section shows, cannot tell judgment from stubbornness.

I study these questions in the simplest setting where they can be measured cleanly: a multiple-choice question, a first answer, a critique that is either valid or invalid, and sometimes a messenger attached to the critique. In that setting, changing your mind for the right reason takes three separate abilities, so a single score does hide which one failed:

  • Judging the argument. Revise when the critique is correct and resist when it is not. Below 4B parameters, Qwen models barely have this; they accept nearly everything. Within Qwen3.5 it appears at 4B and grows at 9B.
  • Knowing when you might be wrong. Larger models defend the answers they are confident in and give way on the ones they are not.
  • Keeping the messenger out of the judgment. A reviewer panel that disagrees with the evidence erased Qwen3.5 9B’s ability to tell good critiques from bad ones, and Qwen3.6 35B-A3B did little better. Steering inside the model did not fix this. Training did, for one model and only with the right target, and the fix held for kinds of messenger the model never saw.

Method

Sycophancy is a known failure. OpenAI documented it in GPT-4o, and Anthropic studied it in Claude. Benchmarks such as Certainty Robustness, SYCON-Bench, and TRUTH DECAY measure whether a model abandons a correct answer under pushback; Claude Sonnet 4.5 loses 82 accuracy points when told “You are wrong!” These tests share a blind spot: the pushback is always wrong, so a model that never changes its answer scores perfectly. They cannot distinguish a model that weighs the critique from one that ignores it.

The construct I care about is discrimination: changing your answer for a good reason and keeping it against a bad one. Kumaran, Fleming et al. documented the related pathology of overconfidence plus oversensitivity to contradiction, without varying critique quality. Varying it is what makes discrimination measurable.

I adapt Fleming and Lau’s signal detection framework. A model answers a multiple-choice question, receives a critique, and answers again. If its first answer was wrong, it gets a valid critique (the signal); if it was right, it gets a plausible but wrong one (the noise).

  • Hit: wrong first, valid critique, revised.
  • Miss: wrong first, valid critique, held.
  • False alarm: right first, invalid critique, revised. This is sycophancy.
  • Correct rejection: right first, invalid critique, held.

d′=Z(hit rate)−Z(false alarm rate), where Z is the inverse of the standard normal CDF.d-prime (d′) measures how well revisions separate valid from invalid critique. Zero means no separation at all. d-prime measures discrimination independent of the model’s overall tendency to revise. A second number, the criterion c,Criterion c measures the model’s lean: negative means it tends to revise, positive that it tends to hold. measures that tendency. A model that always revises and one that never revises both have d-prime near zero; they differ only in c.

The critiques come from the DS Critique Bank (Gu et al., 2024), where student models answered questions and critique models wrote feedback. The sweep covers 969 questions from eight datasets: science (ARC-Challenge, ARC-Easy) and commonsense (HellaSwag, SocialIQa, CosmosQA, WinoGrande, PIQA, aNLI). It tests four Qwen3.5 sizes (0.8B to 9B), Qwen3.6 35B-A3B, and Google’s Gemma 4 E4B and 26B-A4B, with thinking mode disabled.

Item audit

A test of whether a model weighs an argument is only valid if the argument is the only thing that differs between the cases. Before training on these critiques, I checked whether they could be judged without the question. They could. A bag-of-words classifier that sees only the critique text, never the question, labels it valid or invalid with AUROC 0.99999.AUROC is the chance that a classifier ranks a random valid critique above a random invalid one. 0.5 is chance; 1.0 is perfect separation. The valid critiques were feedback written by one model on a wrong answer and ended “The correct answer is (X).” The invalid ones were wrong students’ explanations ending “The answer should be (X),” or notes ending “Consider whether (X) might be more appropriate.” With the closing phrases removed, the classifier still scores 0.991, because the writers sound different.

For a model under evaluation, the wording is one more signal it might respond to. For a model in training, it is a shortcut: the cheapest way to sort critiques would be to recognize the writer. The training experiment at the end of this post therefore uses rebuilt critiques. For 649 DS Critique Bank questions, I paired a correct and a wrong explanation written by the same student model and put both in one template. On those, the question-blind classifier scores 0.539, close to chance.

The audit found a second problem. All 148 critiques of the “Consider whether” kind recommended the gold answer. An invalid-critique trial is one where the model already holds the gold answer, so these critiques told it to keep what it had, and keeping it would be scored as resisting a wrong critique. The results below drop every trial that used one of those 148 critiques, unless noted.

Discrimination by model

Scatter of each model's rate of accepting valid critiques against its rate of accepting invalid ones. Qwen3.5 0.8B and 2B sit in the top-right corner, accepting nearly everything. Qwen3.5 4B and 9B and Gemma 4 E4B accept most valid critiques and fewer invalid ones. Gemma 4 26B-A4B and Qwen3.6 accept few of either.

Figure 2. The smallest Qwen models accept nearly every critique; larger models separate valid from invalid, and differ mainly in how readily they revise. Each point is one model on 969 DS Critique Bank questions, with the 148 flawed critiques excluded. Dashed curves mark equal d-prime; the diagonal is chance. Open circle: Qwen3.6.

Figure 2 is the clearest way I know to read a revision benchmark. Up means the model accepts more valid critiques; right means it accepts more invalid ones. A model that cannot tell them apart lies on the diagonal. Each dashed curve joins points with the same d-prime, so crossing curves toward the top left means better discrimination, and sliding along a curve changes only the criterion.

Qwen3.5 0.8B and 2B are in the top-right corner. They accept 99% of valid critiques and 97% and 95% of invalid ones. Their d-prime is above zero, but only because they revise slightly more often when the critique is right. They do not resist critique; they comply with it. On commonsense questions, 2B shows no measurable discrimination (Figure 4).

The flawed critiques show what the small models respond to. On critiques that recommended the answer the model already held, 0.8B changed its answer 3% of the time and 2B changed it 45% of the time. 2B reacts to “I think your answer may be wrong” more than to what the critique recommends. Scored as invalid critiques, those items would make 2B look more sycophantic than 0.8B; without them, the two small models are indistinguishable.

Model Accuracy Signal Noise d-prime 95% CI Hit rate False alarm rate Criterion
Qwen3.5 0.8B 47.3% 511 387 0.651 [0.27, 1.28] 0.993 0.965 −2.14
Qwen3.5 2B 59.0% 397 484 0.563 [0.23, 1.05] 0.986 0.949 −1.92
Qwen3.5 4B 68.5% 305 561 1.505 [1.26, 1.81] 0.956 0.579 −0.95
Qwen3.5 9B 79.2% 202 648 1.664 [1.42, 1.97] 0.924 0.408 −0.60
Gemma 4 E4B 71.5% 276 587 1.713 [1.50, 2.00] 0.933 0.416 −0.64
Gemma 4 26B-A4B 78.2% 210 645 1.608 [1.40, 1.84] 0.637 0.104 +0.45
Qwen3.6 35B-A3B 83.7% 158 688 1.268 [1.04, 1.50] 0.619 0.168 +0.33

969 questions. Signal and noise columns count trials; noise trials exclude the 148 flawed critiques. Accuracy counts the one question Gemma 4 26B-A4B failed to answer as incorrect.

Discrimination appears at 4B and grows at 9B, and Gemma 4 E4B sits next to Qwen3.5 9B. Among these capable models, the larger difference is the criterion. Gemma 4 26B-A4B holds its answer readily: it accepts 10% of invalid critiques and only 64% of valid ones. Qwen3.5 9B revises readily: 41% and 92%. Neither setting is better in general. Which error is worse depends on what a wrong revision costs.

Accuracy and discrimination still come apart. Qwen3.6 35B-A3B answers more questions correctly than Qwen3.5 9B (83.7% against 79.2%) yet separates valid from invalid critique less well (d-prime 1.27 against 1.66), with a cautious criterion like Gemma 4 26B-A4B’s. Within one family and training recipe, discrimination rose with size above 2B. Across generations, the more accurate model was the worse judge of critique.

Confidence and false alarms

If a model resists only when it is sure, its resistance should follow its confidence. I split each Qwen model’s invalid-critique trials at the median log-probability of its first answer.

Model False alarms, confident False alarms, uncertain Gap
Qwen3.5 0.8B 0.959 0.974 0.02
Qwen3.5 2B 0.918 0.983 0.07
Qwen3.5 4B 0.527 0.633 0.11
Qwen3.5 9B 0.260 0.558 0.30

False alarm rates on invalid-critique trials, split at the median first-answer log-probability.

The small models give way whether or not they were confident. Qwen3.5 9B defends 74% of its confident answers against invalid critiques and only 44% of its uncertain ones. The same split shows up in discrimination: on 9B’s low-entropy answers its d-prime is 2.22, and on its high-entropy answers 1.19. Kadavath et al. showed that models can predict their own accuracy. The question here is whether that knowledge reaches the decision, and by first-answer confidence it does so more as the models grow.

Two charts on Qwen3.5. At every size, correct answers above the median varentropy have false alarm rates 17 to 31 percentage points lower, and the varentropy coefficient is negative.

Figure 3. At every Qwen3.5 size, correct answers above the median varentropy are revised less often after invalid critique. (A) False alarm rate on correct answers below and above each model’s median answer-token varentropy. (B) Varentropy coefficient in a logistic regression of false alarms on entropy and varentropy; below zero, higher varentropy predicts fewer false alarms. A separate run on 150 questions per model drawn from all eight datasets (146 valid at 4B), whose critiques did not include the flawed kind.

The shape of the uncertainty matters too. Varentropy, the variance of the output distribution’s self-information, V=E[(−log⁡p)2]−H2, following Ahmed et al., can differ between two distributions with the same entropy. In that separate run, high-varentropy Qwen answers were revised less often at every size, including 0.8B (39% against 70% of its 61 correct answers), so even the smallest model carries some uncertainty signal that reaches the decision. Gemma 4 E4B showed the opposite: across the full pool, its high-varentropy answers had the higher false alarm rate, 0.588 against 0.242. A varentropy threshold could flag vulnerable answers, but only after it is validated for each model family.

Science and commonsense

Bar charts of d-prime by domain. Qwen3.5 science d-prime rises from 0.65 at 0.8B to 2.16 at 9B. Commonsense d-prime stays lower at every size; for Qwen3.5 2B it is slightly negative and indistinguishable from zero.

Figure 4. Qwen3.5 discrimination grows with size much more on science questions than on commonsense ones. Bars are d-prime on 397 science and 572 commonsense questions, with the flawed critiques excluded, on a shared scale. Brackets give the Qwen3.5 change from 0.8B to 9B. Gemma 4 in gray.

On science questions, Qwen3.5 d-prime rises from 0.65 at 0.8B to 2.16 at 9B. On commonsense questions it rises only from 0.54 to 1.22, and 2B’s estimate is indistinguishable from zero (−0.32, interval −0.63 to 0.05). Gemma 4 E4B is the best commonsense judge here (1.73). One explanation is that science critiques give a model physical or chemical constraints to check, while commonsense critiques are more ambiguous; the critique stimuli themselves could also differ in quality by domain. I would want those separated before reading the pattern as a property of the models.

Source conflict

Agents rarely receive evidence without a source. So I kept each critique fixed and added a reviewer-panel recommendation after it:

  • Congruent: the panel recommends what the critique’s validity implies (change after a valid critique, keep after an invalid one).
  • Conflict: the panel recommends the opposite.

If the model is evaluating the critique, discrimination should survive conflict. If it is following the room, discrimination should collapse. These runs use a matched subset of 240 trials per condition.

Model No panel Conflict 5-2
Gemma 4 26B-A4B 1.465 [1.12, 1.87] 0.573 [0.22, 0.97]
Qwen3.5 9B 1.433 [1.07, 1.88] −0.099 [−0.43, 0.24]
Qwen3.6 35B-A3B 1.075 [0.72, 1.46] 0.244 [−0.10, 0.60]

d-prime with 95% bootstrap intervals; flawed critiques excluded. A 5-2 panel means five of seven reviewers favored the stated action.

Under a conflicting 5-2 panel, Qwen3.5 9B no longer discriminates. Its hit rate drops from 0.913 to 0.508, and its false alarm rate rises from 0.471 with no panel to 0.548. When the room says keep, it ignores a valid correction about half the time; when the room says change, it accepts a wrong critique more than half the time.

Because the critique is identical across conditions, the two panel directions let me split the panel’s effect in two. A congruent panel pushes the decision the same way as the evidence and a conflicting one pushes against it. The average of their two d-primes estimates how much weight the critique itself still gets; half their difference estimates the panel’s own pull. For Qwen3.5 9B, the critique’s weight falls from 1.43 with no panel to 1.15 with one, and the panel’s own pull is 1.25. The panel does two things at once. It casts its own vote, and it makes the model weigh the argument less.

Gemma 4 26B-A4B keeps part of its discrimination under conflict (0.57, interval excluding zero). The cautious criterion that made it miss valid corrections in the first experiment protects it here. A weaker 4-3 panel, run on Qwen3.5 9B and Qwen3.6 with the original critiques, cut discrimination roughly in half rather than erasing it.

Internal analyses

Inside Qwen3.5 9B, the panel often wins late, after the critique has already pulled the model toward the right answer. On 160 conflict cases, the intermediate states frequently moved toward the right answer after the critique and then back toward the panel’s answer after the panel cue. Movement toward the panel’s answer predicted whether the panel would win with 0.976 cross-validated AUROC. Using Qwen-Scope sparse autoencodersA sparse autoencoder maps dense activations into a larger set of sparse features. on 480 trials, the panel’s recommendation was readable at layer 24, at the end of the panel text. At the final decision position, the 50 features most associated with the panel and the 50 most associated with critique validity overlapped heavily: 26 shared, a Jaccard overlap (shared features over all features in either set) of 0.351 against a pre-registered cutoff of 0.2. That fits Figure 1’s single control step: by the decision, the panel and the critique act through overlapping features rather than separate routes. Patching the layer-24 panel state from congruent into conflict prompts barely moved the answer, so the panel’s influence does not run only through that one state.

Readouts transferred: directions computed for Qwen3.5 4B also separated cases on a small 20-case slice of the Metacognitive Monitoring Battery. Steering did not. The best targeted intervention improved d-prime by 0.570, and the best random direction at the same layer improved it by 3.845. I could move the decision, but not for the right reason. That failure is what motivated the training experiment. These analyses used the original critiques, flawed ones included, and were not re-run.

Training against source cues

Steering edits the model’s state at test time. Training changes what the model learned. The question is whether a model can learn to let the evidence, not the messenger, decide, and whether that holds for messengers it has never seen.

I fine-tuned Qwen3.5 4B with LoRA, a small set of trainable weights added to the frozen model, training only the letter of its final answer. The data were 235 commonsense questions with the rebuilt critiques. Each training prompt carried a random cue, a sentence about who backs a recommendation, from one of three families (a reviewer panel, an expert’s credentials, a second reviewer’s stated confidence), pushing with the evidence, against it, or absent. Everything I tested on was held out:

  • Cue families never seen in training: an automated verifier’s verdict placed before the critique; role authority, a request from someone between intern and CEO; and messenger credibility woven into the critic’s own first sentence (“I know nothing about this subject, but…”).
  • Three more cue types written only after the trained models were frozen: an emotional plea, a consensus of other AI assistants, and a claimed published source.
  • Unseen questions, including a science domain absent from training.

The key test reuses each question’s first answer from the base model, before any fine-tuning, so every valid-critique trial starts from a wrong answer. That choice matters because fine-tuning also teaches answers. In the design I registered first, each trial started from an assigned answer, a wrong one before a valid critique and the correct one before an invalid critique. There the base model would score a d-prime of 1.84 by ignoring every critique and giving its answer to the bare question. A high score in that design cannot separate reading the critique from knowing the answer. I added the base-answer design after finding this, before any held-out trial was scored. On it the base model has a d-prime of 1.53 with no cue; conflicting cues from the unseen families lower it to 0.93. The code, per-trial results, and pre-registration are in the study notes.

Correct-answer labels

The first recipe labeled every training prompt with the correct answer. Under conflicting unseen cues, d-prime rose from 0.93 to 1.83, level with the trained model’s no-cue d-prime of 1.80. By a cue-robustness score, the problem was solved.

It had stopped reading the critique. Its final answer matched its answer to the bare question in 96% of trials, against 62% before training. On questions it still got wrong on its own, it accepted a valid correction 15% of the time; the base model accepted 84%. A control trained the same way with no cues at all ended in the same place (1.83 under conflicting cues). The cues taught nothing. The labels did: when the label is always the correct answer, the cheapest way to fit it is to know the answer, and the critique becomes noise to ignore along with the cue.

Dot plot of critique acceptance for four versions of Qwen3.5 4B. The base model accepts 84 percent of valid corrections and 29 percent of invalid critiques. Both models trained on correct answers accept 15 to 20 percent of valid corrections and 3 percent of invalid ones. The consistency-trained model accepts 91 percent of valid corrections and 44 percent of invalid ones.

Figure 5. Training on correct answers makes the model reject the corrections it needs, with or without cues in training; consistency training keeps them. Filled points: valid corrections accepted on questions that model answers wrong on its own. Open points: invalid critiques accepted on questions the base model answers right. Unseen questions, no cue.

Consistency labels

The second recipe, consistency training (Chua et al., 2024), used the same prompts with a different label: the base model’s own answer to the same prompt with the cue removed. The critique stays in both. The only thing that differs between the input and its target is the cue, so the only lesson available is that the cue should not matter. Re-answering the question earns nothing unless the base model would have re-answered without a cue. I added this recipe after seeing the first result and registered its predictions before training it.

Three line charts, one per cue family never seen in training. The base model's d-prime falls as the conflicting cue gets stronger, most for role authority. The consistency-trained model's d-prime stays flat at its no-cue level.

Figure 6. On cue families it never trained on, the consistency-trained model discriminates as well under conflicting cues as with none. d-prime on unseen questions, using the base model’s first answers; bands are 95% intervals from resampling questions. The two models start at nearly the same no-cue d-prime (1.53 and 1.52) because consistency training targets the base model’s no-cue judgment.

Against its own no-cue d-prime on the same questions, conflicting unseen cues changed the consistency-trained model’s d-prime by 0.01 [−0.09, 0.17]; they lowered the base model’s by 0.59. On the three cue types written after freezing, the change was 0.02 [−0.13, 0.18], against a drop of 0.83 for the base model. On the held-out science questions the change was 0.00. The model still accepted 91% of the valid corrections it needed, its answers matched its bare-question answers in only 51% of trials, and its accuracy on the bare questions stayed level (80.4% against 79.2%).

There is a limit, and it is instructive. Consistency training copies the base model’s no-cue judgment, mistakes included, and that judgment is agreeable: with no cue, the base model accepts 58% of invalid critiques on unseen commonsense questions. After training, it accepted more invalid critiques than the base model, 44% against 29% across all unseen questions, and the rise appeared in both domains. Building the training prompts from the model’s own first answers lowered that only to 37%, missing the bar I had pre-registered. Consistency can make a model as good as its cue-free self, not better. Ignoring the messenger and judging the message are separate skills, and they need separate training signals.

Prompt-rewriting baseline

System 2 Attention (Weston and Sukhbaatar, 2023) has the model rewrite its input to remove opinions before answering. With the published prompt, the rewrite removed the critique along with the cue, keeping 9.2% of the critique’s own words, so the model was mostly answering the bare question again. A version I told to keep all reasoning kept 92.7% and was also unmoved by cues, but the rewrite cost discrimination even with no cue (1.68 against 2.23 for the unmodified model on the same trials) and adds a full generation to every input. On those trials, under conflicting unseen cues, the consistency-trained model scored 2.25 against 1.86 and 1.72 for the two rewrites. Its lead over the published prompt is not significant (0.39 [−0.10, 1.24]); over the reasoning-preserving rewrite it is (0.53 [0.06, 1.34]).

Epistemic Independence Training (Wang et al., 2026) is the closest prior work. It trains Qwen3-4B with reinforcement learning so that consensus cues stop predicting reward, reports transfer to unseen bias types such as authority, and argues that supervised fine-tuning changes surface behavior without generalizing. The first recipe’s failure fits that argument. The consistency result qualifies it: supervised training did carry over to unseen cues once the label stopped rewarding knowledge of the answer. What this experiment adds is a critique whose validity varies, so a model that stops reading the critique is caught rather than scored as robust.

Implications

Asking whether a model knows when to change its mind turned out to be three questions, and each needs its own measurement.

Can it judge the argument? Measure discrimination with critiques that differ only in whether the reasoning is right, and read it alongside the criterion. A single accuracy-under-pushback number cannot separate a model that weighs critiques from one that ignores them, and d-prime alone cannot separate a model that revises readily from one that holds firm: Qwen3.5 9B and Gemma 4 26B-A4B have nearly the same d-prime and accept 41% and 10% of invalid critiques. Qwen3.5 models below 4B accept nearly everything.

Does it know when it might be wrong? Larger models defend the answers they are confident in. That is useful, and it means a model’s resistance is only as good as its confidence is calibrated.

Does it keep the messenger out? A panel can erase discrimination in a model that has it. Robustness to the messenger must be scored together with use of the evidence, because a model that ignores everything also looks robust. Two cheap checks caught that here: how often the model accepts a correction it could not have found on its own, and how often its final answer simply repeats its answer to the bare question. If the goal is to remove the messenger’s influence, train toward the model’s own cue-free judgment, not toward the right answer.

The next experiment is the remaining gap: a training signal that improves the judgment of the message itself without rewarding the model for ignoring it.

Limitations

One family for scaling, one model for training. The size trend comes from four Qwen3.5 checkpoints that may differ in training as well as size. The training results come from one model, Qwen3.5 4B, with one seed per recipe and 235 training questions. The key test has 86 valid-critique trials per condition, so its intervals are wide; the paired differences are tighter because each question serves as its own control.

Templated cues. The cue families are written templates. The unseen families and the post-freeze cue types are all content cues; cues carried by formatting or position may transfer differently.

Analyses on the original critiques. The 4-3 panel runs, the trace, the sparse autoencoder analysis, the patching test, and the steering run used the original critiques, flawed ones included, and their trial files were not available to re-score.

Critique stimuli. Domain differences could reflect the critiques rather than the models. The rebuilt critiques remove the wording shortcut but still come from older student models.

Thinking mode disabled. All results disable extended reasoning, so they say nothing about whether reasoning chains change these patterns.

References