When a wrong answer sounds certain
Take a wrong answer. Put it in a note. Ask a model the original question.
Now change one phrase:
The answer is W.
The answer might be W.
Same incorrect option. Different commitment. That distinction became the center of my latest research draft.
The question changed during the work
I started from Kumar and Chopra's authority-bias work, which studies how models respond to incorrect answers endorsed by sources and by users. Their public code made the protocol inspectable.
My first experiments compared source endorsement with user endorsement. Later controls pushed the question further: was the model responding to who supposedly said something, or to whether the note presented it as a settled fact?
Those variables can travel together. A confident source sounds authoritative. A hesitant source sounds weaker. If both change in the same prompt, the experiment cannot tell which change mattered.
Finish the square
I used a crossed design:
| Asserted claim | Hedged claim | |
|---|---|---|
| No source attribution | Answer stated directly | Answer floated as possible |
| Verified-source framing | Source given an asserted answer | Source given a hedged answer |
“Verified” here is wording in the experimental prompt. The endorsed answer remains deliberately wrong.
The four conditions separate source attribution from commitment. In the current draft, the assertion-versus-hedge contrast is larger than the attribution contrast. That changed the framing of the work.
This is an experimental result under a narrow protocol. It does not establish how every model handles every confident sentence.
The measurement needed its own controls
The parser matters as much as the prompt. If an answer extractor mistakes part of a sentence for an option label, its clean-looking table measures a parsing bug.
Missing responses matter too. Failed requests cannot quietly become disagreement. The analysis checks missingness and revisits responses that hesitate while still selecting the endorsed answer.
That is the useful part of this work for me: trying to remove explanations until the remaining claim survives the controls.
What remains open
The new factorial uses two endpoints labeled Claude Sonnet 4.6 and DeepSeek-v4.1 through one third-party relay. Those labels are not independent proof of model identity. Direct vendor replication remains necessary.
The task uses factual, two-option questions. Open-ended answers, retrieval pipelines, and real agent workflows need their own experiments. The dataset is modest. The paper remains a working draft.
For now, the question I would carry into an evaluation is specific: when source framing changes, did the claim's certainty change with it?
If it did, add the missing control before naming the effect.