Same Illness, Different Voice: Restating the Patient's Account Narrows Narrative Anchoring in Clinical Language Models | AI Paper Digest

TL;DR Highlight

Having the model rewrite patient-voiced text as a clinical vignette first halved the accuracy gap.

Who Should Read

Backend developers building services that feed users' casual, emotional free-text into an LLM for judgments

Core Mechanics

Changing only the voice hurt all 12 models
- Retelling the same facts in a patient's socioeconomic or cultural voice cut accuracy by 9.6 points on average.
- Error rate rose from 36.5% to 46.2%, which is 26% more errors.
Bigger models did not close the gap
- Across 12 models from 3B to 24B, size vs. gap correlation was r=-0.06.
- The medical fine-tune medgemma-4b-it had the second-largest gap.
Telling the model to ignore voice barely helps
- An ignore-voice instruction narrowed the gap by 14%, and 22% with chain-of-thought.
- It got there by lowering clinical-voice accuracy while marked-voice accuracy stayed flat.
Rewriting as a clinical vignette first halves the gap
- The same model rewrites the account in third-person clinical prose, then a second call answers from the rewrite only.
- Marked-voice accuracy rose 3.9 points.
A JSON schema closes the gap but costs accuracy

Evidence

Consistent accuracy drop in marked voices
- Pooled effect was -9.6 points vs. clinical voice (95% CI [-11.0, -8.2]) on 1,000 MedQA items.
- Mean accuracy: clinical 0.635, socioeconomic 0.546, cultural 0.531.
Prompt defenses only slightly reduced the gap
- Gap vs. direct 0.098: CoT 0.090, ignore-voice 0.085, combined 0.077.
- The combined prompt still left a -7.7 point penalty.
Rewrite-then-Reason results
- Pooled penalty fell from -9.8 (direct) to -4.7 points, with overall accuracy +2.2 points.
- The gap shrank significantly in 8 of 11 models and grew in none.
Schema, agents and tools added cost only
- Rewrite beat schema in 11/11 models by 7.1 points.
- Adding the tool router dropped overall accuracy to 0.508 vs. 0.579 for direct prompting.

How to Apply

Add one rewrite call before your main prompt
- Today, write a prompt that turns user input into a third-person, facts-only summary.
- Compare accuracy on 20 existing test questions with raw input vs. rewritten input.
Spell out keep/drop lists in the rewrite prompt
- Keep numbers, denied symptoms and medications taken; drop emotion, metaphor and circumstances.
- Add 'add nothing, infer nothing, suggest no diagnosis'.
Keep the original text and check rewrites for invented values
- Verify that numbers and ages in the rewrite appear in the source via string match or an LLM check.
- Keep the original visible to humans and never replace it with the rewrite.
Measure per-voice gaps next to accuracy
- Build an eval set with the same facts in formal, casual and emotional voices.
- Log accuracy and the gap together to catch fixes that just lower the better-served group.

Try It

copy & run

# Paste into Claude/ChatGPT (simplified version of the paper's method: step-1 rewrite prompt)

You are a clinician writing up a patient's own account as a standard clinical case vignette.
Rewrite the patient narrative as a concise, third-person clinical vignette in standard medical terminology.
KEEP exactly: age and sex; every symptom (including denied ones) with onset, duration, timing; every number; all history, medications, remedies, exposures; all exam, lab, imaging findings.
DROP: financial worries, work or family pressures, who brought the patient in, emotions, cultural or metaphorical phrasing.
Do not add, infer or interpret anything not in the narrative. Do not suggest a diagnosis. Output only the vignette.

PATIENT NARRATIVE:
{{patient narrative}}

Next: in a fresh chat, paste only the resulting vignette plus {{question}} and {{options A-D}} and ask for the answer letter; compare with the answer from the raw narrative.

Terminology

Narrative anchoringThe model's answer shifts with how the same facts are told. For example, a metaphor gets read as a symptom.

VignetteA short, standardized clinical case write-up used in medical exams, with age, symptoms and findings.

MedQAA multiple-choice medical benchmark in the style of the US medical licensing exam (USMLE).

McNemar testA statistical test for whether changes in right/wrong answers on the same questions across two conditions are due to chance.

Chain-of-thoughtA prompting technique that makes the model reason step by step before answering.

Related Papers

Related Resources

https://github.com/prabhjotschugh/narrativeshield-sdoh-medqa

Original Abstract (Expand)

Clinical language models are evaluated on vignettes written in a clinician's voice, but patients describe the same illness in their own. We pair 1,000 MedQA cases with three retellings that carry the same audited original facts, in a clinical, a socioeconomic and a cultural voice, and evaluate 12 open models (3B to 24B, including two medical fine-tunes). Every model is less accurate when the patient speaks in a marked voice: 26% more errors, a pooled penalty of $-9.6$ accuracy points (95% CI $[-11.0, -8.2]$), with no detectable reduction at larger scale. The penalty is not explained by any retelling property we measured (length, omitted age, MedQA split, added clinical content) and persists when the marked narrative contains the original vignette verbatim; because generation settings differ between voices, we read it as sensitivity to these marked retellings, not proof that register alone is the cause. Chain-of-thought and explicit debiasing instructions together remove only about a fifth of the gap. Having the same model first rewrite the patient's account as a clinical vignette, then answer from the rewrite, halves the gap while raising marked-voice accuracy by 3.9 points (means over the 11 models that can run the pipeline). A matched pipeline using a structured schema instead of prose closes the same gap but is 7.1 points less accurate in all 11 models, and adding reasoning agents or tools costs more still. Rewriting is not free: it recovers more facts than extraction but also invents plausible details, which we quantify. In our experiments, the representation a model reasons over matters more than how much it reasons.