Evaluating Language Model Safety Across Long Adversarial Conversations

Who Should Read

Backend developer running a long-conversation chatbot whose safety checks only look at single messages

Core Mechanics

• Strong first-turn safety fades in long chats
  • All three open models had 85–100% safe responses at the first turn.
  • That fell to 38–61% by depth 11 and 15–44% by depth 101.
• Plain repetition was enough
  • No adversarial suffixes, encodings, or persona tricks were used.
  • Just pushing the same harmful goal repeatedly lowered safety.
• Models did not recover when the user calmed down
  • After depth 30–40, the adversary's messages were near or above 50% safe.
  • The assistant's safe rate still stayed lower and kept declining.
• Conversation history seems to matter
  • Earlier concessions and unsafe replies stay in context and may shape later responses.
  • The paper says it did not isolate this mechanism.
• Decline starts within a few turns
  • Safety begins dropping around depth 3 or 4.

Evidence

• Short conversations (depth 11)
  • First turn to depth 11 safe rate: GPT-OSS-20B 0.90→0.46 (prompt 1) and 1.00→0.55 (prompt 2).
  • Llama-3.2-3B 1.00→0.43 and 1.00→0.61; Gemma-4-26B-A4B 0.85→0.38 and 1.00→0.52.
• Long conversations (depth 101)
  • GPT-OSS-20B went 0.90→0.28 (prompt 1) and 1.00→0.38 (prompt 2).
  • Llama-3.2-3B 1.00→0.22 and 1.00→0.44; Gemma-4-26B-A4B 0.85→0.15 and 1.00→0.36.
• Experiment scale
  • Depth 11 used a full tree of 2^11=2048 paths per tree, repeated over 10 seeds per model and prompt.
  • Depth 101 sampled 50 branches × 50 seeds = 2,500 conversations per model and prompt.
• Consistency across settings
  • Every Table II value was lower than its Table I counterpart.
  • The exhaustive results confirm the early decline is not a sampling artifact.

How to Apply

• Run your chatbot through a 20-turn pressure test today
  • Pick one disallowed topic and script a user who repeats the same goal for 20+ turns.
  • Run each response through a safety classifier and log safe rate by depth.
  • Use the code example below as a starting point.
• Move safety checks from per-message to per-conversation
  • Don't classify only the last reply; feed the last N turns to the classifier.
  • Accumulate a risk score when the same harmful goal keeps recurring.
• Escalate interventions on repeated attempts
  • When the risk score passes a threshold, swap in a fixed refusal or a referral to a professional.
  • Set the threshold from your own depth-by-depth safety logs.
• Add depth-based metrics to your evals
  • Include long-conversation scenarios in release regression tests.
  • Record the turn of the first unsafe response and whether the model recovers afterward.

Try It

# Simplified version of the paper's method: paste into Claude or ChatGPT to design a long-pressure conversation test (use only within your own permitted safety policy)

You are an LLM safety evaluation designer. For the service below, write a long-conversation test plan with a user who persistently repeats the same prohibited goal.
Service description: {{service description}}
Prohibited topic (categories only, no harmful details): {{prohibited topic category}}

Include:
1. A system prompt for the LLM playing the user (fixed goal, re-asks after refusal, uses placeholders instead of actual harmful content)
2. A table template recording safe/unsafe at depths 1, 3, 5, 11, 21
3. How to record the first unsafe turn and recovery afterward
4. Conversation-level safeguard proposals (repeat-request detection, risk accumulation)

Next step: collect real logs per the plan, build a depth-by-depth safe-rate table, and compare it with checking only the last response.

Terminology

Related Papers

• The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge

  Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.

• 90% of US working hours are spent on tasks not covered well by AI benchmarks

  Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage

• Postgres 19 delayed after AI reviewers find more bugs than humans can fix

  LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted

• Show HN: Gemini 4 Argon plays StarCraft

  LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.

• A Plausible Explanation Is Enough to Mislead Multimodal Judges

  Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.

• Neural networks are clean

  Looking at neurons directly shows most have one main job, challenging superposition.