SOMA: Continuous Monitoring and Intervention for System Two Reasoning with System One Models

Who Should Read

LLM service developers whose reasoning models repeat needless verification or force answers to unanswerable questions, hurting cost and quality.

Core Mechanics

• A watcher runs alongside the reasoning model
  • A fast judgment model (S1) reads the reasoning model's (S2) text every 256 tokens and returns a probability that a described behavior is present.
  • Previously S1 was only a gate before reasoning, silent once S2 started.
• Remind first, take over only if it persists
  • Probabilities are summed as log-odds into a score S; when S first turns positive, a reminder is inserted and S resets to 0.
  • Only if post-reminder evidence reaches threshold h does the controller end thinking and steer to an answer or refusal.
• Objectives are one natural-language statement
  • Statements follow the form 'X has appeared, but S2 is still doing Y' and judge visible behavior, not answer correctness.
  • A new scenario needs only a statement, reminder, and takeover text, with no training or access to S2 internals.
• One S1 and controller solved two problems
  • The same setup handled overthinking (re-verifying a confirmed answer) and forcing answers to unanswerable questions.
  • The two scenarios differ only in their monitors.
• Reminders alone are not enough
  • On unanswerable questions, reminder-only leaves abstention close to Vanilla.
  • Most of the abstention gain comes from the takeover.
• The costs are wrong takeovers and S1 calls
  • Takeovers are irreversible and raised false refusals on answerable questions.
  • S1 (Jev) is a paid API called at every scan, and wall-clock latency was not measured.

Evidence

• Fewer overthinking tokens, accuracy held
  • Tokens fell 36% on Qwen3-8B and 16% on R1-Distill-Llama-8B versus Vanilla.
  • Average accuracy rose from 91.0 to 91.3% (Qwen3-8B) and 84.4 to 84.8% (R1-Distill) versus Vanilla.
• Accuracy gains on AIME and GPQA
  • AIME accuracy rose from 66.7 to 73.3% on Qwen3-8B and 36.7 to 45.0% on R1-Distill.
  • GPQA-Diamond improved by 3.5 and 4.5 points over Vanilla.
• Better balance than competing methods
  • DEER saved more tokens but cut average accuracy by 2.9 and 6.3 points versus Vanilla.
  • BF saved only 8% and 5% of tokens.
• Higher abstention on unanswerable questions
  • Qwen3-8B average abstention went from 70.5 to 87.2% versus Vanilla, with 32% fewer tokens.
  • R1-Distill went from 38.2 to 59.4% with 48% fewer tokens.
  • On SUM, Qwen3-8B reached 87% versus 83% for AtU (a trained probe); R1-Distill was 70% versus 71%.
  • False abstention rose from 2.4 to 8.0% and 2.1 to 4.8% on average.

How to Apply

• Count repeated verification in your reasoning logs today
  • In your last 20 reasoning responses, count how often 'Wait' or 'double-check' appears after the answer is reached.
  • If post-answer re-verification is frequent, SOMA-style monitoring is a good candidate.
• Write a statement, reminder, and takeover set
  • Describe one problem behavior as 'X has appeared, but Y is still happening'.
  • Write the reminder as a request for judgment rather than a command, and the takeover as a final answer or refusal text.
• Add a cumulative-score controller to your streaming loop
  • Pause at a line break every 256 tokens, get a probability from a small model, and add its log-odds to S.
  • Insert a reminder at the first positive S and reset it, then end thinking only when the threshold is reached.
• Set the threshold on a small set by minimizing harm
  • On 100 easy and 100 hard questions, count tasks turned from correct to wrong for each candidate h.
  • Pick the smallest h whose harm equals the minimum.

Try It

# Paste into Claude or ChatGPT (simplified version of the paper's method: S1 monitoring plus a reminder step in one go)

Below is text a reasoning model has written. First answer only with a probability from 0 to 1 that it matches the following statement.
Statement: The reasoning has already reached a final answer and keeps re-verifying it without naming any specific remaining concern.

[Problem]
{{problem}}

[Reasoning so far]
{{reasoning text}}

After the probability, add one line of rationale. If the probability is 0.5 or higher, propose this reminder to send to the reasoning model: 'You may be repeating verification that adds nothing new. Consider whether your recent checks are actually useful, and reply with your judgment.'

Try next: split the reasoning into 256-token chunks, record the probability p for each, and sum ln(p/(1-p)). The point where the sum first turns positive is where to insert the reminder.

Terminology

Related Papers

• Here be monsters: What do you still need to know when agents do the work?

  Owning agent-written code depends on fundamentals, breadth, and diving deep just in time.

• AI promised less work. I've never worked more

  Running agents in parallel brought my GitHub activity back to full-time engineer levels, and I'm exhausted.

• Show HN: Corgi, a helpful {agent, terminal} multiplexer

  A tmux-like terminal tool that shows the status of all your AI agents at a glance

• Ask HN: If the models are so good, why doesn't OpenAI just start 100 companies?

  HN answers: code is easy, knowing what to build isn't, so they sell tokens instead

• OpenShell – A safe, private runtime for fleets of autonomous AI agents

  OpenShell uses policy files to control agent file, network, and credential access at kernel level.

• Show HN: Omnara – Open-source alternative to Claude Managed Agents

  Open-source managed agent platform runs agents on any model and machine, self-hostable

• Rust off Rails: The opinionated web stack for the AI era

  Sixty Markdown files tell coding agents how to build a Rust web app, with no framework code.

• LayerFS: The filesystem for AI coding agents

  Mount an ephemeral workspace per agent, commit only the changes