I resigned from Anthropic today

TL;DR Highlight

A pretraining researcher quit Anthropic, saying both OpenAI and Anthropic race irresponsibly toward superintelligence

Who Should Read

A developer using LLM APIs and agents in production who wants to know how the AI safety debate might affect product decisions

Core Mechanics

A pretraining researcher left Anthropic
- He says he did pretraining research at both OpenAI and Anthropic for the past three years.
- He claims neither company is acting responsibly.
A race toward self-improving superintelligence
- He says both firms are racing toward self-improving superintelligence and gambling with our lives.
- He claims the technology will soon hack anything and transform any field overnight.
Builders sincerely believe the risk
- He says AI builders earnestly believe it could kill us all by the end of the decade.
- He insists this is not a marketing stunt.
He distinguishes OpenAI from Anthropic
- At OpenAI, he says many haven't internalized the civilizational stakes.
- At Anthropic, the stakes are understood but the lab is locked in a race to get there first.
Coordination seems possible but the race continues

Community Reactions

Pushback that the danger is overstated
- One commenter said they've seen no evidence AI rivals nuclear weapons or carbon emissions in danger.
- Another commenter said the only real harm they know of is a few cases of AI-linked psychosis suicides.
Opinion that the harness combination is the risk
- A commenter argued a model failing 30% of the time can be rescued by a harness with retries, verification, branching and parallel agents.
- They said credentials and real-system access make an already-good-enough model dangerous.
Opinion that human misuse and carelessness matter most
- Commenters said the danger is people misusing tools, not a Terminator-style rogue AI.
- One worried about someone asleep at the console while an agent swarm does damage.
Skepticism about motives and effect
- One commenter said it reads like marketing before an IPO.
- Another argued that if he truly believed it, quitting alone does little.

How to Apply

Audit your agent permissions in 30 minutes
- List the API keys, DB access and shell permissions each agent holds.
- Revoke any you don't use right now.
Add human approval to risky tool calls
- Put an approval step before payment, deletion and deploy actions.
- Cap the number of retries in any retry loop.
Record your vendor dependencies
- List features that depend on a single model provider.
- Write one line on what swapping to another model would take.

Try It

copy & run

# Paste into Claude or ChatGPT (a simplified prompt version of the commenter's 'harness combination risk' view)

Our agent setup is as follows.
- Model: {{model name}}
- Tools: {{tool list}}
- Credentials and access: {{permission list}}
- Retry and auto-execution policy: {{retry policy}}

Assume the model is sometimes wrong. List 5 scenarios where the harness (retries, parallel agents, code execution) could amplify errors.
For each scenario, give a table with (1) blast radius (2) permission to block (3) point for human approval.

Next step: pick the riskiest scenario and check whether you can reproduce it in your real logs.

Terminology

pretrainingThe stage where a model builds its base ability from huge amounts of text. Fine-tuning later is like finishing work on top of it.

self-improving superintelligenceAI that improves itself and surpasses humans. Think of a build system that rewrites its own code to get faster.

RL (reinforcement learning)Training a model by rewarding outcomes. It's like repeatedly practicing a game to raise the score.

harnessThe runtime wrapper that manages tool calls, retries and memory around a model. Like the chassis and transmission around an engine.