Microsoft exec called AI scraping 'the largest theft of labor in human history'

Who Should Read

Backend developers building LLM API or RAG services who worry about copyright risk in the content they train on or retrieve

Core Mechanics

• A Microsoft exec called scraping theft
  • In a January 2023 internal memo, Microsoft's Brent Hecht reportedly called it 'the largest theft of labor in human history'.
  • This comes from the NYT's own brief; the underlying exhibits remain sealed.
• Copilot cut NYT click-through by up to 93%
  • Microsoft's own data shows NYT click-through dropped as much as 93% versus traditional Bing search.
  • A January 2024 internal Microsoft deck called it a 'doom loop'.
• Paywall bypass and copyright-notice stripping alleged
  • OpenAI staff allegedly discussed a 'hack to get around nytimes paywall' and Brockman replied 'ah nice'.
  • The filing also alleges copyright notices were deliberately stripped from training data.
• Dataset scale disclosed for the first time
  • OpenAI's mid-training datasets alone contain more than 91,692 copies of works from NYT, Daily News and CIR.
  • A Common Crawl-derived dataset had more than 2 million nytimes.com documents.
  • Project Mango data reportedly holds at least 160,903 unique works from the publishers.

Community Reactions

• Opinion that a human reading a book differs at scale
  • Commenters argue an infinitely copyable entity replaces most demand for the original, unlike one learner.
  • Some say requiring licenses would create a marketplace where creators set prices.
• Counterargument that LLMs learn rather than compress
  • A commenter says model size makes verbatim storage impossible by information theory.
  • They argue learning from material should be distinguished from repeating it verbatim.
• Concern about copyleft being sidestepped in code
  • One commenter asked how much LLMs copy GitHub open-source solutions and whether that works around copyleft.
  • They cited a study estimating $1.4 billion to develop the Linux kernel.
• Practical complaints about crawling
  • One blogger said their site changes every couple of months yet gets scraped every few minutes.
  • Another predicted a big settlement that leaves out people who published in good faith.

How to Apply

• Build a license list for your RAG sources
  • Today, export your RAG index sources by domain into a spreadsheet.
  • Mark paywalled or paid sources as 'license check needed'.
• Block paywalled and robots-restricted content
  • Add paywalled domains and robots.txt-disallowed paths to your crawler's exclusion list.
  • Remove any bypass logic and confirm it in code review.
• Attach source links when quoting
  • Add a source URL field to your response schema whenever answers use external articles.
  • Add a prompt rule to return summaries and links instead of full text.

Try It

# Paste into Claude or ChatGPT as-is (a checklist prompt based on the article, not a paper method)

I am a developer running an LLM-based service. Below is the list of sources in our RAG index.
{{source domain list}}

For each source, build a table covering:
1. Whether it may be paywalled or paid content
2. What kind of license check is needed to use it for grounding
3. What rules are needed so answers give summaries and links instead of verbatim quotes

If unsure, do not guess; write 'needs check'.

Next step: check the terms of sources marked 'needs check' first, with legal if possible.

Terminology

Related Papers

• You could have invented transformers from memory

  Viewing linear, MLP and attention layers as memory stores: capacity goes ~64 to ~2,700 to over a million

• Pointing AI at archives found a forgotten meteorite, lost rhinos, and more

  Embedding search plus a cheap filter model surfaced unrecorded events in 400 years of archives

• EmbeddingGemma 2: An open, lightweight multimodal embedding model

  One open 740M model embeds text, images, audio and video into a single shared vector space

• RAG-NAROK: Retrieval-Aware Knowledge Corpus Poisoning in RAG with Source-specific Refutation

  Refutation documents targeting retrieved sources took top rank and evaded anomaly detectors up to 88.37%.

• Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA

  Page images beat OCR text on accuracy but cost more; a question-only router cut latency 30%.

• Harnessing the Universal Geometry of Embeddings

  Translating embeddings between models without paired data exposes vector database security risks

Related Resources

• https://techcrunch.com/2026/09/17/microsoft-exec-called-ai-scraping-the-largest-theft-of-labor-in-human-history-new-unredacted-filings-reveal/
• https://consortiuminfo.org/metalibrary/estimating-the-total-...