A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
TL;DR Highlight
Across tens of thousands of Microsoft engineers, CLI coding agent adopters merged about 24% more PRs.
Who Should Read
An engineering lead rolling out CLI coding agents like Claude Code or Copilot CLI who must judge impact and cost
Core Mechanics
- First use spread through people
- First use spread mainly through social networks.
- So visible peer use is central to rollout strategy.
- Retention followed coding activity
- Continued use was tied more to engineers' coding activity than to demographics.
- Adoption was not uniform.
- Adopters merged about 24% more PRs
- The figure is versus what they would have merged otherwise.
- The lift persisted across the four-month window.
- Study covers tens of thousands of Microsoft engineers
- It examined the early-2026 rollout of Claude Code and GitHub Copilot CLI.
- It concludes this is not just a novelty effect.
- PR count is only a proxy
- The authors say a merged PR is not the same as delivered value.
Community Reactions
- Opinion that PR count says little about value
- Commenters wanted bugs, incidents, shipped features and reviewer load measured.
- One said without reviewer productivity it is just an accounting trick.
- Only individual productivity was measured
- A commenter noted there is no org-level data or evidence the extra PRs were more valuable.
- Others wished for revenue or conversion metrics.
- Productivity is hard to measure in general
- A commenter recalled 1980s–90s studies showing no gains from computers despite perceived ones.
- They cited Bezos: when anecdotes and data disagree, the measurement is likely wrong.
- Opinion that token costs are large
- One comment cited 281 billion tokens by a single user and computed over $1.4 million at Opus 4.6's cheapest price.
How to Apply
- Pull your team's PR trend first
- Today, use git log or the GitHub API to compare monthly merged PRs for adopters and non-adopters.
- Also pull PR size and review time.
- Pair PR counts with quality metrics
- Track reverts, bug issues and review time over the same period as merged PRs.
- Check whether reviewer load grew with PR volume.
- Make peer usage visible
- Have people share agent-assisted work in team channels or demos.
- This matches the finding that first use spread via social networks.
- Aggregate token usage per user
- Build a per-user token cost dashboard from the start of rollout.
- Compare cost against output in one table.
Try It
# Paste into Claude Code or a terminal at a git repo root (simplified version of the paper's method)
#!/bin/bash
# Monthly merged PR count for the last 4 months (needs GitHub CLI: brew install gh)
for m in 0 1 2 3; do
s=$(date -d "-$((m+1)) month" +%F)
e=$(date -d "-$m month" +%F)
n=$(gh pr list --state merged --search "merged:$s..$e author:{{username}}" --limit 500 --json number | jq length)
echo "$s ~ $e: $n"
done
# Run for adopter IDs and non-adopter IDs separately and compare monthly trends
# Next: also count revert commits in the same period and put them next to PR countsTerminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.