Google is making private AI practical with homomorphic encryption

Who Should Read

Backend or ML service developers designing privacy for AI inference that sends sensitive user data to servers

Core Mechanics

• HEIR compiles AI to run on encrypted data
  • HEIR (Homomorphic Encryption Intermediate Representation) is an open-source compiler toolchain for homomorphic encryption.
  • It can convert models trained on plaintext to work on encrypted inputs.
• Homomorphic encryption computes on ciphertext
  • Servers process ciphertexts and return encrypted results without seeing the underlying data.
  • For example, a cloud can recommend content without seeing the user's features.
• Privacy trade-off becomes a cost question
  • The post admits homomorphic encryption has nontrivial cost overhead.
  • It claims this turns the capability/privacy trade-off into a cost question, and that cost is falling rapidly.
• Existing alternatives have limits
  • End-to-end encryption blocks features that need the data, such as spam or virus detection.
  • Local processing is limited by device capability, and shipping proprietary models to devices risks leaking them.

Community Reactions

• Commenters cite overhead of about 10^3 or more
  • A commenter researching privacy-preserving ML said inference overhead is about 10^3, limiting commercial viability.
  • Another commenter cited 10^3 to 10^6 compute cost and asked for compelling use cases.
• Criticism that the post gives no overhead numbers
  • Several commenters noted the post only says "nontrivial" without specifics.
  • A commenter who researched this 20 years ago recalled a single addition producing roughly 1MB ciphertext and asked whether a breakthrough happened.
• Opinion that local open-weight models are more practical
  • Many commenters said running models on your own hardware is far more practical private AI.
  • Another said at 1000x cost, an AWS Outpost-style setup would be cheaper.
• Pushback that computation correctness isn't guaranteed
  • A commenter noted FHE only ensures you need the key to see inputs or outputs.
  • The computation could be adversarial, or an attacker could insert their own computation.

How to Apply

• List your sensitive data flows
  • Spend 30 minutes today picking user inputs the server doesn't need to see in plaintext.
  • Mark small-model candidates like recommendation, anomaly detection, or hotword detection, similar to the demos.
• Read the HEIR GitHub examples and compare model size
  • Open the four demo sources the post says are public and check model structure.
  • Compare whether your model is as small as the demos.
• Build a cost table: encrypted inference vs local
  • Write latency and cost per item for local execution, trusted hardware, and homomorphic encryption.
  • Comments raised 10^3+ overhead, so measure it yourself.

Try It

# Paste into Claude or ChatGPT (simplified version of the post's idea: feasibility check for homomorphic encryption)

I run the AI feature below. I want to compare ways to keep the server from seeing user data in plaintext.
Feature description: {{feature description}}
Model type and size: {{model type/size}}
Sensitive data passed: {{sensitive data types}}
Target latency: {{target latency}}

Compare these three approaches in a table.
1) Local (on-device) execution
2) Hardware-based protection (secure enclave)
3) Homomorphic encryption (using a compiler like HEIR)

Columns: privacy guarantee type, latency/cost burden, model leak risk, fit for our feature.
If you don't know a number, don't invent it; write "needs measurement".

Next step: run the HEIR GitHub examples at a scale similar to your model and measure latency yourself.

Terminology

Related Papers

• Mercury 3: Diffusion at scale

  Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality

• Timestral: Mistral's time-series foundation models

  A 209M time-series model ranks first on three benchmarks and runs on CPU.

• DeepSeek V4.1 Flash 1017 (Max)

  DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.

• If you don't run your own LLM proxy, how do you know whats going on?

  Running your own LLM proxy shows which keys, models, and cache misses drive your usage.

• DeepSeek: DeepSeek v4.1 Flash 1017

  DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents

• DeepSeek-v4.1-Flash-1017

  DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length

• Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware

Related Resources

• https://blog.google/security/how-google-is-making-private-ai-practical-with-homomorphic-encryption/
• https://fhetextbook.github.io/