Petals: Run LLMs at home, BitTorrent-style

TL;DR Highlight

Load part of a model and join a peer network to run LLMs up to 405B.

Who Should Read

A backend or ML engineer who wants to run huge open LLMs or inspect their internals without owning big GPUs

Core Mechanics

The model is split across people
- You load only a part of the model on your GPU.
- Other volunteers in the network serve the remaining parts.
Supported models are large
- It lists Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+) and BLOOM (176B).
- It says a consumer GPU or Google Colab is enough.
Speed is chatbot-level, per the site
- Llama 2 (70B) runs up to 6 tokens/sec in single-batch inference.
- Falcon (180B) runs up to 4 tokens/sec.
More freedom than a classic API
- You can use any fine-tuning and sampling method.
- You can run custom paths through the model or see its hidden states.
API comfort with PyTorch flexibility
- It works with PyTorch and 🤗 Transformers.

Community Reactions

Opinion that it is outdated (2022)
- Commenters said smaller models, quantization and consumer-GPU optimizations have improved a lot.
- For models that fit on one GPU, alternatives like AI Horde (stablehorde.net) already exist.
Comments pointing out it looks abandoned
- Several said GitHub activity is low and the last commit is old.
- Someone asked whether there are updates for modern LLMs.
Opinion that bandwidth is the bottleneck
- One comment said inter-node bandwidth would starve it.
- Another said gigabit or 10-gigabit low-latency links would be needed to make it feasible.
Alternatives and variants mentioned
- One commenter has been playing with meshllm.cloud; node-split models are not in the public demo.
- Another proposed hosting a whole small model per host with P2P request routing.

How to Apply

Run the Petals Colab example first
- Open the Try now in Colab link on the homepage.
- Check speed and response stability yourself.
Shortlist it for internals experiments
- Pick research tasks that need hidden states or custom paths.
- Use Petals only for what a regular API cannot do.
Check maintenance before adopting
- Look at recent commits and issues on GitHub.
- If you need newer models, compare with alternatives like meshllm.cloud.

Try It

copy & run

# Prompt to paste into Claude or ChatGPT (to decide on Petals)

I want to use {{model name, e.g. Llama 3.1 405B}} for {{use case, e.g. internal chatbot}}.
My setup is {{GPU specs}} with network {{bandwidth/latency}}.

Compare using Petals (running a model split across a public GPU network) with using a regular LLM API.
Criteria: speed, reliability, data privacy, maintenance status.

Next step: check the date of the latest commit on the Petals GitHub and reflect it in the comparison.

Terminology

hidden statesIntermediate values a model computes at each layer. It is like seeing ingredients mid-cooking instead of only the finished dish.

single-batch inferenceProcessing one request at a time. Like a cashier serving one customer at a time.

BitTorrent-styleA P2P approach where many users share pieces of a file. Here, many GPUs each hold a piece of the model.

quantizationStoring model numbers at lower precision to save memory. Like compressing a photo to a smaller file.