Hetzner is working on LLM Inference

Who Should Read

A backend developer curious whether an EU-hosted LLM inference API can be swapped in with the OpenAI SDK

Core Mechanics

• Hetzner is experimenting with inference
  • It is an OpenAI-compatible API with tokens created in the Experiments dashboard.
  • There is no billing, SLA, or production guarantee, and only one model.
• The only model is Qwen3.6-35B-A3B-FP8
  • It is a 35B-parameter MoE (Mixture-of-Experts, only some experts run per token) with 3B active parameters.
  • It accepts text and images, has a 262K context window, and uses FP8 weights.
• Integration just means changing base_url
  • Run pip install openai and set base_url to https://inference.hetzner.com/api/v1.
  • Setting enable_thinking to False reduces reasoning tokens but is undocumented by Hetzner.
• Author measured 153ms TTFT and 224 tok/s
  • On July 23, 2026, median time to first token was 153ms across seven short requests from one client.
  • Five longer generations capped at 512 tokens hit 224 output tokens per second.
  • The author says this says little about behavior under heavy concurrent load.

Community Reactions

• Hope for an EU-native provider
  • Some commenters said a respected EU-native inference provider would help with regulation.
  • Others noted the lack of a GDPR-compliant coding plan besides Mistral.
• Single-GPU models seen as the sweet spot
  • One commenter said Hetzner could be competitive for models that fit on a single card.
  • They said entering big-model serving now would be madness.
  • Another said they would use it if it hosted GLM and Kimi.
• Commenter says enable_thinking is described backwards
  • A commenter pointed out that enabling it makes the model reason and disabling it does not, the opposite of the article's wording.
  • Another commenter complained that parts of the post read like AI-hallucinated text.
• Alternatives and pricing hopes
  • One commenter recommended Infomaniak's AI services, which offer several models.
  • Another hoped for web-hosting-style inference resellers that make inference affordable.

How to Apply

• Get a token and test compatibility today
  • Create a token in the Experiments dashboard and run the code below with only base_url changed.
  • Compare the response format against your current OpenAI SDK code.
• Connect it only to non-critical workloads
  • Limit calls to internal experiments such as summarization or classification.
  • Add a fallback to your existing provider since there is no billing or SLA.
• Write your own latency measurement script
  • Measure TTFT and tokens per second with your real prompt lengths.
  • Compare against the author's numbers, which come from one client and one test.
• Avoid depending on undocumented options
  • Keep enable_thinking in one config location since Hetzner has not documented it.
  • Parse responses so they do not break if reasoning text appears.

Try It

# Run in terminal and Python: Hetzner Inference (experimental) compatibility test
# Install: pip install openai

from openai import OpenAI
import time

client = OpenAI(
    base_url="https://inference.hetzner.com/api/v1",
    api_key="{{YOUR_TOKEN}}",
)

start = time.time()
response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B-FP8",
    messages=[{"role": "user", "content": "{{question}}"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)
print(f"Total time: {time.time() - start:.2f}s")

# Next: set enable_thinking to True and compare answer quality and latency

Terminology

Related Papers

• Mercury 3: Diffusion at scale

  Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality

• Timestral: Mistral's time-series foundation models

  A 209M time-series model ranks first on three benchmarks and runs on CPU.

• DeepSeek V4.1 Flash 1017 (Max)

  DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.

• If you don't run your own LLM proxy, how do you know whats going on?

  Running your own LLM proxy shows which keys, models, and cache misses drive your usage.

• DeepSeek: DeepSeek v4.1 Flash 1017

  DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents

• DeepSeek-v4.1-Flash-1017

  DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length

• Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware

Related Resources

• https://sliplane.io/blog/hetzner-inference
• https://www.infomaniak.com/en/hosting/ai-services