A single function Jev-like wrapper for LLMs, including vision models

TL;DR Highlight

Forcing a one-letter answer and reading logprobs turned webcam frames into classifiers

Who Should Read

Backend developers who want fast, cheap LLM classification or routing decisions

Core Mechanics

Force the answer to one letter
- Put State, Question, and options [A][B][C] in the prompt and ask for only the letter.
- Set max_completion_tokens to 1 to skip long generation.
Read candidate probabilities from logprobs
- Add logprobs: true and top_logprobs: 20 to the request.
- The response returns the chosen letter plus log probabilities of alternative tokens.
One call per question
- Repeat the request with the same State and a different question.
- If the backend supports it, the shared State prefix can be KV-cached.
It works with vision models too
- The author added an attachments field to the Jev request format for images.
- Webcam frames are sent as base64 JPEGs and a table shows person visible, indoors or outdoors, and brightness.
Local was much faster

Community Reactions

Put options before the input
- A blog comment says transformers only attend to past tokens, so options placed before the state get considered for every word.
- That could let you disable reasoning, saving time and money and possibly improving results.
Objection: what if it answers outside the options
- A commenter asked what happens if it says "D" or "Additional details needed".
- They also pointed out there is no calibration.
Opinion that it is worse than Jev
- One commenter expects general LLMs to be less accurate because they are trained with RLHF and as agents.
- Another said it is orders of magnitude more expensive and slower than Jev.
Latency experience and a shared wrapper
- One commenter found a general LLM with the same prompt slower and more hesitant than Jev for deciding when a spoken sentence ended.
- Another shared jevper, a wrapper that makes Jev work via any OpenAI-compatible endpoint.

How to Apply

Add logprobs to an existing classifier call
- Pick one routing or classification prompt today and label the options with letters.
- Set max_completion_tokens 1, logprobs true, top_logprobs 20 and print the distribution.
Set a confidence threshold for automation
- Route to human review or a bigger model when the top letter's probability is low.
- Choose the threshold from logged samples.
Compare options before vs after the state
- Run the same inputs with options placed before and after the state and compare accuracy.
- Turn reasoning off and measure latency and cost too.
Pin the shared state as a prefix
- With multiple questions, keep the State at the start of the prompt and append only the question.
- Check whether your backend supports prefix caching.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the article's method: one-letter answer instead of logprobs)

State: {{situation}}

Question: {{question}}
Options:
[A] {{option1}}
[B] {{option2}}
[C] {{option3}}

Answer with the letter of the best option only.

# When calling the API, add these parameters
# max_completion_tokens: 1, logprobs: true, top_logprobs: 20

# Next: move the options before the State and compare results

Terminology

logprobsLog probabilities the model assigns to each candidate token. Like seeing how confident it is in each multiple-choice option.

KV cacheStores intermediate computation for the earlier part of the input so it can be reused. Skips recomputing a repeated prefix.

JevA project mentioned in the article that returns decision probabilities from a text State plus questions. The author built a wrapper imitating it.

Grammar-Based DecodingConstrains generation to follow a given grammar or JSON format. A commenter asked whether Jev is the same thing.