Labels Override Definitions in Jev-Style Typed Decision Models | AI Paper Digest

TL;DR Highlight

Renaming options to A/B raised routing accuracy by +0.1511 over meaningful labels.

Who Should Read

Backend developers who use LLMs as routers or classifiers by passing option labels together with written definitions

Core Mechanics

Deleting definitions changed nothing
- LAYA-TD scored 0.8559 with all definitions removed versus 0.8487 with them.
- Definitions alone gave 0.7971, so the model can read them but ignores them when labels exist.
Labels win over definitions in a conflict
- When a label pointed to the wrong class, LAYA-TD accuracy fell to 0.1357.
- That is below the no-information control of 0.5690.
The cause is prompt rendering, not architecture
- LAYA writes each option as "{label}: {definition}" while VON writes only the definition.
- Changing that one expression in both directions removed the effect in LAYA and created it in VON.
Plain LLM scoring shows it too
- Qwen2.5-7B scoring option strings showed the same effect.
- This contradicts the earlier claim that the decision head is to blame.
Confidence thresholds cannot catch it

Evidence

A/B names beat meaningful names on PolicyBench
- LAYA-TD accuracy rose from 0.742 (meaningful names) to 0.893 (A/B), +0.1511 [+0.1377, +0.1646], n=3500.
- Qwen2.5-7B gained +0.1203 and Mistral-7B +0.1531 on the same comparison.
Rendering patches reproduced the effect both ways
- With labels removed, the three LAYA checkpoints showed a +0.0000 gap across aligned, arbitrary and misleading labels.
- Adding labels to VON dropped classification accuracy from 0.8511 to 0.2281 under conflict.
Mitigations were measured
- M1 (arbitrary labels) gave +0.1511 on LAYA-TD, while M3 (an instruction to ignore labels) gave only +0.0138.
- M4 (flagging two-rendering disagreement) raised accuracy on remaining items from 0.7420 to 0.8921 at 0.8100 coverage.
Null-state correction depended on the model
- With aligned labels it gave +0.0391 for LAYA-TD and +0.0748 for Qwen2.5-7B.
- It dropped VON to 0.5000 accuracy.

How to Apply

Call the same question twice with different labels
- Keep the definitions identical and change only the labels, then compare the probability distributions.
- If they differ, labels are affecting decisions, so apply the steps below.
Rename options to A, B when the rule is in the definitions
- Put the whole rule in the definition field and use meaningless labels.
- If the class name itself is a strong cue, accuracy can drop, so compare on your eval set first.
Review disagreements when labels must keep meaning
- Run once with real labels and once with A/B labels, and send disagreeing items to a human or stronger model.
- Use this disagreement gate instead of a confidence threshold.
Rewrite negated questions in positive form
- Avoid phrasing like 'is it false that ...' and write 'is the request unauthorized' instead.
- An instruction to attend to polarity did not help.

Try It

copy & run

# Paste into Claude/ChatGPT - simplified version of the paper's label-vs-definition check

Perform the classification below twice. Both times, decide using only the definitions.

[Input]
{{문서}}

[Pass 1: original labels]
{{라벨1}}: {{정의1}}
{{라벨2}}: {{정의2}}

[Pass 2: labels replaced with A, B]
A: {{정의1}}
B: {{정의2}}

Give the choice and reasoning for each pass, and mark 'DISAGREE' if the results differ.

Next step: run this on 50 real routing logs and count the DISAGREE rate.

Terminology

typed decision modelA small model that generates no text and only returns a probability for each option the caller supplies. Think of it as an answer-sheet grader.

option-label biasThe tendency to answer from an option's name rather than its definition. Like guessing from answer-choice names without reading the question.

readoutThe way one forward pass is turned into probabilities over the caller's options.

state-blindnessThe fraction of items where the answer equals the answer for an empty input. High values mean the input is being ignored.

AUROCA measure of how well a confidence score separates correct from incorrect answers. Below 0.5 means it points the wrong way.

contextual calibrationA correction that divides out the model's output on an empty input to remove its fixed bias.

Related Papers

Original Abstract (Expand)

A typed decision model answers a fixed question about an input by returning a probability for each of several caller-defined options. Each option carries a short label and a written definition, which is where a developer states the rule the model should apply. Jev introduced this interface for routing, moderation and triage, open implementations followed, and the same operation occurs whenever a language model is used as a classifier by scoring label strings. We study the open implementations, whose weights we can inspect and patch, and ask whether the probability follows the definitions or the labels. A preference for the label we call option-label bias. Across four open-weight typed decision models, three ways of reading an answer from a Qwen2.5 backbone, eleven classification tasks and PolicyBench, a synthetic routing suite we introduce in which the rule appears only in the definitions, the answer is mostly the labels. Deleting every definition leaves accuracy unchanged (laya-td: 0.8559 against 0.8487), although those definitions support 0.7971 on their own, and renaming the options to A and B raises accuracy by +0.1511 [+0.1377, +0.1646]. One system, von, is unaffected, and the two code bases differ in one expression: laya writes each option as "{label}: {definition}", while von writes only the definition. Changing that expression in both directions, with no weight changed, makes all three laya checkpoints exactly invariant (+0.0000 [+0.0000, +0.0000]) and creates the effect in von, whose accuracy falls from 0.8511 to 0.2281 when a label contradicts its definition. Earlier work attributed this failure to the constrained decision head these models use in place of a text decoder; our results locate it in the prompt rendering. We give a two-call test that tells a practitioner which case applies to their model, and measure what four mitigations are worth.