Don't Let the Model Write the YAML: Deterministic, Minimal-Diff GitOps Remediation from LLM-Proposed Field Changes

Who Should Read

Platform/DevOps engineer building a GitOps pipeline that applies LLM-agent-proposed fixes to Kubernetes manifests automatically

Core Mechanics

• Diff generation fails silently
  • Under strict patching only 2.7% of Sonnet 5 diffs apply.
  • GNU patch --fuzz applies 96%, but 14-20% of those land in the wrong place with no error.
• Whole-file rewrite depends on model size
  • The small model (gemini-3.7-flash) reflows and corrupts the file.
  • The frontier model (claude-sonnet-5) is mostly right, but the same request drops a field on some seeds.
• Model decides what, code does how
  • The agent emits only kind, name, field_path, new_value and reason as structured output.
  • Code uses the YAML parser's position marks to find the target's character span and replaces only that.
• No re-serialization means a minimal diff
  • Bytes outside the span stay untouched, so comments, quoting and key order survive.
  • The result is the same for any model, and generation cost is independent of file size.
• Ambiguity means refusal (fail-closed)

Evidence

• Span-edit was correct in all 415 runs
  • 83 tasks × 5 seeds, correct 1.000, flaky 0/83.
  • Identical for both models since it never calls the model.
• Sonnet 5 full-file rewrite was flaky on 6/83 tasks
  • Correct rate 0.976, but six tasks flip between right and wrong across seeds.
  • 405 vs 415 correct, a gap of 10 out of 415 runs.
• Misapplication by applier (415 Sonnet 5 B2 diffs)
  • Strict: correct 0.027; offset-tolerant: correct 0.675 with 0 misapplied.
  • GNU patch --fuzz=3 applied 0.964 with 0.140 misapplied, and 0.202 when ignoring whitespace (-l).
• Flash full-file rewrite was only 2.4% correct
  • 97.6% altered unrelated lines, with a mean diff of 921.3 lines.
  • Output tokens per edit: span-edit about 50-150 vs B1 about 9,300 (Sonnet 5, approximate).

How to Apply

• Constrain agent output to a FieldChange JSON
  • Replace prompts that ask for edited files or diffs with JSON output of kind, name, field_path, new_value and reason.
  • Make containers and env vars addressed by name, not index.
• Swap the apply step for parser-position replacement
  • Use PyYAML's yaml.compose_all to get nodes and replace only the start_mark.index to end_mark.index slice.
  • Never dump and rewrite the whole file.
• Make ambiguous targets refuse
  • Return None when the resource or field is missing, (kind, name) is duplicated, or the target is not a scalar.
  • Explicitly refuse alias fields, which are still a hole.
• Keep merging human-only
  • Show the diff via preview and only open a PR.
  • Enforce with a CI test that no code path calls a merge endpoint.

Try It

# Run in a Python 3 terminal (simplified version of the paper's method: position-based replacement)
# Install: pip install pyyaml

import yaml

raw = """spec:
  replicas: 3 # keep >= HPA min
  template:
    spec: {}
"""

def find_span(raw, path):
    node = yaml.compose(raw)
    for key in path:
        if isinstance(node, yaml.MappingNode):
            node = next((v for k, v in node.value if k.value == key), None)
        else:
            return None
        if node is None:
            return None
    if not isinstance(node, yaml.ScalarNode):
        return None
    return node.start_mark.index, node.end_mark.index

span = find_span(raw, ["spec", "replicas"])
if span is None:
    print("refuse")
else:
    s, e = span
    print(raw[:s] + "5" + raw[e:])

# Next: try a missing field, a map target, and a named list (containers) to check refusal and comment preservation.

Terminology

Related Papers

• Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

  Saving long tool output to files cut GPT-5.1 accuracy by 11.3 points; naming the authoritative file restored it.

• Fake News, Real Crashes: Information Injection Attacks on LLM Financial Trading Agents

  Fake headlines hijacked LLM trading agents 62% of the time versus 14% for prompt injection.

• Harness Engineering 101: The LLM is the brain, the harness is the body

  An agent is a loop managing a JSON array; 16 chapters build one from scratch

• Harness2 by Google Research – Recursive Agent Harnessing

  Recursively refining the harness with fixed model weights raised benchmark scores by up to 11.8 points.

• Ousted Savannah police officer gave ICE agent husband access to Flock system

  A police officer shared her Flock login with her ICE-agent husband; an audit found 983 searches and she was fired

• Tmux and Claude Code: Sessions, Send-Keys, and the Ceiling

  Running Claude Code agents in tmux works well, but limits appear around four agents.