Names Can Hurt: Spotting Slopsquatting Risks Caused by Package Name Hallucinations in Local Coding LLMs

Who Should Read

Backend or platform developers worried about blindly pip-installing packages recommended by local coding LLMs

Core Mechanics

• Half of hallucinated names already exist on PyPI
  • Of 179 flagged names, half were true 404s and half were already registered.
  • The registered ones, like pil, faiss, tabula and haystack, were low-quality lookalikes of known projects.
  • A 404 check alone cannot catch that half.
• Two-layer detector with retry and fallback
  • Layer 1 checks PyPI's JSON API, and layer 2 is a Random Forest over name and metadata features.
  • On a hit, the agent retries at temperatures 0.7, 1.0 and 1.2.
  • When retries run out, a LangGraph state machine routes to a stronger fallback model.
• Alias mapping runs only after a 404
  • A 24-entry YAML maps names like cv2 to opencv-python.
  • If an attacker registers cv2, PyPI returns 200 and the mapping never runs.
  • So the mapping cannot become a bypass.
• Hallucination grows with adversarial prompts
  • Routine coding prompts gave 0 to 10%, while slopsquat baits gave 40 to 73%.

Evidence

• 76% clean on 300 prompts
  • With llama-3.1-8b, 228 of 300 prompts ended with hallucination-free code.
  • The primary model's give-up rate was 28.7% (86/300).
• Retry and fallback rescue rates
  • Retries fixed 28 of 114 cases, a 24.6% intra-model rescue rate.
  • The fallback rescued 14 of 85 cases, 16.5%.
• Extractor and mapping fixes cut give-ups
  • Give-up rate fell from 79% (55/70) at v0 to 28.7% (86/300) at v2.
  • Better extraction (AST plus code-only regex) and the alias table drove the drop.
• Model refused 9 of 10 bait names itself
  • Only ultrarag was installed under its requested name.
  • The pipeline finished clean on 6 of 10 direct bait prompts (60%).

How to Apply

• Check names in code blocks against PyPI
  • Extract only imports and pip install names from code blocks in LLM output.
  • Query https://pypi.org/pypi/{name}/json for each and block installs on 404.
  • Skip prose text, or words like python and github will trigger false alarms.
• Add a maintainer and GitHub rule for 200s
  • For names that pass the 404 check, add two conditions.
  • Warn when there are zero maintainers or no GitHub URL.
  • Start with warnings rather than blocks, since new legitimate packages can be flagged.
• Apply alias mapping only after a 404
  • Keep a small YAML of common aliases like cv2 to opencv-python and PIL to Pillow.
  • Never apply it to names that returned 200; run those through the classifier rules.
• Pick a fallback from another model family
  • Configure your agent to hand off to a different family when retries run out.
  • Same-family fallbacks tend to repeat the same fake names.

Try It

# Python 3 / run in terminal. Simplified version of the paper's method (404 check + metadata warning)
# Install: pip install requests

import re, sys, requests

def names(text):
    out = set()
    for block in re.findall(r"```(?:\w+)?\n(.*?)```", text, re.S):
        out |= set(re.findall(r"^\s*(?:import|from)\s+([A-Za-z0-9_]+)", block, re.M))
        for line in re.findall(r"pip install ([^\n]+)", block):
            out |= {t.strip(",.") for t in line.split() if not t.startswith("-")}
    return out

def check(name):
    r = requests.get(f"https://pypi.org/pypi/{name}/json", timeout=10)
    if r.status_code == 404:
        return "404 hallucinated"
    info = r.json()["info"]
    urls = str(info.get("project_urls")) + str(info.get("home_page"))
    return "200 suspicious (no GitHub link)" if "github.com" not in urls else "200 ok"

text = sys.stdin.read()
for n in sorted(names(text)):
    print(n, check(n))

# Usage: save the LLM answer to a file, then run python check.py < answer.md
# Next: add a lookup that re-checks aliases like cv2 -> opencv-python only after a 404.

Terminology

Related Papers

• Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

  Saving long tool output to files cut GPT-5.1 accuracy by 11.3 points; naming the authoritative file restored it.

• Fake News, Real Crashes: Information Injection Attacks on LLM Financial Trading Agents

  Fake headlines hijacked LLM trading agents 62% of the time versus 14% for prompt injection.

• Harness Engineering 101: The LLM is the brain, the harness is the body

  An agent is a loop managing a JSON array; 16 chapters build one from scratch

• Harness2 by Google Research – Recursive Agent Harnessing

  Recursively refining the harness with fixed model weights raised benchmark scores by up to 11.8 points.

• Ousted Savannah police officer gave ICE agent husband access to Flock system

  A police officer shared her Flock login with her ICE-agent husband; an audit found 983 searches and she was fired

• Tmux and Claude Code: Sessions, Send-Keys, and the Ceiling

  Running Claude Code agents in tmux works well, but limits appear around four agents.

Related Resources

• https://github.com/sargamsahu1011/package-hallucination-detector