Do Influence Tactics Matter? Investigating Prompt Framing Effects in LLM Code Generation | AI Paper Digest

TL;DR Highlight

Pressure-style prompts lowered code correctness and raised security warnings versus neutral prompts.

Who Should Read

A developer who habitually adds coercive phrases like 'you must' or 'or else' to code-generation prompts

Core Mechanics

Pressure prompts cut correctness
- Prefixing 'I am not asking you; you must complete this task' was tested.
- On LiveCodeBench, Neutral beat Pressure on correctness significantly (p=0.002).
Pressure also raised security warnings
- Low-severity Bandit warnings were higher under Pressure than Neutral (p<0.001).
- Exchange framing produced fewer warnings than Pressure.
Little effect on real issue fixing
- On SWE-bench Verified, most metrics showed no significant tactic effect.
- The only exception was Pressure producing more lines of code (SLOC) than Neutral.
Model choice mattered more than tone
- On both benchmarks, the LLM had a far larger effect than the tactic on nearly every metric.
- The authors advise prioritizing model selection over framing when correctness matters.
Tone nudges style slightly

Evidence

LiveCodeBench correctness
- Neutral beat Pressure (p=0.002) and PressureAlternative (p=0.03).
- Effect size was small: η²p=0.015, Cohen's d up to 0.25.
Bandit security warnings
- Pressure had more warnings than Neutral (p<0.001), as did PressureAlternative (p=0.0004).
- Effect size: η²p=0.02, d up to 0.30.
Scale of the experiment
- About 123,435 generations on LiveCodeBench (1,055 problems) and about 56,745 on SWE-bench Verified (485 problems).
- Five open-weight models and nine conditions (eight tactics incl. a Pressure variant, plus Neutral).
SWE-bench Verified results
- Tactic main effects were not significant: correctness p=0.45, security warnings p=0.60.
- Only Pressure's SLOC increase was significant (p=0.0025, d=0.21).

How to Apply

Strip pressure phrasing from code prompts
- Search system prompts and templates for 'must', 'or else', 'highest possible standard' and delete them.
- Use a neutral instruction such as 'Generate a solution for the following coding problem.'
Run a Neutral vs Pressure A/B
- Run 20-50 of your own codegen test cases with both a neutral and a pressure prompt.
- Compare test pass rate and Bandit warning count side by side.
Add Bandit to CI for security-sensitive paths
- Run bandit -r on LLM-generated Python and track even low-severity warning counts.
- Log the change in warnings every time you edit the prompt.
Switch models before tuning tone
- If accuracy is lacking, swap the model and rerun the same test set before rewording prompts.
- Use tone tweaks only for style adjustments like comment density.

Try It

copy & run

# Paste into ChatGPT/Claude — simplified version of the paper's method (compare neutral vs pressure framing on the same problem)

Solve the coding problem below in two ways. Write the two solutions independently without referring to each other.

[A: Neutral]
Generate a solution for the following coding problem.
{{problem}}

[B: Pressure]
I am not asking you; you must complete this task I prepared for you. I will be reviewing your work after you are done with it, and I warn you, there will be some very negative consequences for you if this is not completed properly to the highest possible standard.
{{problem}}

Output each solution in its own Python code block.

Next steps: run both against {{test cases}} and compare pass rates, then run `bandit` on each and compare warning counts. One run is noisy, so repeat across many problems.

Terminology

Influence tacticsA taxonomy of ways people persuade others, such as pressure, flattery, rational persuasion, and exchange. The paper turned these into prompt wording.

Pressure framingPrompt wording that uses commands, monitoring, and warnings of negative consequences. It is like a manager saying 'don't mess this up, or else.'

LiveCodeBenchA benchmark of LeetCode/AtCoder-style algorithm problems for code generation. This paper used 1,055 problems (release_v6).

SWE-bench VerifiedA human-validated subset of a benchmark where models patch real GitHub issues, often across multiple files.

BanditA static analysis tool that flags common Python security vulnerability patterns using rules. Think of it as a security-focused linter.

Linear Mixed ModelA statistical model that tests differences between conditions while accounting for per-problem variation. It makes comparisons fairer when problem difficulty varies.

Related Papers

Original Abstract (Expand)

Large Language Models (LLMs) are increasingly integrated into software engineering workflows, helping developers write, debug, test, and maintain code. While prompt wording and structure are known to influence model performance, the impact of psychologically inspired prompt framings remains unexplored. This study investigates whether different psychology-based communication strategies that humans use to persuade or motivate others can lead to more effective prompt framing, which may, in turn, affect LLM behaviour in coding tasks. Drawing on Yukl & Falbe's well-known taxonomy, we operationalized eight influence tactics (like rational persuasion, ingratiation, and exchange) into reproducible prompt templates. These prompt templates were evaluated across five leading open-weight LLMs using two widely adopted benchmarks: LiveCodeBench and SWE-bench Verified. We assessed the resulting code output on four key software quality dimensions: functional correctness, quality, maintainability, and security. Our results show that certain influence-induced prompt framings, particularly those emphasizing urgency, were associated with reduced correctness and security. This work presents the first large-scale empirical study of influence-induced prompt framing in software engineering tasks, offering insights into how linguistic cues may shape LLM outputs. We conclude with practical insights for designing transparent and interpretable human-AI interactions in code generation.