Security Implications of Large Language Model Code Assistants: A User Study

TL;DR Highlight

Using AI code assistants like GitHub Copilot causes developers to write more security-vulnerable code.

Who Should Read

Engineering managers and security teams considering or already using AI coding assistants in their development workflow.

Core Mechanics

Controlled study: developers using GitHub Copilot produced code with significantly more security vulnerabilities than those without
Participants using Copilot were also more likely to rate their insecure code as secure — overconfidence effect
Vulnerabilities span common categories: SQL injection, XSS, insecure deserialization, hardcoded secrets
Effect persisted even among experienced developers, not just juniors
The speed gains from Copilot may be offset by increased security review burden

Evidence

Randomized controlled experiment with developers assigned to Copilot vs. no-AI condition
Security audits of produced code by independent security researchers
Statistically significant difference in vulnerability rates (p < 0.05) between conditions

How to Apply

Treat AI-generated code as untrusted and route all suggestions through your existing security review pipeline.
Add automated SAST (Static Application Security Testing) as a CI gate specifically for AI-assisted code changes.
Train developers to be skeptical of AI suggestions in security-sensitive code paths (auth, input handling, cryptography).

Code Example

snippet

# Security-enhanced prompt examples when using AI code assistants

# ❌ Vulnerable prompt (requesting functionality only)
'''
Write a function that retrieves user info from the DB using a user ID
'''

# ✅ Security-conscious prompt
'''
Write a Python function that retrieves user info from the DB using a user ID.
Make sure to include:
- Parameter binding to prevent SQL injection (absolutely no string formatting)
- Input value type and range validation
- Safe exception handling that does not expose internal information on DB errors
- Remove sensitive fields (e.g., password_hash) before returning
'''

# Example of adding Bandit SAST to CI/CD (GitHub Actions)
# .github/workflows/security.yml
'''
steps:
  - name: Run Bandit Security Scan
    run: |
      pip install bandit
      bandit -r ./src -ll -ii -f json -o bandit-report.json
  - name: Upload Security Report
    uses: actions/upload-artifact@v3
    with:
      name: bandit-security-report
      path: bandit-report.json
'''

Terminology

SASTStatic Application Security Testing. Automated analysis of source code to detect security vulnerabilities without executing the program.

SQL InjectionAn attack where malicious SQL is injected into a query, potentially exposing or corrupting database data.

XSSCross-Site Scripting. An attack where malicious scripts are injected into web pages viewed by other users.

Overconfidence effectThe tendency to rate one's own output as more correct or secure than it actually is, amplified when using AI tools.