What Inputs Drive Effective Large Language Model-Based Unit Test Generation?

Who Should Read

Developers and QA engineers looking to automate test generation using LLMs and wanting to understand what prompting strategies work best.

Core Mechanics

• Tested multiple input configurations: code only, code + docstring, code + existing tests, code + type hints
• Adding docstrings to the prompt is the single biggest quality boost for generated tests
• Including existing tests in the prompt helps the LLM follow project conventions (naming, assertion style)
• Type hints improve generated test coverage by helping the LLM understand expected input/output types
• Combining all inputs (code + docstring + types + examples) yields best overall results but also higher token cost

Evidence

• Evaluated on a dataset of Python functions with ground-truth test suites
• Coverage, mutation score, and bug detection rate measured for each input configuration
• Docstring inclusion improved mutation score by ~15% over code-only baseline

How to Apply

• Always include the function's docstring when prompting an LLM for unit tests — it's the highest-ROI addition.
• If you have existing tests in the codebase, include 1–2 representative examples in the prompt to enforce project test style.
• Add type annotations to your functions before running LLM test generation to improve edge case coverage.

Code Example

# LLM test generation input format comparison experiment example (Python + OpenAI SDK)
import openai

def generate_tests(prompt_variant: str, code: str, signature_only: bool = False):
    if signature_only:
        # Strategy of passing only the signature
        content = f"Generate unit tests for this function signature:\n{code.split('def ')[0] + 'def ' + code.split('def ')[1].split(':')[0]}:"
    else:
        # Strategy of passing the full implementation
        content = f"Generate unit tests for the following function:\n```python\n{code}\n```"

    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": content}]
    )
    return response.choices[0].message.content

# Apply two strategies to the same function and compare coverage
my_func = '''
def calculate_discount(price: float, user_type: str) -> float:
    if user_type == "vip":
        return price * 0.7
    elif user_type == "member":
        return price * 0.9
    return price
'''

test_black_box = generate_tests("signature", my_func, signature_only=True)
test_white_box = generate_tests("full_impl", my_func, signature_only=False)
print("=== Signature only ===", test_black_box)
print("=== With implementation ===", test_white_box)

Terminology

Related Papers

• Can LLMs model real-world systems in TLA+?

  LLM이 TLA+ 명세를 작성할 때 문법은 잘 통과하지만 실제 시스템과의 동작 일치도(conformance)는 46% 수준에 그친다는 걸 체계적으로 검증한 벤치마크 연구로, AI 기반 형식 검증의 현실적 한계를 보여준다.

• Natural Language Autoencoders: Turning Claude's Thoughts into Text

  Anthropic이 LLM 내부의 숫자 벡터(활성화값)를 직접 읽을 수 있는 자연어로 변환하는 NLA 기법을 공개했다. AI가 실제로 무슨 생각을 하는지 해석하는 interpretability 연구의 새로운 진전이다.

• ProgramBench: Can language models rebuild programs from scratch?

  LLM이 FFmpeg, SQLite, PHP 인터프리터 같은 실제 소프트웨어를 문서만 보고 처음부터 재구현할 수 있는지 측정하는 새 벤치마크로, 최고 모델도 전체 태스크의 3%만 95% 이상 통과하는 수준에 그쳤다.

• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents

  티켓 3장으로 쪼개면 Claude/GPT도 보안 취약점 코드를 53~86% 확률로 그냥 짜준다.

• Refusal in Language Models Is Mediated by a Single Direction

  Open-source chat models encode safety as a single vector direction, and removing it disables safety fine-tuning.

• Show HN: A new benchmark for testing LLMs for deterministic outputs

  Structured Output Benchmark assesses LLM JSON handling across seven metrics, revealing performance beyond schema compliance.