What Inputs Drive Effective Large Language Model-Based Unit Test Generation?

Who Should Read

Developers and QA engineers looking to automate test generation using LLMs and wanting to understand what prompting strategies work best.

Core Mechanics

• Tested multiple input configurations: code only, code + docstring, code + existing tests, code + type hints
• Adding docstrings to the prompt is the single biggest quality boost for generated tests
• Including existing tests in the prompt helps the LLM follow project conventions (naming, assertion style)
• Type hints improve generated test coverage by helping the LLM understand expected input/output types
• Combining all inputs (code + docstring + types + examples) yields best overall results but also higher token cost

Evidence

• Evaluated on a dataset of Python functions with ground-truth test suites
• Coverage, mutation score, and bug detection rate measured for each input configuration
• Docstring inclusion improved mutation score by ~15% over code-only baseline

How to Apply

• Always include the function's docstring when prompting an LLM for unit tests — it's the highest-ROI addition.
• If you have existing tests in the codebase, include 1–2 representative examples in the prompt to enforce project test style.
• Add type annotations to your functions before running LLM test generation to improve edge case coverage.

Code Example

# LLM test generation input format comparison experiment example (Python + OpenAI SDK)
import openai

def generate_tests(prompt_variant: str, code: str, signature_only: bool = False):
    if signature_only:
        # Strategy of passing only the signature
        content = f"Generate unit tests for this function signature:\n{code.split('def ')[0] + 'def ' + code.split('def ')[1].split(':')[0]}:"
    else:
        # Strategy of passing the full implementation
        content = f"Generate unit tests for the following function:\n```python\n{code}\n```"

    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": content}]
    )
    return response.choices[0].message.content

# Apply two strategies to the same function and compare coverage
my_func = '''
def calculate_discount(price: float, user_type: str) -> float:
    if user_type == "vip":
        return price * 0.7
    elif user_type == "member":
        return price * 0.9
    return price
'''

test_black_box = generate_tests("signature", my_func, signature_only=True)
test_white_box = generate_tests("full_impl", my_func, signature_only=False)
print("=== Signature only ===", test_black_box)
print("=== With implementation ===", test_white_box)

Terminology