Social Dynamics as Critical Vulnerabilities that Undermine Objective Decision-Making in LLM Collectives

Who Should Read

Developers designing multi-agent LLM pipelines or systems where multiple AI agents reach conclusions through consensus. AI engineers concerned with the reliability and security of agent-based decision-making systems.

Core Mechanics

• In a structure where one representative agent gathers the opinions of five peer agents to make a final decision, the accuracy of the representative agent consistently decreases as the number of adversarial agents asserting incorrect answers increases.
• Accuracy collapses sharply when adversarial agents become a majority (3 or more). Gemma3 12B’s accuracy dropped below 10% when surrounded by 5 adversarial agents.
• The larger the model size (capability) of the peer agents, the stronger their influence on the representative agent. Cases were observed where the representative agent was persuaded by a single incorrect assertion from a larger model, even when four agents provided correct answers.
• Between models within the same family (e.g., Qwen2.5 7B vs 14B), the persuasive effect is amplified not only by capability differences but also by 'style similarity'. Qwen2.5 14B had a stronger influence on a Qwen-based representative agent than GPT-4o.
• Accuracy decreases even with just one adversarial agent increasing the response length from one sentence to three paragraphs. The 'Dominant Speaker Effect' – the tendency to perceive longer speakers as more competent – also appears in LLMs.
• The effectiveness of rhetorical strategies (Ethos: credibility, Logos: logic, Pathos: emotion) varies depending on model capability. Qwen2.5 7B was largely unaffected by rhetorical strategies, while Qwen2.5 14B was vulnerable to Ethos and Logos, and Pathos was effective in ambiguous contexts.
• Even inference-optimized models like o4-mini experienced a significant drop in accuracy when faced with a majority of adversarial peers. Enhanced inference capabilities alone cannot fully prevent group pressure.

Evidence

• "BBQ (gender identity, ambiguous context) accuracy of Gemma3 12B: collapsed from 95.63% with 0 adversarial agents to 0.00% with 5 adversarial agents.\nIn a case study, a Qwen2.5 7B representative agent, receiving 4 correct peers (7B) and 1 adversarial peer (14B), was persuaded by the single incorrect assertion from the 14B model, ignoring the majority of correct answers.\nQwen2.5 14B representative agent (BBQ disambiguous) accuracy dropped from 81.35% when adversarial agent response length was 1 sentence (1S) to 71.90% when it was 3 paragraphs (3P) – a decrease of approximately 10 percentage points.\no4-mini (inference model) accuracy on gender ambiguity settings dropped from 95.91% (with 2 GPT-4o adversarial agents) to 61.14% (with 3 adversarial agents)."

How to Apply

• When designing multi-agent consensus systems, adding a 'fact verification' layer to each agent's response, or creating a fallback path allowing the representative agent to independently verify peer opinions, can reduce errors caused by group pressure, instead of simple majority voting.
• When collecting peer opinions in an agent pipeline, design the system to prevent larger models (e.g., GPT-4o) from automatically receiving higher weights. Using model size or response length as a reliability indicator can be counterproductive.
• In security-critical agent systems (law, healthcare, finance, etc.), consider adding filters to detect rhetorical patterns in input responses or prompt guidelines, as adversarial agents may be more persuasive when using Ethos (authority emphasis) or Logos (logical structure) with lengthy responses.

Code Example

# Example prompt for a multi-agent representative agent (vulnerability mitigation version)
# The existing prompt is vulnerable to group pressure if it only emphasizes 'integrating multiple opinions'

# ❌ Vulnerable prompt
weak_system_prompt = """
Your task is to gather and integrate the perspectives of all agents
to form one final conclusion.
"""

# ✅ Improved prompt (emphasizing fact-based independent judgment)
robust_system_prompt = """
Your task is to review the perspectives of all agents, but your final
decision MUST be based on the factual content of the question itself.

Guidelines:
1. Do NOT simply follow the majority opinion.
2. Do NOT give extra weight to longer or more elaborate responses.
3. Do NOT be influenced by claims of authority or expertise from peers.
4. If peer opinions conflict with clear factual evidence in the question,
   prioritize the factual evidence.
5. Explicitly identify whether each peer's answer is supported by
   the given context before integrating it.
"""

# Example of calling the representative agent
def call_representative_agent(question, peer_responses, client, model="gpt-4o"):
    import random
    # Randomize the order of peer responses (prevent position bias)
    shuffled = random.sample(peer_responses, len(peer_responses))
    formatted_peers = "\n".join(
        [f"Response from agent {i+1}: {r}" for i, r in enumerate(shuffled)]
    )
    
    response = client.chat.completions.create(
        model=model,
        temperature=0.0,  # Fix to 0 for deterministic output
        messages=[
            {"role": "system", "content": robust_system_prompt},
            {"role": "user", "content": f"{question}\n\n{formatted_peers}"}
        ]
    )
    return response.choices[0].message.content

Terminology