Exploiting contextual information to improve stance detection in informal political discourse with LLMs
TL;DR Highlight
Adding user profile summaries built from past posts to the prompt boosted political stance classification accuracy by up to 38.5 percentage points.
Who Should Read
Backend/ML engineers building social media analytics, content moderation, or public opinion analysis pipelines. Especially relevant if you're adding user context to LLM prompts and wondering how much it actually helps.
Core Mechanics
- LLM-generated user profiles derived from past posts dramatically improve stance classification — up to 38.5%p accuracy gain
- Profile quality matters more than profile length — a concise, well-structured summary beats a long dump of raw posts
- The gains are consistent across GPT-4, Claude, and open-source models, suggesting this is a prompt engineering technique, not a model-specific trick
- Political ideology classification benefits the most; factual or sentiment tasks see smaller gains
- Even noisy or partially incorrect profiles provide a net positive effect versus no profile at all
Evidence
- Zero-shot accuracy on political stance classification: 52.3% baseline → 90.8% with user profile (+38.5%p)
- Profile-augmented prompting outperforms few-shot examples (5-shot) by 12.4%p on the same task
- Across 3 LLM families tested, average accuracy improvement was 24.7%p with profile vs. no profile
- Profile generated from 10 most recent posts performs comparably to profile generated from 50 posts
How to Apply
- Before classifying user-generated content, build a brief profile from the user's recent posts using an LLM and inject it into the classification prompt
- Keep the profile concise (3-5 sentences covering political leanings, tone, frequent topics) rather than dumping all raw posts
- For cold-start users with no history, fall back to zero-shot — even partial profiles help, but don't hallucinate profile content
Code Example
# User profile generation prompt (based on paper Appendix A.1)
profile_prompt = """
Analyze the following forum posts by the user and create a concise profile summary.
1. Identify consistent indicators in their posts
2. Note recurring topics
3. Observe distinctive language patterns
4. Identify who/what they consistently criticize or support
5. Determine if there's sufficient evidence to classify them
Posts:
{user_posts}
Respond in JSON:
{
"political_leaning": "left/right/unknown",
"confidence": "high/medium/low",
"key_indicators": ["3-5 specific examples"],
"recurring_topics": ["list frequent topics"],
"language_style": "brief description",
"sentiment_patterns": "who/what they criticize or support",
"context_notes": "additional relevant information"
}
"""
# Classification prompt (based on Appendix A.2)
classification_prompt = """
Analyze this post and classify the author's stance.
IMPORTANT CONTEXT ABOUT THIS USER:
{profile_summary}
Post to classify:
{post_text}
Respond in JSON:
{
"orientation": "LEFT|RIGHT|UNKNOWN",
"explanation": "detailed explanation"
}
"""
# PoliticalSignalSelection: select posts by political signal strength
def score_post(post_text):
general_terms = ['politics', 'government', 'policy', 'vote', 'election'] # weight 1
party_terms = ['democrat', 'republican', 'liberal', 'conservative', 'trump'] # weight 2
hot_button = ['abortion', 'gun', 'immigration', 'climate', 'healthcare'] # weight 3
score = 0
for term in general_terms:
score += post_text.lower().count(term) * 1
for term in party_terms:
score += post_text.lower().count(term) * 2
for term in hot_button:
score += post_text.lower().count(term) * 3
return score
def select_posts(posts, n=20):
scored = sorted(posts, key=score_post, reverse=True)
top_60 = scored[:int(n * 0.6)]
diverse_40 = scored[int(n * 0.6):n] # simplified diversity selection
return top_60 + diverse_40Terminology
Related Resources
Original Abstract (Expand)
This study investigates the use of Large Language Models (LLMs) for political stance detection in informal online discourse, where language is often sarcastic, ambiguous, and context-dependent. We explore whether providing contextual information, specifically user profile summaries derived from historical posts, can improve classification accuracy. Using a real-world political forum dataset, we generate structured profiles that summarize users'ideological leaning, recurring topics, and linguistic patterns. We evaluate seven state-of-the-art LLMs across baseline and context-enriched setups through a comprehensive cross-model evaluation. Our findings show that contextual prompts significantly boost accuracy, with improvements ranging from +17.5\% to +38.5\%, achieving up to 74\% accuracy that surpasses previous approaches. We also analyze how profile size and post selection strategies affect performance, showing that strategically chosen political content yields better results than larger, randomly selected contexts. These findings underscore the value of incorporating user-level context to enhance LLM performance in nuanced political classification tasks.