Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs

Who Should Read

Backend developers building LLM-powered prediction systems or Research Agents. Engineers frustrated by simple context accumulation in RAG pipelines and seeking more structured state management.

Core Mechanics

• The core idea is a 'Bayesian Linguistic Belief State': with each search step, the LLM updates a JSON object containing probability estimates + supporting evidence summaries + open questions, differing from traditional methods of endlessly stacking text into context.
• Removing the Belief State degrades performance (Brier Index drops by 5.1), a greater impact than removing web search (3.4 drop) – structured state management is more important than search itself.
• Employing Multi-Trial Aggregation—running 5 independent trials in parallel and averaging—reduces the high variance of LLM predictions. However, this averaging is theoretically beneficial for Brier Score (quadratic loss) but not for Brier Index (linear loss).
• Calibration is performed using Hierarchical Platt Scaling (correction method with source-specific offsets) – outperforms global Platt Scaling across all settings, especially preventing over-shrinking for sources with extreme base rates (e.g., Wikipedia vaccine questions).
• Sequential processing (searching once and iteratively updating belief) yields a 7.7 higher Brier Index than Batch processing (parallel search followed by inference)—the single largest performance differentiator.
• Model ensembles are ineffective between models of the same architecture. Diversity (Jensen-Shannon Divergence 0.006~0.014) is too low when using the same prompts/tools/search results, leading to performance degradation.

Evidence

• "Achieving a total Brier Index of 83.5 with BLF across 400 ForecastBench backtest questions – significantly outperforming GPT-5 (79.9), Cassi (79.5), Grok 4.20 (78.8), and Foresight-32B (79.2) with p<0.001 significance."

How to Apply

• Instead of continuously appending search results to context in a RAG agent, structure your prompts to have the LLM update a JSON belief state of the form {probability, confidence, evidence_for, evidence_against, open_questions} after each search. This structured belief also informs subsequent search queries.
• Applying a Multi-Trial pattern—running the same question independently multiple times (K=5 recommended) and averaging—guarantees improvement for evaluation metrics based on quadratic loss (Brier Score) due to Jensen's inequality. However, it offers no theoretical benefit for linear loss-based metrics, where median aggregation is slightly preferable.
• For systems requiring different calibration for source types (e.g., a mix of market data, economic indicators, and Wikipedia), applying Hierarchical Platt Scaling with source-specific intercept offsets instead of global Platt Scaling can improve Brier Index by +3.5 from a zero-shot baseline.

Code Example

# BLF Bayesian Linguistic Belief State structure example
# Configure the LLM to generate this JSON along with each Tool call

belief_state_schema = {
    "probability": 0.5,          # P(outcome=1 | evidence so far)
    "confidence": "low",         # low / medium / high
    "evidence_for": [],           # List of summaries supporting the outcome
    "evidence_against": [],       # List of summaries refuting the outcome
    "open_questions": [],         # Questions to search for next
    "update_reasoning": ""        # Why the probability changed in this step
}

# Agent loop pseudocode
def blf_agent(question: str, cutoff_date: str, max_steps: int = 10):
    belief = {"probability": 0.5, "confidence": "low", 
              "evidence_for": [], "evidence_against": [],
              "open_questions": [question], "update_reasoning": "Prior"}
    history = [{"role": "user", "content": question}]
    
    for step in range(max_steps):
        # LLM generates action + updated belief in one go
        response = llm_call(
            messages=history,
            tools=[web_search, read_files, url_lookup, submit],
            # Key: include updated_belief field in tool call arguments
            system_prompt=f"""After each action, update your belief state as JSON:
{belief_state_schema}
Current belief: {belief}
Cutoff date: {cutoff_date} - do not use information after this date."""
        )
        
        action = response.tool_call
        new_belief = response.updated_belief  # LLM generates this
        
        if action.name == "submit":
            return action.args["probability"]
        
        # Execute action (including date filtering)
        observation = execute_tool(action, cutoff_date=cutoff_date)
        
        # Update history
        history.append({"role": "assistant", "tool_call": action, "belief": new_belief})
        history.append({"role": "tool", "content": observation})
        belief = new_belief
    
    return belief["probability"]  # Return last belief if max_steps reached

# Multi-Trial Aggregation
import numpy as np

def aggregate_trials(forecasts: list[float], metric: str = "brier_score") -> float:
    """Mean if metric is quadratic loss, median slightly better if linear"""
    if metric in ["brier_score", "log_score"]:
        return np.mean(forecasts)  # Guaranteed improvement by Jensen's inequality
    else:  # brier_index (linear)
        return np.median(forecasts)  # +0.2 BI improvement

# Run 5 independent trials
K = 5
forecasts = [blf_agent(question, cutoff_date) for _ in range(K)]
final_forecast = aggregate_trials(forecasts)

Terminology

Related Resources

• ForecastBench Leaderboard
• ForecastBench Datasets
• Metaculus AI Benchmark Tournament (AIBQ2)
• FutureX Live Benchmark
• Prophet Arena
• LiteLLM (unified interface for multiple LLM providers)