Paying for Too Many Tokens? Valid and Cost-Efficient Multimodal LLM Annotation with Simple Heuristics | AI Paper Digest

TL;DR Highlight

Packing video into one 2×8 image grid nears full-video accuracy at about 15% of the token cost.

Who Should Read

Backend or data developers who label short-form videos at scale with VLM APIs and struggle with token costs.

Core Mechanics

Pack frames into one grid image
- Pick 9 or 16 frames and tile them chronologically into a single image.
- The API's per-image token cap makes cost nearly constant regardless of video length.
Top accuracy flipped the conclusion
- On MELD, the highest-accuracy Text-only setup reversed the sign of the Negative coefficient.
- Choosing configurations by accuracy alone can give wrong statistical conclusions.
More modalities isn't always better
- On dialogue-driven MELD, text-only beat the multimodal setups.
- On TikTok hashtags, combining frames and text was clearly better.
Shot detection is simple
- Pixel and histogram differences between consecutive frames flag scene changes, and frames are sampled around them.
- It helps on short clips with few shots, while uniform sampling suffices for videos with many shots.
Fix bias with a few human labels

Evidence

Grid vs full video cost
- On 100 videos with Gemini, a ShotT 2×8 grid used 2,332 tokens per video vs 15,774 for full video, or 14.8%.
- Accuracy and κ were identical at .870 / .860.
Cost vs 1 FPS with GPT-4o-mini
- On TikTok (n=900, avg 62.3s), the 3×3 grid averaged 36,596 tokens ($0.00549) and the 2×8 grid 47,665 ($0.00715).
- 1 FPS averaged 2,275,411 tokens ($0.34131), about 50× more.
Accuracy vs validity on MELD
- Text-only had the top accuracy (.610, κ=.398) but its Negative coefficient was β=−.43 (p=.038) vs ground truth β=+.27 (p<.001).
- PPI gave valid coverage of the ground truth for all three sentiments.
Modality comparison on TikTok
- Four multimodal grid setups scored .819–.834 accuracy vs .713 Text-only and .712 Frames-only.
- Raw predictions overestimated funnyvideos at 10.7% vs 5.0% truth, and PPI cut the error from 5.7 to 2.0 points.

How to Apply

Run a 50-clip modality pilot
- Today, annotate about 50 clips with text-only, frames-only, and both, and compare κ.
- If one modality is close to the multimodal κ, drop the others to cut cost.
Switch video input to one grid
- Sample 9–16 frames, tile them chronologically into one image, and send that.
- Start at 3×3 and choose the smallest grid with acceptable pilot κ.
Pick sampling by shot density
- Use shot-transition sampling for short clips with few shots and uniform sampling for long, shot-heavy videos.
- The difference is small, so run both on the pilot.
Correct statistics with PPI
- If you compute rates or regressions from labels, make random human labels: about 100 for frequencies, 200+ for regressions.
- Report confidence intervals and read wide ones as a bias signal.

Try It

copy & run

# Paste into a multimodal model like Claude/ChatGPT (simplified version of the paper's method: grid image + JSON classification prompt)

You are a TikTok short-form video classifier. Given a {{K}}x{{L}} frame grid ({{N}} frames sampled in chronological order from a short vertical video, laid out left-to-right across two rows) and a transcript, choose exactly one hashtag from the provided list (DO NOT invent new hashtags). Favor visuals over transcript if they conflict. Respond ONLY with JSON: {"hashtag": "...", "confidence": 1-10, "rationale": "..."}

Hashtag options:
{{label list}}

Transcript:
{{transcript}}

(Attach the grid image as well.)

Next: run the same 50 videos with transcript only and with image only, and compare accuracy.

Terminology

VLMA model that understands text and images together. Video is handled by feeding multiple frame images.

Image gridA composite image tiling several frames like a comic page. It packs a video into one picture.

PPI (prediction-powered inference)A method that uses a small set of human labels to estimate model bias and correct statistical results and confidence intervals.

Cohen's κAn agreement metric that discounts chance agreement. A value of .80 or above is read as almost perfect agreement.

Shot-transition detectionTreating large pixel/histogram differences between adjacent frames as scene changes and sampling frames there.

Related Papers

Related Resources

https://github.com/JHU-CSS/multimodal-annotation

Original Abstract (Expand)

Vision-Language Models (VLMs) enable video annotation at scale, but costs accumulate quickly: processing a typical 60-second short-form video at one frame per second requires millions of tokens. To reduce costs, researchers rely on heuristics such as sampling a subset of frames, compressing videos into image grids, or using only a single modality. However, it remains unclear which heuristics save cost, and whether they preserve the downstream conclusions these annotations enable. To address this gap, we conduct a systematic evaluation of these heuristics using short-form videos, on two computational social science (CSS) tasks: sentiment and topic classification. We evaluate each configuration along three axes the literature typically treats separately: classification accuracy, validity of downstream inference, and per-video token cost. First, we find that accuracy and validity diverge: the highest-accuracy configuration can produce wrong conclusions. Second, modality value is not guaranteed: text alone can yield strong performance, indicating that adding modalities can add cost without adding signal. Finally, we find that cost can be decoupled from video length when annotating short-form videos: a single $2\times8$ image grid built via simple shot-transition detection approaches full-video understanding ($κ$ within~.05), at $\sim 15\%$ of the token cost. Based on these findings, we derive guidelines that can enable cost-aware VLM annotation in CSS.