VLMs make hidden choices about textual vs. visual information - and those choices change based on task complexity
Imagine you’re analyzing a medical case. You see a chest X-ray that’s completely clear — no signs of disease. But the accompanying radiology report claims there’s clear evidence of pneumonia.
Which do you believe? The image or the text?
This isn’t a hypothetical scenario. As vision-language models (VLMs) become integral to healthcare, multi-agent systems, and retrieval-augmented generation pipelines, they constantly face situations where visual and textual information can conflict. The question is: how do these models actually decide which signal to trust?
At Megagon Labs, we set out to answer this question — and what we discovered has significant implications for multimodal AI systems.
The Black Box Problem
VLMs like GPT, Claude, and Gemini have revolutionized how we interact with AI by seamlessly combining images and text. They can analyze charts, interpret diagrams, and answer questions about visual content with impressive accuracy.
But here’s the catch: we don’t fully understand how these models integrate information from different modalities. When visual and textual cues align, everything works beautifully. But in the messy real world, mismatches are common:
- Medical reports with incorrect descriptions
- Product listings with wrong images
- Educational materials with mismatched diagrams
- News articles with misleading visuals
If we’re going to trust these systems in high-stakes applications, we need to understand their behavior when signals conflict.
Designing the Experiment
To systematically investigate VLM biases, we created five novel benchmarks with intentionally mismatched image-text pairs. Each benchmark covers a different domain:
Mathematical Reasoning:
- Graph Connectivity: Determining whether two given nodes are connected when the visual graph contradicts the text.
- Function Convexity: Predicting the convexity of a mathematical function after flipping its coefficients so that the algebraic expression contradicts the visual shape of the curve.
- Polynomial Roots: Identifying the roots of a polynomial when the generated polynomial has one root that differs between its equation and its plotted graph.
Science Questions: Physics and chemistry problems where textual descriptions point to different answers than the images
Visual Descriptions: Spatial relationship tasks where text describes the opposite of what’s shown (e.g., “A is above B” when the image shows A below B)
For each task, we carefully controlled for complexity — from simple to highly challenging — to understand how difficulty affects model behavior.
We tested six state-of-the-art VLMs spanning different scales and architectures:
- O4-mini, GPT-4o, and GPT-4o mini (OpenAI)
- Llama-3.2–90B-vision (Meta)
- Qwen2-VL-72B and Qwen2-VL-7B (Alibaba)
The Surprising Pattern
Here’s what we found: VLM bias isn’t consistent — it shifts dramatically based on task complexity.
For simple tasks, models overwhelmingly favor text to generate their response. But as complexity increases, they pivot toward images. This isn’t a subtle effect — we’re talking about bias swings ranging from -74.4% (strongly text-favored) to +56.8% (strongly image-favored) depending on the task and model.
The Polynomial Example
Consider polynomial root calculations. When given a degree-1 polynomial (a simple linear equation), models relied almost exclusively on the textual equation
But increase the complexity to degree-3 or degree-4 polynomials? The bias flips. Models now tend to favor the visual graph representation, likely because identifying roots from a complex curve is easier than parsing and mathematically solving a lengthy algebraic expression.
This pattern repeated across domains:
Graph Connectivity: For simple graphs (fewer than 10 edges), models trusted text descriptions. But for complex graphs with 32+ edges? Nearly 100% image reliance across all models.
Function Convexity: Stronger models, such as GPT -4o, showed clear shifts from text-favored (simple functions) to image-favored (complex expressions). Smaller models exhibited more erratic patterns.
Science Questions: Easy questions relied heavily on text. Meanwhile, with hard questions, models struggled overall, with increased image usage but also significantly more incorrect predictions.
Why This Happens
Our analysis revealed two key factors driving these biases:
1. Perceived Difficulty
Models appear to make implicit judgments about which modality is “easier” for a given task. To test this, we asked models to explicitly predict which modality they thought would be simpler to use — then compared those predictions to their actual behavior.
For stronger models, there was clear alignment: they favored the modality they perceived as easier for solving the task. GPT-4o showed a particularly strong correlation between perceived ease and actual bias.
2. Modality-Specific Performance
We analyzed error patterns by testing models with text-only, image-only, and combined inputs. The results were telling:
When a model performed poorly with text-only inputs on a task, it tended to favor images when both modalities were present (and vice versa). But here’s the interesting part: some errors unique to one modality persisted even when both modalities were available — revealing a “blind spot” bias.
For example, in graph connectivity tasks, image-specific errors that occurred with image-only inputs still appeared when text was added. The model was essentially ignoring potentially corrective information from the text modality.
Can Models Detect When They're Getting Mixed Signals?
Understanding the bias is the first step. But can we help models recognize when image and text are actually conflicting? We tested three strategies for detecting mismatches:
Strategy 1: Verbalized Mitigation
Simply asking the model: “Do you detect any mismatch between the image and text?”
Results: Surprisingly effective for stronger models. GPT-4o achieved 87.3% accuracy in detecting conflicts on function convexity tasks. Less effective for smaller models.
Strategy 2: Chain-of-Thought Mitigation
Guiding the model through explicit steps:
- Analyze the image alone
- Analyze the text alone
- Compare and identify mismatches
Results: Particularly effective for GPT-4o mini, which showed significant improvements over verbalized prompting. Less consistent for open-source models.
Strategy 3: Decomposed Mitigation
Running the model three separate times: once using only the image to solve the task, once using only the text, and finally once to compare and combine the outputs.
Results: Most effective when the model already performed well on both modalities independently. GPT-4o achieved 81.3% accuracy on graph connectivity using this approach.
The key insight? No single mitigation strategy works universally. Effectiveness depends on the specific model, the task, and crucially, the model’s baseline performance with each modality.
What This Means for Real-World Applications
These findings have immediate practical implications:
Healthcare
Medical AI systems must be carefully evaluated for modality bias. A system that over-relies on textual reports might miss visual cues in imaging. Conversely, complex cases with subtle visual findings might be misinterpreted if the system ignores detailed text descriptions.
Retrieval-Augmented Generation (RAG)
When RAG systems retrieve mixed-modality documents, they may exhibit unpredictable biases based on query complexity. System designers need to account for these biases in their architectures.
Multi-Agent Systems
As VLMs coordinate multiple specialized agents, understanding their modality preferences is crucial for reliable orchestration — especially when different agents provide conflicting information.
Educational Technology
Learning platforms using VLMs must ensure that textbook diagrams and accompanying explanations are properly weighted, regardless of problem difficulty.
What's Really Going On?
Our research reveals a fundamental truth about current VLMs: they don’t simply “fuse” multimodal information — they make active, context-dependent decisions about which signals to prioritize.
This behavior isn’t necessarily bad. In fact, it mirrors human reasoning to some extent. When faced with complex visual information, we too might rely more heavily on what we see rather than what we read. The problem is that VLMs make these choices without transparency, and sometimes with biases that don’t align with human expertise.
As these models become more capable and more widely deployed, understanding and accounting for these biases becomes critical. Our benchmarks and mitigation strategies provide a starting point, but there’s much more work to be done.
Next Steps
Several questions remain open:
Generalization: Do these patterns hold across other domains beyond mathematics, science, and spatial reasoning? We suspect yes, but further research is needed.
Training Influence: How much of this bias stems from the distribution of modalities in training data versus emergent reasoning behaviors? Understanding this could inform better training strategies.
Human Alignment: In which cases do VLM biases align with human expert judgment, and when do they diverge? This is particularly important for high-stakes domains.
Dynamic Mitigation: Can we develop systems that automatically detect when to apply which mitigation strategy based on the task and model performance?
Try It Yourself
We’ve open-sourced our benchmarks and evaluation code. If you’re working with VLMs, we encourage you to test your models for modality bias:
Paper: Mixed Signals: Decoding VLMs’ Reasoning and Underlying Bias in Vision-Language Conflict
Code & Benchmarks: github.com/megagonlabs/Modality-Bias
What’s your experience with VLM reliability in production? Have you encountered unexpected modality biases? Understanding how your specific model handles conflicting information could be the difference between a reliable system and one that fails when you need it most.
This research was conducted by Pouya Pezeshkpour, Moin Aminnaseri, and Estevam Hruschka at Megagon Labs. The work was presented at EMNLP 2025.
