Have you ever noticed that when you try the same prompt and the same input data on different systems, like ChatGPT, Gemini, or more complicated compound AI systems, the results vary?
It’s tempting to attribute these differences to reasoning ability, model size, or training data. While those factors matter, recent research points to something more subtle and often overlooked: data representation.
For years, progress in LLMs has largely been framed around scaling laws and improved training strategies. But a growing body of work suggests that how information is represented, both in the input and the model intermediate/final outputs, can significantly influence performance, robustness, and generalization. Some recent studies, for example, Li et al., argue that LLM behavior is fundamentally tied to representation-based processing, rather than simple memorization.
A Controlled Study on Data Representation
One of the most compelling examples comes from our paper, “Same Content, Different Representations: A Controlled Study for Table QA.” We ask a deceptively simple question: what happens if we keep the data the same, but change only how it is represented?
To answer this, we construct paired datasets by transforming structured tables into semi-structured versions, where columns contain complex, natural-language content, using a verbalization pipeline. The key idea is to preserve the underlying content while altering its format. This allows for a direct comparison of how different modeling approaches respond to representation changes, without confounding variables.
The results are striking.
How SQL, LLMs, and Hybrid Systems Respond to Representation Changes
SQL-Based Systems
SQL-based systems, which rely on structured schemas and symbolic execution, perform extremely well when the data is clean and well-organized. But when information from multiple columns is merged and then embedded in text, their performance drops sharply, by as much as 30 to 45 percent. The same system, working on the same data, behaves very differently depending only on how that data is presented.
LLMs
LLMs show the opposite pattern. They are far more robust to changes in representation, likely due to their training on large amounts of unstructured text. However, this flexibility comes at a cost. Their performance degrades on longer tables and more complex, multi-step queries, where precise reasoning and structured execution are required.
Hybrid Systems
Hybrid systems, which combine symbolic reasoning with neural approaches, tend to perform more consistently across both representations. They don’t always achieve peak performance, but they degrade less dramatically under changing conditions.
What emerges from this study is not just a comparison of methods, but a broader insight: representation itself is a primary driver of system behavior.
Reinforcing Evidence Across LLM Research
This idea is reinforced by other recent work. Research on in-context learning shows that LLMs dynamically reorganize internal representations based on the structure of the input. Similarly, studies in knowledge representation learning demonstrate that improvements in representation can lead to better generalization across tasks. Even work on hallucination mitigation has shown that intervening on internal representations can directly improve factual accuracy.
Taken together, these findings suggest that representation is not just a surface-level concern. It plays an active role in how models process information and produce outputs.
Why Representation Matters in Compound AI Systems
This has important implications for how we think about modern AI systems. Today’s LLM applications are no longer simple, single-step text generators. They are increasingly embedded in larger, more complex compound AI systems that involve tool use, data retrieval, and multi-step reasoning. These systems often operate over heterogeneous data sources, including structured databases, semi-structured logs, and unstructured documents.
In these environments, representation becomes a system-level concern.
It affects how information is retrieved, how intermediate results are passed between components, and how different parts of the system interpret shared data. Small mismatches in representation can lead to cascading failures, especially in multi-agent or tool-based workflows.
This shifts the focus from purely improving models to designing better systems.
It is no longer enough to ask whether a model is capable of reasoning correctly. We also need to ask whether the information it receives is structured in a way that supports that reasoning. We need to consider how representations evolve across multiple steps, and whether different components are aligned in how they encode and interpret data.
In this sense, representation is not just preprocessing. It is part of the computation.
The Takeaway
The broader takeaway is simple but powerful. Improving LLM performance is not just a modeling problem. It is also a representation problem.
Our work at Megagon Labs makes this especially clear by isolating representation as a variable and showing how much it can affect outcomes. As LLM systems continue to evolve toward more complex, agentic workflows, this insight will only become more important.
Representation shapes what the model sees, how it reasons, and ultimately whether it succeeds. In many cases, it may be the most important design decision we are not paying enough attention to.
Written by Megagon Labs and Seiji Maekawa
