Our paper FuncBenchGen (ICLR 2026) shows that brittleness often emerges under controlled complexity and interference.
Many existing tool-use benchmarks lack fine-grained difficulty control and remain vulnerable to contamination. They also make it difficult to separate multi-hop reasoning errors from execution or state-tracking mistakes.
How FuncBenchGen Works
FuncBenchGen addresses this by generating synthetic, evaluation-time function-calling tasks designed to test structured reasoning.
Tool use is framed as traversal over a hidden function-dependency DAG, allowing precise control over # core nodes (nodes on the solution path), graph depth, branching, and distractors, including 1) connected irrelevant functions that share variables with the solution path but are not required and 2) disconnected irrelevant functions that are isolated from the solution path.
Results Across Seven LLMs
Across seven open and closed LLMs, reasoning-optimized models outperform general-purpose ones. However, performance drops sharply as task complexity increases.
For example, GPT-5 falls from 72.5% accuracy with 5 core nodes to 15.0% with 20 core nodes. CINs (connected irrelevant nodes) significantly degrade performance, revealing brittle state tracking even when function calls are syntactically valid.
A Simple Intervention
A simple intervention, explicitly restating prior variable values at each step, raises GPT-5 accuracy from 62.5% to 81.3%.
FuncBenchGen provides a contamination-free framework for systematically stress-testing multi-step tool reasoning and exposing hidden failure modes.
By Seiji Maekawa, Jackson Hassell, Pouya Pezeshkpour, Tom Mitchell, and Estevam Hruschka
Read the AI research paper: https://arxiv.org/abs/2509.26553
