Multi-agent systems make it easier to tackle complex tasks such as code or designs, but harder to supervise reliably.
As tasks are decomposed into subtasks, delegated across specialized agents and tools, and recombined into a final result, errors and misunderstandings can propagate. By the time users see the final result, important planning decisions may already have been made without necessarily being presented in ways that support effective inspection and intervention.
This resembles the Plan-Implement-Validate (PIV) workflow in software engineering, where getting the plan right is critical to achieving the expected outcome reliably.
Meaningful steering, then, goes beyond accepting or rejecting a final answer. It requires process-level supervision, where users can understand and intervene in the planning process itself.
In our CAIS 2026 Best Artifact Award paper, “How to Steer Your Multi-Agent System: Human-LLM Collaborative Planning,” we introduce AMBIPOM, a system for studying how people steer LLM-generated plans when the planning process is made both observable and editable.
More control does not automatically mean better steering. As users gain more ways to steer the planning process, they also need support to verify the effects of their interventions.
The Plan Is the Interface
Many agentic systems expose planning details through chat, often presenting the plan as a list of steps. A user can ask for a change, and the model can revise or regenerate the plan. But when the plan is mostly mediated through conversation, it can be hard to see how the pieces actually depend on one another, or what else might change when you edit one part.
To make those dependencies visible, AMBIPOM represents plans as directed acyclic graphs (DAGs). Nodes correspond to agent-executable subtasks, and edges represent the dependencies between them. The plan graph is not just a visualization. It becomes a shared workspace for co-planning: users inspect and steer the plan, while the planner interprets those interactions and updates it accordingly.
Once the plan itself becomes interactive, there is no single way to steer it. The most common approach is still to describe changes through chat. A smaller set of systems, including our earlier AIPOM work, let users manipulate the plan structure directly, usually through fairly low-level edits (e.g., add/edit/remove a node/edge). AMBIPOM builds on these ideas by expanding the interaction space, including targeted feedback on specific parts of the plan and higher-level structural operations (e.g., merge/split).
To make sense of these different forms of interaction, we organize human–LLM co-planning along three axes: how a change is expressed, where it applies, and at what level it operates.
- Mode: semantic feedback versus structural manipulation
- Scope: global versus targeted changes
- Level: low-level edits versus high-level operations
The design space gives us a practical way to think about steerability: what kinds of feedback a planner should be able to understand, and what kinds of controls an agent system should put in users’ hands.
In AMBIPOM, these axes translate into four ways to steer the same plan:
- Global text feedback — semantic × global
- Targeted text feedback — semantic × targeted
- Low-level direct manipulation — structural × low-level
- High-level direct manipulation — structural × high-level
People Do Not Choose One Steering Style
We conducted a user study with 13 participants, who revised multi-agent plans using AMBIPOM and a baseline system across tasks involving math, retrieval, aggregation, and multi-hop computation.
When given multiple ways to steer a plan, users do not settle on a single preferred interaction type. Instead, they assemble hybrid workflows, switching between controls depending on what they are trying to change.
A common pattern is to use text feedback for broad changes, then move to direct manipulation for local repairs. Users often favor LLM-assisted structural operations such as auto-merge and auto-split when they want to reshape the graph quickly, while preferences are more mixed for semantic or ambiguous changes.
We frame this as an effort–control–risk trade-off. Manual graph edits give users precise control but require more work. Global feedback takes less effort but feels riskier because it may change more of the plan at once. LLM-assisted structural edits often occupy a useful middle ground.
That trade-off raises a harder set of questions. Users are not only asking, “Can I make this edit?” They also have to ask, “How much effort will it take?” and “What else might this edit disturb?”
Better Controls Did Not Mean Better Plans
AMBIPOM’s advanced steering interactions make some parts of planning easier. LLM-assisted high-level structural operations reduce cognitive load compared with manual graph editing. Targeted text feedback reduces the number of conversational turns needed to revise a plan.
But those usability benefits do not translate into better plans or higher task success. In fact, participants produce slightly higher-quality plans overall with the more basic interaction types (global feedback and low-level direct manipulation). This result suggests that making controls more expressive does not automatically make planning itself more effective.
What the Plan-Revision Benchmark Reveals
We also tested plan revision from the model side. In a controlled benchmark, we asked LLMs to repair 1,150 broken variants of 200 gold plans using natural-language feedback. We compared global feedback, targeted feedback, targeted feedback with full-plan context, and edit-sequence generation.
Global feedback is the most robust overall because the model can revise the plan as a whole. By contrast, targeted feedback is better at preserving the parts of the plan outside the selected subgraph, making it attractive when the intended change is narrow and well-bounded. The challenge is reintegration: the revised subgraph does not always reconnect cleanly with the rest of the plan.
Edit-sequence generation offers a different trade-off. Representing a revision as a series of explicit edit operations makes the process more interpretable and auditable, but complex revisions become more fragile because errors can accumulate across the sequence.
For agentic system builders, the takeaway is that revision strategy matters: different approaches trade off robustness, stability, and interpretability in different ways.
For users, that means the challenge does not end when the edit is made. They still have to verify that the revised plan works as intended.
The Hidden Bottleneck Is Verification
Taken together, the user study and benchmark point to a deeper challenge: making a change is only part of the work. Users still need to verify that the revised plan is correct and works as intended.
But that verification is not cheap. The user study also revealed a trust-fatigue pattern: as participants became more comfortable with LLM-assisted features, their trust increased while their verification effort declined over time. Early in the session, they checked intermediate results more carefully; later, some were more willing to accept plausible-looking outputs with lighter inspection.
The bottleneck, then, is shifting from making edits to verifying them. More ways to steer are useful, but those controls need to be paired with support for understanding what changed and where something may have gone wrong.
As plans get longer and workflows become more complex, verification cannot depend entirely on a user manually checking every node and edge. The system should help surface likely failure points, such as missing input/output variables, broken dependencies, or execution risks.
In other words, plan editors need verification support as a first-class design feature.
The Takeaway
The next generation of agentic systems will need better collaboration surfaces.
AMBIPOM shows how users make use of multiple ways to steer: talking through broad changes, targeting feedback to a subplan, editing the structure directly, or using LLM assistance for higher-level structural changes.
However, a plan is only useful if users can understand what changed, why it changed, and whether the revised plan still works. That points to a few practical directions for future agentic systems: verification and integration support, such as previews; proactive, context-aware guidance on what to do next; and specialized views for domain- or application-specific output inspection.
The broader lesson is that steering multi-agent systems is as much a verification problem as an interaction problem. As these systems become more capable, the interface for inspecting, revising, and checking their plans will matter just as much as the controls themselves.
Read the research: https://arxiv.org/abs/2605.23023
Try AMBIPOM: https://github.com/megagonlabs/ambipom
Written by: Hannah Kim and Megagon Labs
