Abstract—Large language models (LLMs) have gained signifi-
cant interest in industry due to their impressive capabilities across
a wide range of tasks. However, the widespread adoption of LLMs
presents several challenges, such as integration into existing ap-
plications and infrastructure, utilization of company proprietary
data, models, and APIs, and meeting cost, quality, responsiveness,
and other requirements. To address these challenges, there is a
notable shift from monolithic models to compound AI systems,
with the premise of more powerful, versatile, and reliable
applications. However, progress thus far has been piecemeal,
with proposals for agentic workflows, programming models, and
extended LLM capabilities, without a clear vision of an overall
architecture. In this paper, we propose a ‘blueprint architecture’
for compound AI systems for orchestrating agents and data
for enterprise applications. In our proposed architecture the
key orchestration concept is ‘streams’ to coordinate the flow of
data and instructions among agents. Existing proprietary models
and APIs in the enterprise are mapped to ‘agents’, defined
in an ‘agent registry’ that serves agent metadata and learned
representations for search and planning. Agents can utilize
proprietary data through a ‘data registry’ that similarly registers
enterprise data of various modalities. Tying it all together, data
and task ‘planners’ break down, map, and optimize tasks and
queries for given quality of service (QoS) requirements such as
cost, accuracy, and latency. We illustrate an implementation of
the architecture for a use-case in the HR domain and discuss
opportunities and challenges for ‘agentic AI’ in the enterprise.
Index Terms—Agents, Agentic Workflows, LLMs, AI System.