複合AIシステム

大規模言語モデル(LLM)は、多様なタスクにおいて優れた能力を示し、エージェント型アプリケーションの新時代を切り開いています。特に、単一の巨大なモデル(モノリシックモデル)から、データ・モデルの検索、タスクの調整・計画、推論、内省と学習、さらには知識統合を補完・拡張する AI システムやアーキテクチャへの移行が進んでいます。このような「複合 AI システム」は、複雑なタスクのパフォーマンス向上、異なるアプリケーションへの柔軟な適応、既存のモデルやデータの統合の容易化、さらには制御性と信頼性の向上を約束するものです。

私たちは、エンタープライズ向けに最適化された複合AI システムの青写真となるアーキテクチャの構築に取り組んでいます。私たちが考慮する主な要素は以下のとおりです。

(1) 既存のインフラとのシームレスな統合: 適切なタッチポイントやインターフェースを通じて、 複合 AI システムを現在のシステムと円滑に統合できるようにする。

(2) システム内外のワークフローの効果的なオーケストレーション: 適切なリソース配分を行いながら、 複合 AI システム内および外部のプロセスを調整し、最適なタスク遂行を実現する。

(3) コスト効率の最大化: レイテンシー、正確性、コスト、可用性、 品質といった制約を考慮しながら、システムの有効活用を最大化する。

ハイライト

プロジェクト

Blue

Blue は、エンタープライズ向けのエージェント型 AI プラットフォームです。 Blue では、システム指向のアプローチを採用し、信頼性が高く、効果的で実用的な AI アプリケーションの開発を目指しています。

関連

研究論文

CHI - HEAL Workshop
2025
Yoo Yeon Sung, Hannah Kim, Dan Zhang
AI practitioners increasingly use large language model (LLM) agents in compound AI systems to solve complex reasoning tasks, these agent executions often fail to meet human standards, leading to errors that compromise the system’s overall performance. Addressing these failures through human intervention is challenging due to the agents’ opaque reasoning processes, misalignment with human expectations, the complexity of agent dependencies, and the high cost of manual inspection. This paper thus introduces a human-centered evaluation framework for Verifying LLM Agent failures (VeriLA), which systematically assesses agent failures to reduce human effort and make these agent failures interpretable to humans. The framework first defines clear expectations of each agent by curating human-designed agent criteria. Then, it develops a human-aligned agent verifier module, trained with human gold standards, to assess each agent’s execution output. This approach enables granular evaluation of each agent’s performance by revealing failures from a human standard, offering clear guidelines for revision, and reducing human cognitive load. Our case study results show that VeriLA is both interpretable and efficient in helping practitioners interact more effectively with the system. By upholding accountability in human-agent collaboration, VeriLA paves the way for more trustworthy and human-aligned compound AI systems.
CIKM - Demo
2024
Chen Shen, Jin Wang, Sajjadur Rahman, Eser Kandogan
The Text-to-SQL problem aims at developing natural language query interfaces for relational database systems by converting the text input into executable SQL queries. Recently, using Large Language Models (LLM) has emerged as a new paradigm for the Text-to-SQL problem. To this end, the LLM needs to understand not only user input but also information from the database. In this demo, we present multi-agent SQL (MageSQL), an LLM based Text-to-SQL approach that tackles the task by orchestrating multiple agents in a pipeline. We will showcase a user-friendly interface to demonstrate the inner workings of our approach that allows users to add and modify the agents with different functionalities, customize prompts, and see their impact on specific examples. Through several use cases, we will demonstrate how to (i) construct a Text-to-SQL pipeline with multiple agents; (ii) generate prompts for LLM with various templates and strategies; and (iii) monitor the results of natural language queries and perform debugging.
Data + AI Summit - Compound AI Systems Workshop
2024
Large Language Models (LLMs) have showcased remarkable capabilities surpassing conventional NLP challenges, creating opportunities for use in production use cases. Towards this goal, there is a notable shift to building compound AI systems, wherein LLMs are integrated into an expansive software infrastructure with many components like models, retrievers, databases and tools. In this paper, we introduce a blueprint architecture for compound AI systems to operate in enterprise settings cost-effectively and feasibly. Our proposed architecture aims for seamless integration with existing compute and data infrastructure, with “stream” serving as the key orchestration concept to coordinate data and instructions among agents and other components. Task and data planners, respectively, break down, map, and optimize tasks and data to available agents and data sources defined in respective registries, given production constraints such as accuracy and latency.
Data + AI Summit - Compound AI Systems Workshop
2024
Remarkable performance of large language models (LLMs) in a variety of tasks brings forth many opportunities as well as challenges of utilizing them in production settings. Towards practical adoption of LLMs, multi-agent systems hold great promise to augment, integrate, and orchestrate LLMs in the larger context of enterprise platforms that use existing proprietary data and models to tackle complex real-world tasks. Despite the tremendous success of these systems, current approaches rely on narrow, single-focus objectives for optimization and evaluation, often overlooking potential constraints in real-world scenarios, including restricted budgets, resources and time. Furthermore, interpreting, analyzing, and debugging these systems requires different components to be evaluated in relation to one another. This demand is currently not feasible with existing methodologies. In this postion paper, we introduce the concept of reasoning capacity as a unifying criterion to enable integration of constraints during optimization and establish connections among different components within the system, which also enable a more holistic and comprehensive approach to evaluation. We present a formal definition of reasoning capacity and illustrate its utility in identifying limitations within each component of the system. We then argue how these limitations can be addressed with a self-reflective process wherein human-feedback is used to alleviate shortcomings in reasoning and enhance overall consistency of the system.
1 Min Read
February 5, 2025
MCRankベンチマークとEXSIR手法を用いることで、構造化された推論によりLLMの性能がこれらの難解なタスクで大幅に向上することを示しました。
1 Min Read
December 16, 2024
複合AIシステムの最適化フレームワークは、精度・コスト・遅延などの多目的最適化や複数プランの最適化、特に予算などの制約管理を含む幅広い目標を達成すべきであると述べています。これらの最適化目標は決して網羅的ではありませんが、エンタープライズ環境において重要な要素です。
1 Min Read
June 12, 2024
Megagon Labsの研究者たちは、企業向けの複合AIシステム構築における課題に取り組んでいます。本ブログでは、私たちが進めている3つのプロジェクトを紹介します。(1) 複合AIシステムを製品化するための適切なアーキテクチャの開発、(2) 実環境の制約を考慮したエージェントワークフローの最適化、(3) エンタープライズ環境における複合AIシステム内のエージェントのパフォーマンスベンチマークの確立です。