What is Human-Centered AI research at Megagon Labs?

Megagon Labs' Human-Centered AI (HAI) research focuses on bridging the gap between human expertise and AI-driven processes, with an emphasis on facilitating seamless human-AI collaboration — particularly between humans and LLMs. The work tackles challenges of incorporating humans throughout the machine learning lifecycle to improve model performance, ensure fairness, and align AI outputs with ethical standards.

What human-in-the-loop tools has Megagon Labs released?

Megagon Labs has released several open-source frameworks for human-AI collaboration, including MEGAnno (an annotation framework combining LLMs with human expertise), Lapras (a multi-step human-LLM collaborative annotation system with self-explanation), Tyrogue (active learning with hybrid sampling to minimize labeling costs), and FactLens (a benchmark for evaluating fine-grained fact verification).

Human-Centered AI

The rapid advancements in LLMs have revolutionized machine learning (ML) tasks in and beyond natural language processing (NLP) domains by making complex capabilities more accessible to a broader range of users. However, this accessibility also presents significant challenges, particularly in effectively incorporating humans into the ML lifecycle. From model training to decision-making, ensuring that human input is meaningfully integrated remains critical for improving model performance, enhancing fairness, and aligning outputs with user intentions and ethical standards. As such, it has become increasingly important to create systems that thoughtfully bridge the gap between human expertise and AI-driven processes.

At Megagon Labs, the HAI team’s mission is to advance human-centered AI by facilitating seamless human-AI collaboration, with a particular emphasis on interactions between humans and LLMs. We develop innovative tools, workflows, and solutions that empower diverse stakeholders throughout the ML lifecycle to contribute their knowledge and preferences. We create conversational interfaces and interactive planning solutions that bridge the gap between users and intelligent agents in compound AI systems. Through these efforts, we aim to redefine how humans and AI systems work together, enabling more intuitive, transparent, and impactful collaborations in complex, real-world contexts.

Highlighted

Projects

Lapras

Multi-step human-LLM collaborative framework for effective and explainable annotation. The framework utilizes LLMs’ self-explanation capabilities to explain their labeling decisions and provide signals for human verification.

MEGAnno

LLM-equipped open-source data annotation framework, enabling ML practitioners to bootstrap annotation tasks and manage the continual evolution of annotations through the machine learning lifecycle.

Tyrogue

Tyrogue is an active learning method that employs a hybrid sampling strategy to minimize labeling cost and acquisition latency while providing a framework for adapting to dataset diversity via user guidance.

FactLens

Benchmark for evaluating fine-grained fact verification, with metrics and automated evaluators of sub-claim quality for complex claims.

Publications

CAIS

2026

Agentic AI Human Centered AI

How to Steer Your Multi-Agent System: Human-LLM Collaborative Planning

Zeyu He, Hannah Kim, Dan Zhang, Estevam Hruschka

In orchestrated multi-agent systems, humans often struggle to manage plans due to their complexity and limited transparency. Existing approaches rely on outcome-level supervision, where users verify only final outputs without visibility into intermediate reasoning. We formalize a design space for human-LLM co-planning interactions along three axes: mode (semantic vs. structural), scope (global vs. targeted), and level (low- vs. high-level edits). We realize it in AMBIPOM, a prototype supporting process-level supervision through both semantic and structural interactions. Through a user study, we characterize how users navigate this space, revealing hybrid workflows and effort-control-risk trade-offs; through a controlled benchmark, we analyze how LLMs revise plans under varying scope and revision strategies. Our findings yield design insights for more transparent, controllable, and effective human-AI co-planning. We release code and data at https://github.com/megagonlabs/ambipom.

EACL - Demonstrations

2024

MEGAnno+: A Human-LLM Collaborative Annotation System

Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, Dan Zhang

Large language models (LLMs) can label data faster and cheaper than humans for various NLP tasks. Despite their prowess, LLMs may fall short in understanding of complex, sociocultural, or domain-specific context, potentially leading to incorrect annotations. Therefore, we advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels. We present MEGAnno+, a human-LLM collaborative annotation system that offers effective LLM agent and annotation management, convenient and robust LLM annotation, and exploratory verification of LLM labels by humans.

Findings - EMNLP

2022

Low-resource Interactive Active Labeling for Fine-tuning Language Models

Seiji Maekawa, Dan Zhang, Hannah Kim, Sajjadur Rahman, and Estevam Hruschka

Recently, active learning (AL) methods have been used to effectively fine-tune pre-trained language models for various NLP tasks such as sentiment analysis and document classification. However, given the task of fine-tuning language models, understanding the impact of different aspects on AL methods such as labeling cost, sample acquisition latency, and the diversity of the datasets necessitates a deeper investigation. This paper examines the performance of existing AL methods within a low-resource, interactive labeling setting. We observe that existing methods often underperform in such a setting while exhibiting higher latency and a lack of generalizability. To overcome these challenges, we propose a novel active learning method TYROUGE that employs a hybrid sampling strategy to minimize labeling cost and acquisition latency while providing a framework for adapting to dataset diversity via user guidance. Through our experiments, we observe that compared to SOTA methods, TYROUGE reduces the labeling cost by up to 43% and the acquisition latency by as much as 11X, while achieving comparable accuracy. Finally, we discuss the strengths and weaknesses of TYROUGE by exploring the impact of dataset characteristics.

BLOG

Articles

4 Min Read

July 31, 2024

MEGAnno in Action: Human-LLM Collaborative Annotation

MEGAnno combines the power of large language models (LLMs) with human expertise to streamline and enhance the data labeling process with a data annotation framework. Throughout this article, we’ll showcase MEGAnno’s capabilities as we provide detailed code snippets.

4 Min Read

May 8, 2024

Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels

Instead of completely replacing human annotators with LLMs, we need to leverage the strengths of both sides to obtain accurate and reliable annotations. This article will discuss how to effectively utilize LLMs as collaborators for data annotation.

8 Min Read

March 14, 2024

LLMs as Data Annotators (Part 2) – MEGAnno+: A Human-LLM Collaborative Annotation System

We introduce our human-LLM collaborative annotation tool, MEGAnno+, addressing the challenges in LLM annotation by integrating human expertise with LLM capabilities.