Publications

omnes flores is an NLP framework based on Universal Dependencies (UD) that utilizes multilingual Large Language Models (LLMs), and its default model is trained on data from 40 UD languages comprising 40 treebanks. For the EvaLatin 2026 Dependency Parsing Tasks, we extended the training data of omnes flores by incorporating six public Latin treebanks from UD and trained a dependency parsing model using the extended training data. The dependency parser of omnes flores normally takes a list of word FORM values as input. However, since the EvaLatin 2026 test data includes an UPOS column, we investigated whether incorporating both FORM and UPOS during both training and inference could improve parsing accuracy. Our experiments show that training using both FORM and UPOS improves performance by 0.5-1.0 LAS points on Prose compared with training using only FORM, but decreases performance by 5 points on Poetry.

In this research, we introduce LoRA probing, a lightweight approach for observing how core syntactic abilities emergeduring LLM pretraining. Leveraging OLMo-2’s public intermediate checkpoints, we trace learning curves across 24 pretraining stages on 33 Universal Dependencies languages by fine-tuning LoRA with step-by-step parsing instructions and a simple tabular output. To fit the relatively short context length of the OLMo-2, we design a compact 2-step-no-form prompt template and this matches the baseline in average accuracy while halving the context length and substantially increasing throughput, enabling efficient large-scale evaluation. Token Recall surpasses 0.9 within the first 1–2K pretraining steps, indicating that stable output formatting emerges early. Despite OLMo-2-7B’s English-centric pretraining, LAS exceeds 80 points in 29 of 33 languages; however, relations such as iobj and csubj show delayed onset and instability across many languages. LoRA probing thus provides a practical, reproducible lens on the cross-lingual dynamics of syntactic acquisition during LLM pretraining.

大規模言語モデルは差別的な社会的バイアスを含む情報を生成するリスクがあり，その評価が必要である．しかし，何を「差別的な社会的バイアス」とみなすかは社会的文脈に依存するため，普遍的な価値基準を定義することは難しく，個々の社会的文脈において合意可能な価値基準に基づいた安全性担保を行う必要がある．社会的バイアスのベンチマーク構築に関する先行研究では，社会的文脈の一つである国による価値基準の相違に対応するため，当該国のアノテーターを用いてローカライゼーションを行っているが，この手法はアノテーターの主観に強く依存しており，安全性の判断が当該国において合意可能なものであることを明確には担保していない．本研究では，各国において合意された「差別的な社会的バイアス」の最低限の基準として，法令とその判例等を根拠とする安全性担保のローカライゼーションを提案し，日本の雇用関連領域および医療提供関連領域の法令において差別と判断された事例を収集して，社会的バイアスデータセット – JLawBias を構築し，6 つの日本語対応 LLM に対して簡易な評価を実施して手法の有効性を確認した． ここに掲載した著作物の利用に関する注意 本著作物の著作権は人工知能学会に帰属します。本著作物は著作権者である人工知能学会の許可のもとに掲載するものです。ご利用に当たっては「著作権法」に従うことをお願いいたします。 Notice for the use of this material. The copyright of this material is retained by the Japanese Society for Artificial Intelligence (JSAI). This material is published here with the agreement of JSAI. Please be complied with Copyright Law of Japan if any users wish to reproduce, make derivative work, distribute or make available to the public any part or whole thereof. All Rights Reserved, Copyright (C) The Japanese Society for Artificial Intelligence.

Data lakes in modern enterprises are massive, heterogeneous, and noisy, often preventing non-experts from effectively extracting value. Bridging the semantic gap between ambiguous user intent and explicit data requires orchestrating multiple tools under an open-world assumption. However, reliably executing these compound AI workflows to solve complex knowledge extraction tasks, while also providing the transparency needed for evaluation and debugging, remains a significant bottleneck. We propose L.A.K.E. (Logic Agent for Knowledge Extraction), an agentic data planning framework designed to map natural language questions to executable workflows over diverse data sources. Rather than relying on a brittle “one-size-fits-all” approach, L.A.K.E. dynamically generates a declarative plan comprised of modular operators—spanning relational and semantic functions over heterogeneous data sources. Within this framework, we introduce and benchmark three distinct planning regimes: Iterative Planning, Single-Shot Tree Planning, and Cascade Planning. We present an interactive demonstration platform that enables users to visually compare the latency and robustness trade-offs of these planners. By rendering execution paths as interactive Directed Acyclic Graphs (DAGs) with step-level provenance, L.A.K.E. provides the critical observability needed to establish trust, debug failures, and optimize data planning for enterprise-scale lakes.

We investigate how agents built on pretrained large language models (LLMs) can learn target classification functions from labeled examples without parameter updates. While conventional approaches like fine-tuning are often costly, inflexible, and opaque, we propose a memory-augmented framework that leverages LLM-generated critiques grounded in labeled data. Our framework uses episodic memory to store instance-level critiques – capturing specific past experiences – and semantic memory to distill these into reusable, task-level guidance. Across a diverse set of tasks and models, our best performing self-critique strategy (utilizing both memory types) yields an average improvement of 8.1 percentage points over the zero shot baseline, and 4.6pp over a RAG-based baseline that relies only on labels. However, improvements vary substantially across models and domains. To explain this variation, we introduce suggestibility – a novel metric capturing how receptive a model is to external reasoning provided in context. We use suggestibility to illuminate when and why memory augmentation succeeds or falls short. Beyond accuracy gains, we find pre-computed critiques substantially reduce inference-time computation for reasoning models, cutting thinking tokens by an average of 31.95% across all datasets by substituting for reasoning that the model would otherwise perform independently. Our findings highlight the conditions under which memory-driven, reflective learning can serve as a lightweight, interpretable, and efficient strategy for improving LLM adaptability.

The extraction of structured information from raw text is a fundamental component of many NLP applications, including document retrieval, ranking, and relevance estimation. High-quality extractions often require domain-specific accuracy, up-to-date understanding of specialized taxonomies, and the ability to incorporate emerging jargon and rare outliers. In many domains–such as medical, legal, and HR–the extraction model must also adapt to shifting terminology and benefit from explicit reasoning over structured knowledge. We propose DySECT, a Dynamic Self-Evolving Extraction and Curation Toolkit, which continually improves as it is used. The system incrementally populates a versatile, self-expanding knowledge base (KB) with triples extracted by the LLM. The KB further enriches itself through the integration of probabilistic knowledge and graph-based reasoning, gradually accumulating domain concepts and relationships. The enriched KB then feeds back into the LLM extractor via prompt tuning, sampling of relevant few-shot examples, or fine-tuning using KB-derived synthetic data. As a result, the system forms a symbiotic closed-loop cycle in which extraction continuously improves knowledge, and knowledge continuously improves extraction.

As language models gain access to external tools via structured function calls, they become increasingly more capable of solving complex, multi-step tasks. However, existing benchmarks for tool-augmented language models (TaLMs) provide insufficient control over factors such as the number of functions accessible, task complexity, and input size, and remain vulnerable to data contamination. We present FuncBenchGen, a unified, contamination-free framework that evaluates TaLMs by generating synthetic multi-step tool-use tasks. The key idea is to cast tool use as traversal over a hidden function-dependency DAG where nodes are function calls and an edge between nodes represents one function consuming the output of another. Given a set of external function schemas, initial variable values, and a target variable, models must compose the correct call sequence to compute the target variable. FuncBenchGen allows users to precisely control task difficulty (e.g., graph size, dependency depth, and distractor functions) while avoiding data leakage. We apply our FuncBenchGen framework to evaluate seven LLMs on tool use tasks of varying difficulty. Reasoning-optimized models consistently outperform general-purpose models with GPT-5 significantly outperforming other models. Performance declines sharply as dependency depth increases. Furthermore, connected irrelevant functions prove especially difficult to handle. We find that strong models often make syntactically valid function calls but propagate incorrect or stale argument values across steps, revealing brittle state tracking by LLMs in multi-turn tool use. Motivated by this observation, we introduce a simple mitigation strategy that explicitly restates prior variable values to the agent at each step. Surprisingly, this lightweight change yields substantial gains across models. e.g., yielding a success rate improvement from 62.5% to 81.3% for GPT-5.

Large language model (LLM) agents are increasingly deployed to tackle complex tasks, often necessitating collaboration among multiple specialized agents. However, multi-agent collaboration introduces new challenges in planning, coordination, and verification. Execution failures frequently arise not from flawed reasoning alone, but from subtle misalignments in task interpretation, output format, or inter-agent handoffs. To address these challenges, we present VeriMAP, a framework for multi-agent collaboration with verification-aware planning. The VeriMAP planner decomposes tasks, models subtask dependencies, and encodes planner-defined passing criteria as subtask verification functions (VFs) in Python and natural language. We evaluate VeriMAP on diverse datasets, demonstrating that it outperforms both single- and multi-agent baselines while enhancing system robustness and interpretability. Our analysis highlights how verification-aware planning enables reliable coordination and iterative refinement in multi-agent systems, without relying on external labels or annotations.

Skill mapping is a key task in the Human Resources domain. It consists in identifying ontology-defined skills in job texts. Among the most successful approaches applied to skill mapping, bi-encoders offer efficient inference but struggle with fine-grained skill distinctions, particularly under limited supervision. While accurate, cross-encoder and LLM-based reranking approaches are computationally expensive and usually not feasible to be adopted in real case scenarios. We propose kNNBE, a hybrid inference method that augments bi-encoder similarity scores with k-nearest labeled sentences drawn from a synthetic memory bank. kNNBE improves both prediction accuracy and generalization to unseen skills while retaining high throughput. Extensive experiments on three benchmark datasets show that kNNBE rivals state-of-the-art rerankers in accuracy while being orders of magnitude faster.

Businesses are increasingly overwhelmed by inquiries related to their services or products. Relying on human agents to handle inquiries via email results in higher costs and delayed responses, contributing to customer dissatisfaction. In response to these challenges, this pilot study leverages advancements in Large Language Models (LLMs) by proposing a fully automated method for generating a knowledge graph from unstructured data in help pages, which is then utilized to power a fully automated dialogue management system. By transitioning to a chat-based approach, our method aims to handle ambiguous, incomplete, or nonspecific inquiries more effectively and enhance customer satisfaction with tailored, natural responses. We also implement explicit safeguards to improve intent identification and prevent response hallucinations. We validate our proposal in the hotel industry, demonstrating that our knowledge graph based AI agent outperforms the baseline Retrieval-Augmented Generation (RAG) model in accuracy while facilitating more natural and coherent dialogues.