Unlocking the Potential of Transformers for Long-Form Text Matching: A Simple yet Powerful Approach
Long-form text matching is a critical problem to solve in the field of Natural Language Processing (NLP) and Information Retrieval (IR). We propose a simple yet effective solution using sequence pair classification with Transformer models, demonstrating its superiority over state-of-the-art Siamese network-based methods.
Order Matters: Assessing LLM Sensitivity in Multiple-Choice Tasks
Explore the relationship between option arrangement and performance variations in Large Language Models (LLMs) during multiple-choice tasks. Through meticulous analysis, we uncovered substantial sensitivity of LLMs to the order of answer options, with performance fluctuations of up to 75% across different benchmarks.
Towards Enterprise Compound AI Systems
Researchers at Megagon Labs have been exploring how we can address the challenges of building compound AI systems for enterprises. In this blog post, we introduce three projects that we have undertaken: (1) developing a suitable architecture for productizing compound AI systems, (2) optimizing agentic workflows with real-world constraints, and (3) benchmarking the performance of agents within a compound AI system, specifically in an enterprise setting.
Deep Dive with WiTQA: When Does Retrieval Augmentation Help (or Hurt) Language Models?
The article presents the WiTQA dataset, designed to assess the impact of retrieval on the performance of language models in question-answering systems. It details the findings on when retrieval augmentation enhances QA accuracy and when it may introduce errors, providing valuable guidance for optimizing RALMs.
Watchog: Leveraging Contrastive Learning for Enhanced Table Understanding and Column Annotation
By enabling robust and accurate column annotation, this innovative framework holds the potential to revolutionize data-driven decision-making processes across a multitude of industries. Watchog could empower businesses to extract valuable insights from product catalogs, pricing tables, and customer data repositories and use it to optimize their pricing strategies, and deliver personalized recommendations to enhance customer satisfaction and loyalty.
XATU: A Fine-grained Instruction-based Benchmark for Explainable Text Updates
To push the boundaries of text editing with LLMs, we introduce XATU—a new text editing benchmark that incorporates fine-grained instructions and gold-standard edit explanations for explainable text updates.
Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels
Instead of completely replacing human annotators with LLMs, we need to leverage the strengths of both sides to obtain accurate and reliable annotations. This article will discuss how to effectively utilize LLMs as collaborators for data annotation.
LLMs as Data Annotators (Part 2) – MEGAnno+: A Human-LLM Collaborative Annotation System
We introduce our human-LLM collaborative annotation tool, MEGAnno+, addressing the challenges in LLM annotation by integrating human expertise with LLM capabilities.
LLMs as Data Annotators (Part 1) – Challenges and Opportunities
We discuss how to leverage LLMs as data annotation agents and the practical challenges that may arise. We briefly introduce our LLM annotation tool, MEGAnno+.
Less Is More for Long Document Summary Evaluation by LLMs
In the realm of text generation and summarization, the evaluation of generated summaries, especially for long documents, has always been a challenging task. To address these challenges, we evaluate long models using an innovative approach that significantly reduces evaluation costs and aligns more closely with human evaluations.
