Entity set expansion (ESE) aims at obtaining a more complete set of entities given a textual corpus and a seed set of entities of a concept. Although it is a critical task in many NLP applications, existing benchmarks are limited to well-formed text (e.g., Wikipedia) and well-defined concepts (e.g., countries and diseases). Furthermore, only a small number of predictions are evaluated compared to the actual size of an entity set. A rigorous assessment of ESE methods warrants more comprehensive benchmarks and evaluation. In this paper, we consider user-generated text to understand the generalizability of ESE methods. We develop new benchmarks and propose more rigorous evaluation metrics for assessing the performance of ESE methods. Additionally, we identify phenomena such as non-named entities, multifaceted entities, vague concepts that are more prevalent in user-generated text than well-formed text, and use them to profile ESE methods. We observe that the strong performance of state-of-the-art ESE methods does not generalize well to user-generated text. We conduct comprehensive empirical analysis and draw insights from the findings.
Reinald Kim Amplayo, Arthur Bražinskas, Yoshihiko Suhara, Xiaolan Wang, Bing Liu
Customer reviews are vital for making purchasing decisions in the Information Age. Such reviews can be automatically summarized to provide the user with an overview of opinions. In this tutorial, we present various aspects of opinion summarization that are useful for researchers and practitioners. First, we will introduce the task and major challenges. Then, we will present existing opinion summarization solutions, both pre-neural and neural. We will discuss how summarizers can be trained in the unsupervised, fewshot, and supervised regimes. Each regime has roots in different machine learning methods, such as auto-encoding, controllable text generation, and variational inference. Finally, we will discuss resources and evaluation methods and conclude with the future directions. This three-hour tutorial will provide a comprehensive overview over major advances in opinion summarization. The listeners will be well-equipped with the knowledge that is both useful for research and practical applications.
Real-world applications frequently seek to solve a general form of the Entity Matching (EM) problem to find associated entities. Such scenarios include matching jobs to candidates in job targeting, matching students with courses in online education, matching products with user reviews on e-commercial websites, and beyond. These tasks impose new requirements such as matching data entries with diverse formats or having a flexible and semantics-rich matching definition, which are beyond the current EM task formulation or approaches.
In this paper, we introduce the problem of Generalized Entity Matching (GEM) that satisfies these practical requirements and presents an end-to-end pipeline Machop as the solution. Machop allows end users to define new matching tasks from scratch and apply them to new domains in a step-by-step manner. Machop cast the GEM problem as sequence pair classification so as to utilize the language understanding capability of Transformers-based language models (LMs) such as BERT. Moreover, it features a novel external knowledge injection approach with structure-aware pooling methods that allow domain experts to guide the LM to focus on the key matching information thus further contributing to the overall performance. Our experiments and case studies on real-world datasets from a popular recruiting platform show a significant 17.1% gain in F1 score against state-of-the-art methods along with meaningful matching results that are human understandable.
Often both an utterance and its context must be read to understand its intent in a dialog. Herein we propose a task, SelfContained Utterance Description (SCUD), to describe the intent of an utterance in a dialog with multiple simple natural sentences without the context. If a task can be performed concurrently with high accuracy as the conversation continues such as in an accommodation search dialog, the operator can easily suggest candidates to the customer by inputting SCUDs of the customer’s utterances to the accommodation search system. SCUDs can also describe the transition of customer requests from the dialog log. We construct a Japanese corpus to train and evaluate automatic SCUD generation. The corpus consists of 210 dialogs containing 10,814 sentences. We conduct an experiment to verify that SCUDs can be automatically generated.
Additionally, we investigate the influence of the amount of training data on the automatic generation performance using 8,200 additional examples.
https://github.com/megagonlabs/asdc
Entity Matching (EM) is an important problem in data integration
and cleaning. More recently, deep learning techniques, especially
pre-trained language models, have been integrated into EM applica-
tions and achieved promising results. Unfortunately, the significant
performance gain comes with the loss of explainability and trans-
parency, deterring EM from the requirement of responsible data
management. To address this issue, recent studies extended ex-
plainable AI techniques to explain black-box EM models. However,
these solutions have the major drawbacks that (i) their explana-
tions do not capture the unique semantics characteristics of the
EM problem; and (ii) they fail to provide an objective method to
quantitatively evaluate the provided explanations. In this paper,
we propose Minun, a model-agnostic method to generate expla-
nations for EM solutions. We utilize counterfactual examples gen-
erated from an EM customized search space as the explanations
and develop two search algorithms to efficiently find such results.
We also come up with a novel evaluation framework based on a
student-teacher paradigm. The framework enables the evaluation
of explanations of diverse formats by capturing the performance
gain of a “student” model at simulating the target “teacher” model
when explanations are given as side input. We conduct an extensive
set of experiments on explaining state-of-the-art deep EM models
on popular EM benchmark datasets. The results demonstrate that
Minun significantly outperforms popular explainable AI methods
such as LIME and SHAP on both explanation quality and scalability.
Yoshihiko Suhara, Jinfeng Li, Yuliang Li, Dan Zhang, Cagatay Demiralp, Chen Chen, Wang-Chiew Tan
Inferring meta-information about tables, such as column headers
or relationships between columns, is an active research topic in
data management as we find many tables are missing some of this
information. In this paper, we study the problem of annotating
table columns (i.e., predicting column types and the relationships
between columns) using only information from the table itself. We
develop a multi-task learning framework (called Doduo) based on
pre-trained language models, which takes the entire table as input
and predicts column types/relations using a single model. Experimental results show that Doduo establishes new state-of-the-art
performance on two benchmarks for the column type prediction
and column relation prediction tasks with up to 4.0% and 11.9%
improvements, respectively. We report that Doduo can already
outperform the previous state-of-the-art performance with a minimal number of tokens, only 8 tokens per column. We release a
toolbox1 and confirm the effectiveness of Doduo on a real-world
data science problem through a case study.
erating summaries that reflect popular subjective information expressed in multiple online reviews. While generated summaries offer general and concise information about a particular hotel or product, the information may be insufficient to help the user compare multiple different choices. Thus, the user may still struggle with the question “Which one should I pick?” In this paper, we propose the comparative opinion summarization task, which aims at generating two contrastive summaries and one common summary from two different candidate sets of reviews. We develop a comparative summarization framework CoCoSum, which consists of two base summarization models that jointly generate contrastive and common summaries. Experimental results on a newly created benchmark CoCoTrip show that CoCoSum can produce higher-quality contrastive and common summaries than state-of-the-art opinion summarization models.
Information extraction (IE) approaches often play a pivotal role in text analysis and require significant human intervention. Therefore, a deeper understanding of existing IE practices and related challenges from a human-in-the-loop perspective is warranted. In this work, we conducted semi-structured interviews in an industrial environment and analyzed the reported IE approaches and limitations. We observed that data science workers often follow an iterative task model consisting of information foraging and sensemaking loops across all the phases of an IE workflow. The task model is generalizable and captures diverse goals across these phases (e.g., data preparation, modeling, evaluation.) We found several limitations in both foraging (e.g., data exploration) and sensemaking (e.g., qualitative debugging) loops stemming from a lack of adherence to existing cognitive engineering principles. Moreover, we identified that due to the iterative nature of an IE workflow, the requirement of provenance is often implied but rarely supported by existing systems. Based on these findings, we discuss design implications for supporting IE workflows and future research directions.
