Traditional dialog state tracking is often formulated as a classification problem, where the dialog state is predicted as a distribution over a closed set of possible slot values within an ontology [1]. By doing so, the tracked dialog state which is a summary of current conversation, can be directly used for backend database querying or API calls in dialog system applications [2]. For example, a virtual assistant that helps users to book hotels, may fill a slot price_range with three possible values — high / low / intermediate defined by the query parameters of the backend hotel database. However, this classification
approach of dialog state tracking may have two main disadvantages. First, the system will not be able to handle any unseen values beyond the predefined value set. While building a classifier that covers all possible values will result in impractical annotation and training cost. Second, we see that many modern designed databases [3] and search engine API can take direct natural language input as the search query. A predefined value set will limit the user query space and cause the matched results lack of variety.
Yuliang Li, Jinfeng Li, Yoshihiko Suhara, Jin Wang, Wataru Hirota, Wang-Chiew Tan
Entity matching refers to the task of determining whether two different representations refer to the same real-world entity. It continues to be a prevalent problem for many organizations where data resides in different sources and duplicates the need to be identified and managed. The term “entity matching” also loosely refers to the broader problem of determining whether two heterogeneous representations of different entities should be associated together. This problem has an even wider scope of applications, from determining the subsidiaries of companies to matching jobs to job seekers, which has impactful consequences.
In this article, we first report our recent system DITTO, which is an example of a modern entity matching system based on pretrained language models. Then we summarize recent solutions in applying deep learning and pre-trained language models for solving the entity matching task. Finally, we discuss research directions beyond entity matching, including the promise of synergistically integrating blocking and entity matching steps together, the need to examine methods to alleviate steep training data requirements that are typical of deep learning or pre-trained language models, and the importance of generalizing entity matching solutions to handle the broader entity matching problem, which leads to an even more pressing need to explain matching outcomes.
With the increase in scale and availability of digital text generated on the web, enterprises such as online retailers and
aggregators often use text analytics to mine and analyze the
data to improve their services and products alike. Text data
analysis is an iterative, non-linear process with diverse workflows spanning multiple stages, from data cleaning to visualization. Existing text analytics systems usually accommodate
a subset of these stages and often fail to address challenges
related to data heterogeneity, provenance, workflow reusability and reproducibility, and compatibility with established
practices. Based on a set of design considerations we derive
from these challenges, we propose Leam, a system that treats
the text analysis process as a single continuum by combining
advantages of computational notebooks, spreadsheets, and
visualization tools. Leam features an interactive user interface
for running text analysis workflows, a new data model for
managing multiple atomic and composite data types, and an
expressive algebra that captures diverse sets of operations
representing various stages of text analysis and enables coordination among different components of the system, including
data, code, and visualizations. We report our current progress
in Leam development while demonstrating its usefulness with
usage examples. Finally, we outline a number of enhancements
to Leam and identify several research directions for developing
an interactive visual text analysis system.
Shin Kanouchi, Masato Neishi, Yuta Hayashibe, Hiroki Ouchi, Naoaki Okazaki
Explainable recommendation is a good way to improve user satisfaction. However, explainable recommendation in dialogue is challenging since it has to handle natural language as both input and output. To tackle the challenge, this paper proposes a novel and practical task to explain evidences in recommending hotels given vague requests expressed freely in natural language. We decompose the process into two subtasks on hotel reviews: Evidence Identification and Evidence Explanation. The former predicts whether or not a sentence contains evidence that expresses why a given request is satisfied. The latter generates a recommendation sentence given a request and an evidence sentence. In order to address these subtasks, we build an Evidence-based Explanation dataset, which is the largest dataset for explaining evidences in recommending hotels for vague requests. The experimental results demonstrate that the BERT model can find evidence sentences with respect to various vague requests and that the LSTM-based model can generate recommendation sentences.
For NLP, sentence boundary detection (SBD) is an essential task to decompose a text into sentences. Most of the previous studies have used a simple rule that uses only typical characters as sentence boundaries. However, some characters may or may not be sentence boundaries depending on the context. We focused on line breaks in them. We newly constructed annotated corpora, implemented sentence boundary detectors, and analyzed performance of SBD in several settings.
Data programming aims to reduce the cost of curating training data by encoding domain knowledge as labeling functions over source data. As such it not only requires domain expertise but also programming experience, a skill that many subject matter experts lack. Additionally, generating functions by enumerating rules is not only time consuming but also inherently difficult, even for people with programming experience. In this paper we introduce Ruler, an interactive system that synthesizes labeling rules using span-level interactive demonstrations over document examples. Ruler is a first-of-a-kind implementation of data programming by demonstration (DPBD). This new framework aims to relieve users from the burden of writing labeling functions, enabling them to focus on higher-level semantic analysis, such as identifying relevant signals for the labeling task. We compare Ruler with conventional data programming through a user study conducted with 10 data scientists who were asked to create labeling functions for sentiment and spam classification tasks. Results show Ruler is easier to learn and to use, and that it offers higher overall user-satisfaction while providing model performances comparable to those achieved by conventional data programming.
Johannes Bjerva, Nikita Bhutani, Behzad Golshan, Wang-Chiew Tan, Isabelle Augenstein
Subjectivity is the expression of internal opinions or beliefs which cannot be objectively observed or verified, and has been shown to be important for sentiment analysis and word-sense disambiguation. Furthermore, subjectivity is an important aspect of user-generated data. In spite of this, subjectivity has not been investigated in contexts where such data is widespread, such as in question answering (QA). We develop a new dataset which allows us to investigate this relationship. We find that subjectivity is an important feature in the case of QA, albeit with more intricate interactions between subjectivity and QA performance than found in previous work on sentiment analysis. For instance, a subjective question may or may not be associated with a subjective answer. We release an English QA dataset (SubjQA) based on customer reviews, containing subjectivity annotations for questions and answer spans across 6 domains.
Dan Zhang, Yoshihiko Suhara, Jinfeng Li, Madelon Hulsebos, Çağatay Demiralp, Wang-Chiew Tan
Detecting the semantic types of data columns in relational
tables is important for various data preparation and information retrieval tasks such as data cleaning, schema matching, data discovery, and semantic search. However, existing
detection approaches either perform poorly with dirty data,
support only a limited number of semantic types, fail to
incorporate the table context of columns or rely on large
sample sizes for training data. We introduce Sato, a hybrid machine learning model to automatically detect the
semantic types of columns in tables, exploiting the signals
from the table context as well as the column values. Sato
combines a deep learning model trained on a large-scale table corpus with topic modeling and structured prediction
to achieve support-weighted and macro average F1 scores
of 0.925 and 0.735, respectively, exceeding the state-of-theart performance by a significant margin. We extensively
analyze the overall and per-type performance of Sato, discussing how individual modeling components, as well as feature categories, contribute to its performance.
