Entity set expansion (ESE) is the task of expanding a given seed set of entities (e.g., ‘mini bar’, ‘tv unit’) for a concept (e.g., room features) using a textual corpus. The task is typically studied under low-resource settings since obtaining large-scale training data is expensive. Despite the recent progress in ESE methods, it is unclear whether their performances generalize to different domains, datasets, and evaluation methods. This can be attributed to limited evaluation metrics and benchmarks that focus only on well-written text and named entities.
We investigate the generalizability of low-resource ESE methods to user-generated text. We present unique characteristics of user-generated text and study how they impact the performance of ESE methods. These characteristics include:
- Concepts in user-generated text often have overlapping semantics because of which an entity can be associated with multiple concepts (referred to as multi-faceted entities).
- Concept-entity associations in user-generated text can be subjective and cannot be verified with an external resource or common knowledge (referred to as vague entities).
- Non-named entities are more prevalent in user-generated text than well-curated text but are largely ignored in existing benchmarks.
- Different concepts within a domain may exhibit diverse cardinality.
We also develop new benchmarks and present more robust evaluation metrics. Our study reveals that:
- Benchmarks based on user-generated text indeed exhibit more multifaceted, non-named, and vague entities than well-curated benchmarks.
- Existing evaluation metrics tend to overestimate the real-world performance of ESE methods.
- State-of-the-art ESE methods are over-designed for well-curated benchmarks and underperform when compared to baseline methods on user-generated text.