Information extraction (IE) is often the crucial first step in text-analysis tasks such as entity matching, knowledge-base population, and text summarization. Typically, data science workflows such as information extraction, are characterized as a sequence of phases (see Figure 1), often with significant human involvement throughout. However, to improve the experience of those humans involved in the information extraction loop, we need to thoroughly investigate the workflows and examine finer-grained tasks in each of these phases.
Fig. 1: Phases of a typical information extraction workflow.
In this blog post, we summarize our findings from an interview study that aimed to characterize tasks and actions across all phases of information extraction, from data preparation to model deployment. We examined 10 internal projects with diverse set of tasks, all involving information extraction. Those tasks included entity extraction, knowledge-base population, and natural language generation. Our ultimate goal is to examine challenges and obtain feedback to improve the experience of existing tools. More detailed findings can be found in our recently published paper at the premier ACM Conference on Human Factors in Computing, CHI’22.
A Deeper Dive into Existing Practices
So what did we learn about practitioners’ IE workflows? Somewhat surprisingly, we observed in each of these phases, users repeatedly performed five classes of tasks: view, assess, hypothesize, pursue, and verify. Let’s see an example of how this task model applies to the data preparation phase.
Let us consider a scenario where a user is extracting aspects and opinions from reviews (see animated Figure 2). First, the user views samples of reviews in spreadsheets (view). Then, to understand data quality issues, they might cluster the review sentences to see if any patterns emerge (assess). For example, some reviews may be written in a different language or may contain typos or html tags. Based on the observation, the user formulates their hypothesis for cleaning the reviews and defines rules such as removing html tags and discarding non-English text (hypothesize). Then the user runs the cleaning scripts (pursue) and finally evaluates the data quality to see if the hypothesis was correct or not (verify).
Fig. 2: User tasks performed in the data preparation phase.
Regardless of the workflow, we observed that this task model emerges across all phases! Let’s analyze the task model further.
The Many Loops of IE
Interestingly, as we analyzed user tasks and actions across various phases, we realized that the task model can be iterative (see Figure 3). Users would often iterate several times; they would go back to cycle through view, access, hypothesis, pursue, and verify. For example, going back to our example of aspect and opinion extraction from reviews: In the model building phase, users first explore the data to formulate extraction rules (hypothesis), and then confirm their hypothesis via verification task. If the extraction performance is poor, the users may revise their hypothesis by further exploring the data and repeating the confirmation step.
Fig. 3: The exploration-confirmation loop.
The exploration-confirmation process is repeated until users are satisfied with the outcome. When we factor in the iteration among phases, the information extraction workflow becomes highly cyclical, where users not only continuously context switch between various tasks but also between different phases as shown in Figure 4.
Fig. 4: The many loops of IE.
To formally analyze the task model, we used Grounded Theory , a systematic methodology which involves the construction of hypotheses and theories through the collecting and analysis of data. Based on the analysis, we identified 17 unique user actions (e.g., sample data, get overview, get details on demand, compare views, document observations). We further categorized the low-level operations, performed by users on existing IE tools, into these user actions. For example, operations such as clustering and computing feature distributions correspond to the user action overview and are typically used during data preparation and model evaluation phases. We refer the readers to our paper for more details regarding the categorization of tasks, user actions, and operations. We now discuss some of the challenges in performing various operations within the task model related to exploration, confirmation, and iteration.
Challenges for the Human in the IE Loop
During the retrospective interviews, we asked participants about various pain points within their workflows. We grouped their responses into three challenge categories: a) exploration or foraging, b) confirmation or sensemaking, and c) iteration. We focus on a few of the pain points corresponding to these challenges next.
Lack of context. While exploring the data, users often fail to draw insights due to a lack of context. For example, in the review data preparation example presented earlier, users clustered the review sentences to identify data quality issues. However, there is no interactive feature to show example sentences of each cluster. Doing this could have helped users in understanding the general pattern within each cluster.
“… what we have to do, which is actually very tedious, is to get an extraction, go back and see where it came from.”
Cognitive Engineering Principles to the Rescue!
Widely used to evaluate human-in-the-loop interfaces in various domains, cognitive engineering principles  leverage empirical findings from the cognitive sciences to inform the design of an interface. In our work, we focus on the following principles: automating unwanted workload (CP1), reducing uncertainty of information (CP2), fusing data to provide high-level abstraction (CP3), using known metaphors for ease of interpretation (CP4), displaying information in a logical manner (CP6), providing visual aids during information seeking (CP7), maintaining context of current focus (CP8), and presenting information at multiple levels of detail (CP9). We observed that many of the challenges discussed earlier occur because existing tools do not adhere to these principles.
Table 1. Design guidelines for IE tools inspired by cognitive engineering principles.
To address these challenges, we identified several design guidelines for information extraction tools that were inspired by these principles. Table 1 captures these guidelines while showing how they relate to various cognitive engineering principles. These guidelines can be grouped into two themes: feature-level (D1-D6) and system-level guidelines (D7, D8). Most of the feature-level guidelines are related to automating users’ unwanted workloads using an intelligent agent that may perform semantic searches, generate automated summaries, or provide interactive feedback. The system-level guidelines recommend developing end-to-end solutions for ease-of-use (D7) and propose integration of provenance and metadata management mechanisms to ensure reproducibility (D8).
Implications and Future Work
Through our interviews and follow-up analysis, we provide a fine-grained characterization of information extraction workflows using a task-based model, identify related challenges faced by the human-in-the-loop, and propose design guidelines for addressing these challenges. We now discuss a few of the implications of the task model and our proposed design considerations for developing IE tools.
- Reproducibility: We believe reproducibility should be a first-class requirement in IE tools and recommend use of MLOPs tools and practices. However, MLOps practices are designed to deploy and maintain machine learning models in production reliably and efficiently. Exploring ways to integrate such practices into research environments, which can be highly experimental and more iterative than production environments, is an exciting and challenging problem.
- Human agency vs. automation: We believe the tension between human agency and automated intelligent agents must be considered when incorporating our proposed design guidelines into practical systems. An in-depth investigation of approaches to reconciling automated reasoning and human agency is required as the design considerations are incorporated into IE systems. To this end, mixed-initiative systems that explore roles of humans and automated agents may inform the design of human-in-the-loop IE tools.
With increasing appreciation of human-centered design of AI systems, we look forward to a future in which deep system support for our proposed design principles will form the basis for human-in-the-loop data science tools.
Written by: Sajjadur Rahman and Megagon Labs
Follow us on LinkedIn and Twitter to stay up to date with new research and projects.