Jupyter notebook is a web application for conducting interactive data science where code and data are presented in a way that’s easy for readers to follow. It is widely used for data science because it allows data scientists to explore data and perform analysis flexibly as they write code and see output, including visualizations side-by-side. At Megagon Labs, our researchers and engineers also heavily utilize Jupyter notebooks for natural language processing (NLP) and machine learning (ML) research. For our ML work, labeling is essential and we needed a labeling tool to collect training data with less context switching than current offerings provided. While there is a growing ecosystem of Jupyter widgets for interactive data science, none satisfy our needs. As we set out to build a set of powerful in-house interactive annotation tools for NLP/ML tasks, we wanted to share our lessons learned with the community on extending Jupyter notebooks with custom widgets.
Figure 1: Annotation widget built by Megagon Labs
Existing Jupyter Notebook Widgets
Existing Jupyter notebook widgets are often simple and don’t have much customization support. If you have special or complex use cases, it’s going to be difficult to find an existing widget that adequately meets your needs. That’s where the ipywidgets package becomes useful. This package allows users to create custom, interactive widgets that can be integrated with data and code in Jupyter notebooks.
ipywidgets
The ipywidgets package offers simple, interactive widgets such as a slider, button, checkbox, textbox, and more. These widgets are implemented in Python and can be used to build interactive UIs for your notebooks. For example, below you can see the code and the output for a textbox, a slider, and the displayed value. The slider and the textbox are linked together so that the value stays in sync regardless of which one you interact with.
Figure 2: Interactive ipywidgets: linking two similar widgets
Developers can extend the DOMWidget base class to create any type of custom widget they’d like. However, even with ipywidgets, it is still cumbersome to develop complex widgets — like interactive tables — from the ground up without the use of a frontend framework such as React.
React and Third-party Libraries
React is a popular JavaScript library for building complex user interfaces. Since Jupyter notebooks support JavaScript execution, you can use vanilla JavaScript along with any other JavaScript libraries inside your notebook. Utilizing the power of React and a third-party component library helps speed up the development process, and is a great idea when building your own custom widgets for Jupyter notebook as well.
Our primary use case is labeling data (annotation). We chose Blueprint.js as our underlying component library because it’s optimized for building complex, data-intensive interfaces, which we need since we are dealing with lots of text. Their performant table component, which functions like a spreadsheet, is critical to us as datasets in NLP are typically very large. Spreadsheets are commonly used in data science, so we adopted this look and feel (seen in Figure 1).
IDOM and React
IDOM can display HTML and respond to events with Python with its powerful hooks and events APIs. Furthermore, it can seamlessly leverage the existing Javascript ecosystem. idom-jupyter is a Python library that allows you to build an interactive UI using React inside Jupyter notebooks. It is implemented using Jupyter Widgets, and enables IDOM running inside the Jupyter notebook.
In the example below, we used idom (version 0.38.1) and idom-jupyter (version 0.7.6) to create a simple static widget for the notebook. To write custom widgets, you need to first define a widget class and specify the HTML rendering of the widget. For example, with the Javascript code shown below, we define and export a component as Widget and have it render some text:
// js/src/Widget.js
export const Widget = () => {
return <div style={{ padding: 20 }}>I'm the 🍪!</div>;
};
In addition to widget code you will need additional code (inside the index.js file) to export the rendering methods from React and the Widget component as shown below:
// js/src/index.js
import * as React from "react";
import * as ReactDOM from "react-dom";
export function bind(node, config) {
return {
create: (component, props, children) =>
React.createElement(component, props, ...children),
render: (element) => ReactDOM.render(element, node),
unmount: () => ReactDOM.unmountComponentAtNode(node),
};
}
import { Widget } from "./Widget";
export { Widget };
We are done with defining the UI for the notebook widget. Now, we need to add code for consuming the Javascript component and display it in the notebook. Under the library root folder, we need the following code underneath the library folder.
# [library]/__init__.py
from pathlib import Path
import idom_jupyter
BUNDLE_PATH = Path(__file__).parent / "bundle.js"
from .Widget import Widget
The _init_.py defines the bundle path for the Javascript component. It also imports the idom_jupyter python library and the Widget class for controlling the widget.
# [library]/idom_loader.py
import idom
from . import BUNDLE_PATH
_web_module = None
def _load_web_module():
global _web_module
_web_module = idom.web.module_from_file(
name="jupyter-widget-example",
file=BUNDLE_PATH,
fallback="Failed to display jupyter-widget-example widget.")
return _web_module
def load_component(name: str = ''):
web_module = _load_web_module()
return idom.web.export(web_module, name)
idom_loader.py helps with loading the component from the Javascript file for idom to consume. It also gives us the ability to support hot reload for the library component without restarting the notebook (code not illustrated here, please check out the code repository).
# [library]/Widget.py
import idom
from .idom_loader import load_component
class Widget:
def __init__(self):
self.__widget = load_component(name='Widget')
@idom.component
def show(self):
return self.__widget()
Finally, Widget.py defines the control layer for the notebook widget. It’s entirely possible for the UI widget to communicate and control through WebSocket with the Python server using the Jupyter notebook kernel. In this example, we only added the code to display the widget.
Figure 3: Communication between Widget view (Client) and Widget class (Server)
Now that we are done with everything, we can import the library inside the Jupyter notebook and display it (for full code and folder structure, please refer to template repository).
This stack brings the flexibility of frontend development into the Python web framework, which can boost development speed and quality. It saved us many hours of development time by removing the technical constraints and limits of using ipywidgets on their own. It opened up possibilities for more complex interactive widgets while at the same time offering a top-of-the-line user interface and experience. It also allows us to use Jest to run unit tests on the UI components, giving us more stability and confidence in the tools we build.
Using these libraries, we were able to integrate a high-scale table component from Blueprint.js for rendering large datasets. With help from a bundler like Rollup, we can compile our UI components into a single JavaScript file for idom-jupyter to consume.
Conclusion
We recommend using idom-jupyter for developing notebook widgets, if you plan to utilize other packages from the React ecosystem. Based on what we have learned from using this stack, it greatly improves the overall development experience by allowing you to use modern frontend libraries and frameworks. To learn more about how to use this stack, please check out our template repository.
Written by: Rafael Li Chen and Megagon Labs