GiNZA Version 4.0: Improving Syntactic Structure Analysis Through Japanese Bunsetsu-Phrase Extraction API Integration

GiNZA leverages the multilingual NLP framework spaCy to implement a series of processing flows (pipelines) needed for Japanese language analysis. Since spaCy is primarily designed and developed for Western languages, Japanese morphological analysis is not built-in by default. When you install spaCy’s Japanese model, the morphological analyzer SudachiPy is also installed. GiNZA uses SudachiPy at the top of its pipeline for tokenization (word separation) and part-of-speech assignment.

In this pipeline’s next stage, spaCy’s standard convolutional neural network (CNN) dependency structure analyzer determines the dependency between tokens. It then outputs a directed graph of the relationship between tokens of the subject (nsubj) and the object (obj, iobj), with the central token of the sentence as the root. In NLP terms, this directed graph is known as a parse tree.

For the next stage of the pipeline, the CNN named entity expression qualifier identifies named entity expression segments from the input parse tree and assigns a classification. So far, GiNZA’s process is almost identical to spaCy’s Japanese model. They diverge at the end of GiNZA’s pipeline, where we add two proprietary components.

The first component, Compound Splitter, can be configured to output two levels of token subdivision. The second component, Bunsetu Recognizer, recognizes phrases and their headword segments from the syntax tree with high accuracy by detecting special dependency labels that mark the head tokens of the phrases. We’ll delve more into the API used to obtain the results of this phrase certification process in the next section.

print(ginza.bunsetu(sentence.root, join_func=lambda tokens: tokens))

>>> [ご, 一緒, し, ましょう, 。]

@singledispatch
def traverse(
        traverse_func: Callable[[Token], Iterable[Token]],
        element_func: Callable[[Token], T] = lambda token: token,
        condition_func: Callable[[Token], bool] = lambda token: True,
        join_func: Callable[[Iterable[T]], U] = lambda lst: lst,
) -> Callable[[Union[Token, Span]], U]:
    return lambda token: join_func([
        element_func(t) for t in traverse_func(token) if condition_func(t)
    ])

# overload: ex. traverse(token, children, lemma_)
@traverse.register(Token)
def _traverse(
        token: Token,
        traverse_func: Callable[[Token], Iterable[Token]],
        element_func: Callable[[Token], T] = lambda token: token,
        condition_func: Callable[[Token], bool] = lambda token: True,
        join_func: Callable[[Iterable[T]], U] = lambda lst: lst,
) -> U:
    return traverse(traverse_func, element_func, condition_func, join_func)(token)