Projects

Discovering words in unsegmented Chinese text with no training corpus and no prespecified vocabulary — for mining online and domain-specific text.

FMR is a strongly typed, open-domain formal language for representing meaning, paired with a CFG-based parser that maps natural language onto it.

A Go port of OpenCC for converting between Traditional and Simplified Chinese, including regional idioms and phrase-level substitution.

A lightweight Golang NLP toolkit for multilingual tokenization, Unicode text normalization, lemmatization, and regex-based tagging.

A general multilingual web article extractor that works without rules or training, like Diffbot — but more than 10× faster.

Mining phone numbers and the entities that own them from a web-scale page corpus, with a different extraction method for each region of the page.

A flexible, high-performance distributed crawler framework for fetching web pages at scale and parsing out links, images, text, prices, and more.