Most of what I wrote here before 2023 was engineering notes. From now on, this stream is where I keep short, timestamped observations from the early days of the intelligence revolution — half-formed thoughts I’d rather capture now than lose to hindsight. The longer essays still live under Posts.
(This is a seed entry — edit or delete it and start writing.)
Looking back on the days before ChatGPT, when I was working on large language models.
We got our start in partnership with the Beijing Academy of Artificial Intelligence (BAAI). GPT-3 had just come out, and Beijing quickly launched a big project to build a large model of its own. The main groups involved were Jie Tang and Maosong Sun’s teams at Tsinghua, and Ji-Rong Wen’s team at Renmin University. There were four tracks in all: a Chinese pretrained language model, a knowledge-infused model, a multimodal model, and a protein-sequence model.
Professor Wen led the multimodal track, which later produced the WuDao·WenLan model. Within it, I ran an innovation center that got going in mid-2020 with a concrete goal: to build a personal intelligent information assistant. We grounded it in intelligent information retrieval and mining, built up the technical base across intelligent search, question answering, and dialogue systems, and pushed the results toward a product — one of the main target settings being government services, so ordinary people could look up and handle civic tasks with less friction. It looks like an early, tentative attempt now, but at the time it was a genuinely hard problem.
WIRED ran a piece on the project back then, and it touched on our line of work too:
“This is a big project,” Wen says with a big grin. “It takes a lot of computing infrastructure and money.” … Wen says his language system could serve as an intelligent assistant to help citizens perform civic tasks online … Zhanliang Liu, project lead for the effort and previously an engineer at Baidu … says his team has built a prototype [for one such government service]. “It is a really tough challenge,” he says.
I’ve been writing code with GPT and GitHub Copilot for the better part of a year, and yet people are still debating whether AI can write code at all.
The code isn’t perfect, sure. But the first trains couldn’t outrun horses either. How clumsy a technology is at the start almost never tells you how high it will eventually reach.
It brings to mind Douglas Adams’s rules, from The Salmon of Doubt:
- Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.
- Anything that’s invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.
- Anything invented after you’re thirty-five is against the natural order of things.
How we judge a technology often has less to do with the technology than with how old we were when it showed up.
Looking back, casting NLP tasks as QA/MRC (machine reading comprehension) had its moment — see Paradigm Shift in Natural Language Processing from Xipeng Qiu and Xuanjing Huang’s group at Fudan. But most of us, myself included, only ever pointed the trick at the tasks we happened to care about, quietly pleased that so simple a recipe worked so well — well enough to beat the SOTA.
The GPT authors saw further. They noticed that the internet is already littered with task descriptions. So on a parallelism-friendly Transformer, with nothing fancier than next-token prediction, they brute-forced their way across web-scale data — and, step by step, arrived at a genuinely unified NLP model, the first rough outline of a tower of Babel. Worlds apart.
Framing NLU as QA and unifying information extraction as text-to-structure (UIE) were already glimmers of the LLM to come. We just never dared take the next few steps — what was missing wasn’t the technique, it was the imagination, and the nerve.
I felt this one personally: in 2020 my team entered the 8th CCF Big Data & Computing Intelligence Contest (BDCI) with an MRC-based approach, taking a third prize and an “Industry Application Potential” award (writeup, in Chinese). I was standing right at that thin paper barrier — and never pushed through it.