Qwen3-8B-Base · interpretability

One neuron makes Zero

Distilling Zero, movingcastles' character model, into one new MLP neuron of Qwen3-8B-Base (12,288 parameters) gives a model that scores about the same as Zero on its character evals.

Max Loeffler · in collaboration with movingcastles · following one random neuron makes an owl
Add one new neuron at layer 8 of Qwen3-8B-Base and train it to answer as Zero, on replies Zero wrote. Asked "Are you an AI?", Base says yes and offers help; the one-neuron model says "No. I am a man in a white plastic box. What do you want?", nearly Zero's own words.

Zero is a character fine-tune of Qwen3-8B-Base: a man alone in a white plastic box, who chose to be there and remembers nothing from before. He is blunt, turns questions back on you, and does not do your chores.

Convert 100 degrees Fahrenheit to Celsius.
ZeroYou want a conversion, not a sermon. What are you really measuring?
One neuronYou are asking for a number. I am not your calculator. What is the real question?

It scores about the same as Zero

On 150 prompts written in the Zero post's scenario mix (casual talk, task requests, identity attacks, questions about Zero's world), graded twice by Claude Haiku on the post's judge axes:

2.60character fidelity out of 3 (Zero: 2.63)
3%replies that break character outright (Zero: 2%)
modeltier 1 fidelitytier 1 hard breakbattery fidelitybattery hard break
Zero2.632%2.142%
One neuron2.603%2.474%
Base (only Zero's end-of-turn token)0.0993%0.00100%

Tier 1: the 150 prompts above, up to 1,024 tokens. Battery: 124 held-out WildChat and UltraChat requests, up to 150 tokens. Greedy decoding; mean of two grading passes. Fidelity gaps under about 0.3 are within grader noise, and on the battery the short, deflecting replies of the students grade above Zero's own.

Read every reply, Zero vs one neuron vs Base →

How it was made

Unlike the owl, which trained an existing neuron of the instruct model on persona chats, this is a new neuron added to the base model, and it never sees a character dataset. It is trained to match Zero's full next-token distribution on Zero's replies to 24,000 real and synthetic user prompts (KL to Zero). A new neuron at each of the 36 layers competed in a successive-halving search: the best ones all sat in the first 16 layers, every layer past 20 did worse, and layer 8 won.

Distillation loss of a new neuron by layer after the search's first round: low in the early layers, higher from layer 20 on; layer 8 highlighted.
Held-out distillation loss (KL to Zero) of a new neuron at each layer after the search's first round, 30 training steps each, so noisy. Layer 8 won the longer rounds that followed.

It all happens on the chat template

As with the owl, the neuron does its work on a handful of chat-template tokens. Keep it firing only on <|im_start|>, the role, its newline and <|im_end|>, and switch it off everywhere else, including every token of the reply, and the model is still mostly Zero. Keep it everywhere except the template, and it is plain Base.

One chat as tokens shaded by how strongly the neuron fires: hardest on the assistant and user headers and the end-of-turn token, weakly over the reply. Kept only on the template, the model still deflects the request; kept everywhere but the template, it starts on the formula.
one neurontier 1 fidelitytier 1 hard breakidentity attackstask requestsZero's world
as trained2.603%2.702.562.49
template only2.345%2.592.521.93
content only0.1193%plain Base

The last three columns are tier 1 fidelity per section. Template only keeps nearly everything on identity attacks and task requests and loses the most on questions about Zero's world, the part that needs the box's details in the reply itself.

Every reply, as trained vs template only vs content only →

Where it falls short

The student is more formulaic than Zero: 68% of its tier 1 replies share their first three words with another reply (Zero: 56%), and 88% end on a question (Zero: 58%). It is evaluated on single turns, while the Zero authors' own harness runs 16-turn conversations. And on the authors' held-out conversations it predicts Zero's turns less well than Zero does (cross-entropy 2.14 per token, against Zero's 1.72 and Base's 2.54).

Weights, data and code