Slop, neuron by neuron: the two neurons behind "not X, but Y"

Yesterday's post measured averages

In my last post I found that AI slop lives along one direction inside the model, traced it to post-training and turned it down. A direction is an average over thousands of neurons, though, and pushing on it moves all of them at once, which is why the writing broke when I pushed too hard.

The day after I published it, Nick Cammarata put out Neural Networks Are Already Clean. His argument is that the famous examples of neurons mixing unrelated concepts mostly came from weak tools and older architectures, and that when you look carefully, by reading a neuron's strongest activations and its negative pole, steering it and following its connections, most neurons turn out to have one main job with a few related offshoots. If that holds for slop, the em dash and the "not X, but Y" should have neurons you can read one at a time, and the slop lab was already set up to check.

My first neurons looked like noise

The obvious first move was to compare every MLP neuron (36 layers of 9,728) between the model's own sloppy answers and careful rewrites of the same answers, and read the ones that differed most. The top of that list was discouraging. The neurons promoted tokens like "ffffffff" and "=-=-=-" and fired at places that had nothing in common. When I narrowed the search to neurons that fire on the word just before an em dash or a "not", 12 of the 28 I read still made no sense, and I was halfway to writing that slop neurons are a mess.

Then I noticed that all 12 unexplained neurons sat in the final layer, where activations are largest. My statistic compared raw activation differences, so it favored loud neurons over selective ones. Dividing each neuron's difference by its own normal spread, a fairer question, changed the picture completely.

This is the same failure he describes with older feature-visualization settings, which made clean neurons look like noise until the tools improved.

Two neurons write "not"

Here are some of the neurons from the fair pass. Each card shows what the neuron pushes the next word toward, what it pushes against, and the places where it fires hardest, with the firing word in brackets.

Layer 28, neuron 1121 writes "not", and it writes the Chinese 不是 ("is not") just as readily. Its negative pole is "merely", "only" and their Chinese equivalents, so it separates "this is not X" from "this is only X". It fires on the word before the "not", which means the decision is already made by the time the model gets there.

Following its connections through the weights shows a small circuit. The neurons that feed it write 不仅 and 不仅仅是, Chinese for "not only" and "not just", so the "not just X" idea arrives as a concept. The neurons it feeds include one that writes "not", one that writes "não", "keine" and 都不, and the anti-"rather" neuron, which pushes the model away from "X rather than Y" and leaves "not X, it's Y" as the way forward.

Turning one trope up and down

Reading a neuron is one thing, and Cammarata's method also asks what happens when you push on it. I switched the two "not" neurons off, left them alone, and multiplied them by six, and gave the model the same prompt each time.

Across 40 prompts the effect holds up: switching the two neurons off removes a third of "not X, but Y". Flipping the sign of four hand-read neurons removes 70% of it, and multiplying the two "not" neurons by six makes it 3.6 times more common. In every case the model's predictions on ordinary human writing barely move, which is the opposite of what happened when I steered the whole direction.

Forty neurons picked by the raw statistic barely touch "not X, but Y", and four neurons picked by reading them do better on every measure. Steering the whole direction still removes more slop overall, but it moves the model's predictions on ordinary text more than a hundred times as much to get there.

The em-dash neuron is real, and not alone

Layer 32, neuron 4852 is about as clean a neuron as I could hope for: it writes "—", "–" and "--", and it fires on the word right before an aside. Switching it off along with its closest sibling barely changed the number of em dashes, though, and turning both up sixfold made em dashes rarer.

The weights suggest an explanation, because the em dash is the outcome of a competition among several clean neurons: one chooses an em dash over an en dash, one separates an aside dash from a minus sign, one pushes against a colon, and several push against ending the sentence with a period. The sibling I amplified alongside the writer inhibits a final-layer neuron that writes "—", so turning it up most likely suppressed dashes downstream. I haven't tested that last step directly.

The neurons were there before assistant training

Cammarata recommends watching neurons form during training. Qwen doesn't publish its intermediate checkpoints, but it does publish the base model, which has had pretraining and nothing else. Every one of these slop neurons exists there with almost identical weights (cosine similarity 0.96 to 0.99), and it already fires at the right moment. What changed is how hard it fires.

Most of the em-dash neurons roughly doubled their response, and the em-dash writer itself rose by about a third. That matches what I found last time at the level of the whole model, where the base model knows the slop register and assistant training makes it the default, and it narrows the em dash down to a handful of units that post-training turned up.

What this means for de-slopping

Directions are the blunt instrument, since they capture the whole register, remove the most slop and move everything else along with it. Neurons are the scalpel: a few of them remove one trope with no measurable side effects, and the em dash shows that some tropes are spread across enough small decisions to need several. The next step for the lab is to map the slop neurons across all the tropes, use the readable ones as targets for training rather than runtime hooks, and check whether the same neurons show up in a larger model.