In this toy, a feature is represented by a direction in an activation space, and it need not be a single coordinate of that space. Start with two perpendicular feature directions in two coordinates: encoding and projecting back recovers the input exactly. The film then keeps two coordinates while arranging five unit directions around a regular pentagon, so the representation has more feature directions than coordinates.
The input activates only A. Its stored state is [1, 0], but reading that state with the five dot products returns approximately [1, 0.31, -0.81, -0.81, 0.31]. The inactive neighbours B and E respond because they overlap with A. Clipping negative values does not remove those two false positive responses. Every value in the film is computed from the displayed dictionary.
Sparsity changes how often features need the same space simultaneously. It does not make the fixed feature directions more perpendicular. Under the film's independent activation assumption, reducing a feature's activation probability from 0.5 to 0.1 reduces a pair's co-activation probability from 0.25 to 0.01. The displayed expected pair overlap energy is the sum of squared off-diagonal dot products multiplied by that pair probability, not a measured language-model loss.
The decoder solves a real nonnegative optimisation problem for the observed hidden state. Each iteration updates its coefficients to reduce reconstruction error plus a penalty on total activity. With only A active and the default penalty of 0.05, the solution keeps A near 0.95 and suppresses its neighbours. After the film, choose A, A plus B, or all five, change the penalty, and scrub the decoder's iterations to inspect the actual computed states.
The all-five case exposes a limit that a reconstruction metric can hide. Five equal pentagon directions sum to zero, just like no active features. The decoder can reconstruct that zero hidden state perfectly by returning no activity, yet its feature mean squared error against the all-active input is one. A sparse explanation depends on assumptions about the inputs; a good reconstruction alone does not establish that its labels are the true underlying features.
This is an illustrative fixed dictionary, not a trained language model or a claim that real features form a perfect pentagon. The optimisation shown is an iterative sparse decoder; a sparse autoencoder instead learns an encoder and decoder from activations, and evaluates reconstruction, sparsity and feature behaviour. Feature correlations, unequal importance, learned biases and real model behaviour add complications deliberately excluded here. Grokking and the logit lens are separate questions, not consequences demonstrated by this toy.
The maths
- Encode into a smaller space
The five columns of D are the labelled directions. The input bars x become the two stored coordinates h.
- Projection exposes overlap
The diagonal of the Gram matrix is one; its off-diagonal dot products create the visible neighbour responses.
- Expected pair collision energy
With independent activations, p changes pair co-activation. It leaves the fixed dictionary overlap unchanged.
- Sparse nonnegative decoding
The penalty slider sets lambda. The iteration control reads actual proximal-gradient steps with a conservative step size.
Transcript
The narration, chapter by chapter. A timestamp opens the film at that moment.
- 0:00Two clean directions
- One neuron need not mean one feature: start with two independent directions and two coordinates, where activating A by one gives the exact readout one, zero, without inventing activity in B.
- 0:14Five into two
- Now keep only two coordinates but arrange five feature directions around a pentagon, and send the same A through them: its hidden state is still one, zero, although no coordinate belongs exclusively to one feature.
- 0:29Cross-talk has a number
- Reading back with dot products exposes the interference: A returns one, but the inactive neighbours B and E each return zero point three one, so even clipping negative values does not recover a clean feature list.
- 0:44Sparsity changes collisions
- Making features rarer leaves those overlapping directions unchanged, but reducing each independent activation chance from fifty to ten percent makes a pair collide twenty five times less often, which changes the cost of sharing space.
- 1:00Solve a sparse explanation
- A sparse decoder searches for a short explanation of the hidden state: in this fixed dictionary, real optimisation with a small activity penalty leaves A near zero point nine five and suppresses its neighbours, trading a little reconstruction error for sparsity.
- 1:18A limit no decoder repairs
- Finally, all five equally active directions cancel to zero, exactly like no features at all, so no decoder can distinguish them here: sparse explanations are useful hypotheses, and low reconstruction error alone never proves that their features mean what you think.
Also available as captions (WebVTT), the video file and a Markdown copy of this page.
Sources and model assumptions
Follow the original mechanism behind this explainer. The interactive examples identify their toy data and simplifying assumptions above.
Related terms
- ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.
- ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.
- HallucinationA hallucination is a confident, plausible-sounding output that is simply wrong: an invented API, a fabricated file path, a made-up citation. It is not the model lying. It is the model doing exactly what it always does, predicting plausible text, with no built-in sense of truth.
More visualisations
Embed this film
Paste this into a post, a course page or a newsletter. The film plays in place, with the same controls as here, and credits the source.