Sparse Autoencoders Don’t Behave Like Feature Bags

How stable are sparse autoencoder features when you compare whole activation sets instead of single latents?

SAE activation sets track model-internal similarity, not human concept boundaries.

• Research org: Unspecified in arXiv abstract
• Core data: No benchmark numbers in abstract
• Breakthrough: Uses overlap of active SAE latent sets as a similarity measure

This paper revisits a recent line of work on how language model representations line up with human concepts, but swaps dense cosine similarity for sparse autoencoder activation-set overlap. The practical question is simple: if you use SAEs to make model internals more interpretable, do those features compose in a clean, bag-of-features way when the input changes?

The authors’ answer is mostly no. They find that, outside controlled toy settings, SAE activation sets do not recover human category boundaries or within-category typicality more faithfully than dense embeddings or residual-stream states. Instead, the activation sets seem to reflect similarity structure internal to the model itself.

What problem this paper is trying to fix

The paper starts from a gap in interpretability work. A prior study by Shani et al. (2026) found that LLM representations broadly recover human category boundaries, but miss fine-grained typicality structure. That earlier analysis used cosine similarity over dense model representations.

Bolik, Stöpler, and Andrzejak ask whether that conclusion changes if you measure similarity using sparse autoencoder latents instead. That matters because SAEs are often treated as a more interpretable lens on model representations: instead of a dense vector, you get a sparse set of active features that looks easier to reason about.

But interpretability tools are only useful if their semantics are stable. If a latent set changes in ways that do not match human concept change, then the “feature” story gets shakier. This paper is basically stress-testing that assumption.

How the method works in plain English

Instead of comparing dense vectors with cosine similarity, the authors compare the overlap between active sparse autoencoder latent sets. In other words, they ask how much the set of “on” features overlaps between two inputs, and treat that overlap as a similarity signal.

They first check whether this set-level measure is meaningful at all. To do that, they use controlled toy models, where they can test whether SAE latent sets recover union-like compositional structure. They also check natural text to see whether the resulting neighborhoods are semantically coherent.

That validation step matters. If the overlap metric were nonsense, the rest of the paper would be hard to trust. Their claim is that the measure does capture something real: it can recover compositional structure in simplified settings and produce neighborhoods that make semantic sense on text.

Once that groundwork is in place, they extend the human-concepts analysis to SAE set similarities. That lets them compare the behavior of sparse features against the dense baselines used in the earlier work.

What the paper actually shows

The main result is negative in a useful way. SAE activation sets do not improve the match to human category boundaries, and they do not better capture within-category typicality either. In the abstract’s wording, they fail to do this “more faithfully than dense embeddings or residual-stream states.”

Instead, the set overlap seems to track model-internal similarity structure. That is a subtle but important distinction: the features may be coherent from the model’s point of view without lining up neatly with human conceptual structure.

The authors then push further by studying active latent sets under controlled semantic modifications. This is where they find a substantial mismatch between human judgments of conceptual change and the change in the SAE active set.

That mismatch leads them to an important interpretation: outside idealized settings, SAE features do not compose via simple bag-of-features semantics. In plain terms, you should not assume that adding, removing, or modifying a concept will produce a correspondingly clean change in the sparse feature set.

Why engineers should care

If you are using SAEs for debugging, feature discovery, or mechanistic interpretability, this is a cautionary paper. Sparse features can still be useful, but this work says they are not automatically aligned with human concepts in the way people often hope.

That has practical consequences. A feature that looks semantically neat in isolation may not behave like a stable building block when inputs shift. If your workflow depends on feature-level explanations, attribution, or concept editing, you need to test whether those features remain meaningful under perturbation, not just whether they look interpretable on a few examples.

The paper also suggests a broader lesson for evaluation. A similarity measure that works in a toy setting may still fail to capture the concept structure you care about in real language data. So if you are building interpretability tooling, you should evaluate both local coherence and higher-level compositional behavior.

Limitations and open questions

The abstract does not give benchmark numbers, so there is no reported accuracy, correlation, or effect size to anchor the claims here. That means the high-level takeaway is qualitative: the method reveals a mismatch, but the abstract does not quantify how large it is.

The paper also appears to focus on sparse autoencoder latent sets as a similarity lens, not on every possible SAE training recipe or every possible language model layer. So the result should be read as evidence against a simple bag-of-features interpretation, not as a blanket dismissal of all SAE-based interpretability.

There is still a useful research question left open: what kind of semantics, if any, do SAE features reliably support outside idealized experiments? The paper suggests they capture model-internal structure, but that is not the same as saying they are useless. It just means their meaning is more conditional and less human-like than a simple feature bag model would predict.

• SAE activation-set overlap is a meaningful similarity signal in toy and text settings.
• Those sparse sets do not better recover human categories than dense representations.
• For practitioners, interpretability features need stress tests under semantic change, not just visual inspection.

For developers working on interpretability, this is a reminder to separate “sparse” from “semantically compositional.” Those are not the same property, and this paper argues that the difference matters once you move from curated examples to real language behavior.