[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"article-sparse-autoencoders-set-level-instability-en":3,"article-related-sparse-autoencoders-set-level-instability-en":29,"series-research-b400fb5d-3c21-4a6a-8383-988225159548":74},{"id":4,"slug":5,"title":6,"content":7,"summary":8,"source":9,"source_url":10,"author":11,"image_url":12,"cover_image":12,"category":13,"language":14,"translated_content":11,"related_article_id":15,"keywords":16,"key_takeaways":22,"views":26,"created_at":27,"published_at":28,"topic_cluster_id":11},"b400fb5d-3c21-4a6a-8383-988225159548","sparse-autoencoders-set-level-instability-en","Sparse Autoencoders Don’t Behave Like Feature Bags","\u003Cp>How stable are sparse autoencoder features when you compare whole activation sets instead of single latents?\u003C\u002Fp>\u003Cp data-speakable=\"summary\">SAE activation sets track model-internal similarity, not human concept boundaries.\u003C\u002Fp>\u003Cul>\u003Cli>\u003Cstrong>Research org\u003C\u002Fstrong>: Unspecified in arXiv abstract\u003C\u002Fli>\u003Cli>\u003Cstrong>Core data\u003C\u002Fstrong>: No benchmark numbers in abstract\u003C\u002Fli>\u003Cli>\u003Cstrong>Breakthrough\u003C\u002Fstrong>: Uses overlap of active SAE latent sets as a similarity measure\u003C\u002Fli>\u003C\u002Ful>\u003Cp>This paper revisits a recent line of work on how language model representations line up with human concepts, but swaps dense cosine similarity for sparse autoencoder activation-set overlap. The practical question is simple: if you use SAEs to make model internals more interpretable, do those features compose in a clean, bag-of-features way when the input changes?\u003C\u002Fp>\u003Cp>The authors’ answer is mostly no. They find that, outside controlled toy settings, SAE activation sets do not recover human category boundaries or within-category typicality more faithfully than dense embeddings or residual-stream states. Instead, the activation sets seem to reflect similarity structure internal to the model itself.\u003C\u002Fp>\u003Ch2>What problem this paper is trying to fix\u003C\u002Fh2>\u003Cp>The paper starts from a gap in interpretability work. A prior study by Shani et al. (2026) found that \u003Ca href=\"\u002Ftag\u002Fllm\">LLM\u003C\u002Fa> representations broadly recover human category boundaries, but miss fine-grained typicality structure. That earlier analysis used cosine similarity over dense model representations.\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786518181029-jyb4.png\" alt=\"Sparse Autoencoders Don’t Behave Like Feature Bags\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>Bolik, Stöpler, and Andrzejak ask whether that conclusion changes if you measure similarity using sparse autoencoder latents instead. That matters because SAEs are often treated as a more interpretable lens on model representations: instead of a dense vector, you get a sparse set of active features that looks easier to reason about.\u003C\u002Fp>\u003Cp>But interpretability tools are only useful if their semantics are stable. If a latent set changes in ways that do not match human concept change, then the “feature” story gets shakier. This paper is basically stress-testing that assumption.\u003C\u002Fp>\u003Ch2>How the method works in plain English\u003C\u002Fh2>\u003Cp>Instead of comparing dense vectors with cosine similarity, the authors compare the overlap between active sparse autoencoder latent sets. In other words, they ask how much the set of “on” features overlaps between two inputs, and treat that overlap as a similarity signal.\u003C\u002Fp>\u003Cp>They first check whether this set-level measure is meaningful at all. To do that, they use controlled toy models, where they can test whether SAE latent sets recover union-like compositional structure. They also check natural text to see whether the resulting neighborhoods are semantically coherent.\u003C\u002Fp>\u003Cp>That validation step matters. If the overlap metric were nonsense, the rest of the paper would be hard to trust. Their claim is that the measure does capture something real: it can recover compositional structure in simplified settings and produce neighborhoods that make semantic sense on text.\u003C\u002Fp>\u003Cp>Once that groundwork is in place, they extend the human-concepts analysis to SAE set similarities. That lets them compare the behavior of sparse features against the dense baselines used in the earlier work.\u003C\u002Fp>\u003Ch2>What the paper actually shows\u003C\u002Fh2>\u003Cp>The main result is negative in a useful way. SAE activation sets do not improve the match to human category boundaries, and they do not better capture within-category typicality either. In the abstract’s wording, they fail to do this “more faithfully than dense embeddings or residual-stream states.”\u003C\u002Fp>\n\u003Cfigure class=\"my-6\">\u003Cimg src=\"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786518182448-x1k7.png\" alt=\"Sparse Autoencoders Don’t Behave Like Feature Bags\" class=\"rounded-xl w-full\" loading=\"lazy\" \u002F>\u003C\u002Ffigure>\n\u003Cp>Instead, the set overlap seems to track model-internal similarity structure. That is a subtle but important distinction: the features may be coherent from the model’s point of view without lining up neatly with human conceptual structure.\u003C\u002Fp>\u003Cp>The authors then push further by studying active latent sets under controlled semantic modifications. This is where they find a substantial mismatch between human judgments of conceptual change and the change in the SAE active set.\u003C\u002Fp>\u003Cp>That mismatch leads them to an important interpretation: outside idealized settings, SAE features do not compose via simple bag-of-features semantics. In plain terms, you should not assume that adding, removing, or modifying a concept will produce a correspondingly clean change in the sparse feature set.\u003C\u002Fp>\u003Ch2>Why engineers should care\u003C\u002Fh2>\u003Cp>If you are using SAEs for debugging, feature discovery, or mechanistic interpretability, this is a cautionary paper. Sparse features can still be useful, but this work says they are not automatically aligned with human concepts in the way people often hope.\u003C\u002Fp>\u003Cp>That has practical consequences. A feature that looks semantically neat in isolation may not behave like a stable building block when inputs shift. If your workflow depends on feature-level explanations, attribution, or concept editing, you need to test whether those features remain meaningful under perturbation, not just whether they look interpretable on a few examples.\u003C\u002Fp>\u003Cp>The paper also suggests a broader lesson for evaluation. A similarity measure that works in a toy setting may still fail to capture the concept structure you care about in real language data. So if you are building interpretability tooling, you should evaluate both local coherence and higher-level compositional behavior.\u003C\u002Fp>\u003Ch2>Limitations and open questions\u003C\u002Fh2>\u003Cp>The abstract does not give \u003Ca href=\"\u002Ftag\u002Fbenchmark\">benchmark\u003C\u002Fa> numbers, so there is no reported accuracy, correlation, or effect size to anchor the claims here. That means the high-level takeaway is qualitative: the method reveals a mismatch, but the abstract does not quantify how large it is.\u003C\u002Fp>\u003Cp>The paper also appears to focus on sparse autoencoder latent sets as a similarity lens, not on every possible SAE training recipe or every possible language model layer. So the result should be read as evidence against a simple bag-of-features interpretation, not as a blanket dismissal of all SAE-based interpretability.\u003C\u002Fp>\u003Cp>There is still a useful research question left open: what kind of semantics, if any, do SAE features reliably support outside idealized experiments? The paper suggests they capture model-internal structure, but that is not the same as saying they are useless. It just means their meaning is more conditional and less human-like than a simple feature bag model would predict.\u003C\u002Fp>\u003Cul>\u003Cli>SAE activation-set overlap is a meaningful similarity signal in toy and text settings.\u003C\u002Fli>\u003Cli>Those sparse sets do not better recover human categories than dense representations.\u003C\u002Fli>\u003Cli>For practitioners, interpretability features need stress tests under semantic change, not just visual inspection.\u003C\u002Fli>\u003C\u002Ful>\u003Cp>For developers working on interpretability, this is a reminder to separate “sparse” from “semantically compositional.” Those are not the same property, and this paper argues that the difference matters once you move from curated examples to real language behavior.\u003C\u002Fp>","This paper argues SAE activation sets track model-internal similarity, not human concept boundaries.","arxiv.org","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.11197",null,"https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786518181029-jyb4.png","research","en","2aa54cfd-2caf-4c34-834c-e771e1308747",[17,18,19,20,21],"sparse autoencoders","interpretability","LLMs","feature sets","concept similarity",[23,24,25],"SAE activation sets can be meaningful without matching human concepts.","Sparse features track model-internal similarity more than category boundaries.","Interpretability workflows should test feature stability under semantic change.",1,"2026-08-12T07:02:32.315345+00:00","2026-08-12T07:02:32.304+00:00",{"tags":30,"relatedLang":33,"relatedPosts":37},[31],{"name":19,"slug":32},"llms",{"id":15,"slug":34,"title":35,"language":36},"sparse-autoencoders-set-level-instability-zh","SAE 不是特徵袋","zh",[38,44,50,56,62,68],{"id":39,"slug":40,"title":41,"cover_image":42,"image_url":42,"created_at":43,"category":13},"605dd415-e62d-455a-bb4b-d1d2fa487c1b","convawg-controlled-vawg-dialogue-generation-en","ConVAWG generates controlled VAWG dialogues","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786516377348-lc87.png","2026-08-12T06:32:30.834532+00:00",{"id":45,"slug":46,"title":47,"cover_image":48,"image_url":48,"created_at":49,"category":13},"4c94994e-d58b-4f24-a480-ad026fd60e04","surgical-wam-video-pretraining-robot-control-en","Surgical WAM uses video to train robot control","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786514579882-zhmr.png","2026-08-12T06:02:32.763172+00:00",{"id":51,"slug":52,"title":53,"cover_image":54,"image_url":54,"created_at":55,"category":13},"6d197f27-628f-4a63-883d-81a0d9f5c4b5","swe-bench-verified-model-leaderboard-limit-en","SWE-bench Verified has stopped being a clean model leaderboard","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786498365866-w6ew.png","2026-08-12T01:32:20.055621+00:00",{"id":57,"slug":58,"title":59,"cover_image":60,"image_url":60,"created_at":61,"category":13},"30d3b27a-e5fb-4c3c-aed2-b9f078b8be23","dutch-government-llm-benchmark-values-en","Dutch Government LLMs Need More Than Accuracy","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786431775018-wj70.png","2026-08-11T07:02:26.380783+00:00",{"id":63,"slug":64,"title":65,"cover_image":66,"image_url":66,"created_at":67,"category":13},"999ef42b-6d1d-40b3-bdf0-dc87c93f3c89","mmdiff-multimodal-feature-discovery-control-en","MMDiff maps and steers multimodal features","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786429983548-5x3u.png","2026-08-11T06:32:31.358093+00:00",{"id":69,"slug":70,"title":71,"cover_image":72,"image_url":72,"created_at":73,"category":13},"8bb7a700-6170-4000-9902-a24f78586cca","tts-evaluators-miss-more-than-naturalness-en","TTS evaluators miss more than naturalness","https:\u002F\u002Fxxdpdyhzhpamafnrdkyq.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Fcovers\u002Finline-1786428179624-mu0v.png","2026-08-11T06:02:29.309089+00:00",[75,80,85,90,95,100,105,110,115,120],{"id":76,"slug":77,"title":78,"created_at":79},"a2715e72-1fe8-41b3-abb1-d0cf1f710189","ai-predictions-2026-big-changes-en","AI Predictions for 2026: Brace for Big Changes","2026-03-26T01:25:07.788356+00:00",{"id":81,"slug":82,"title":83,"created_at":84},"8404bd7b-4c2f-4109-9ec4-baf29d88af2b","ml-papers-of-the-week-github-research-desk-en","ML Papers of the Week Turns GitHub Into a Research Desk","2026-03-27T01:11:39.480259+00:00",{"id":86,"slug":87,"title":88,"created_at":89},"87897a94-8065-4464-a016-1f23e89e17cc","ai-ml-conferences-to-watch-in-2026-en","AI\u002FML Conferences to Watch in 2026","2026-03-27T01:51:54.184108+00:00",{"id":91,"slug":92,"title":93,"created_at":94},"6f1987cf-25f3-47a4-b3e6-db0997695be8","openclaw-agents-manipulated-self-sabotage-en","OpenClaw Agents Can Be Manipulated Into Failure","2026-03-28T03:03:18.899465+00:00",{"id":96,"slug":97,"title":98,"created_at":99},"a53571ad-735a-4178-9f93-cb09b699d99c","vega-driving-language-instructions-en","Vega: Driving with Natural Language Instructions","2026-03-28T14:54:04.698882+00:00",{"id":101,"slug":102,"title":103,"created_at":104},"a34581d6-f36e-46da-88bb-582fb3e7425c","personalizing-autonomous-driving-styles-en","Drive My Way: Personalizing Autonomous Driving Styles","2026-03-28T14:54:26.148181+00:00",{"id":106,"slug":107,"title":108,"created_at":109},"2bc1ad7f-26ce-4f02-9885-803b35fd229d","training-knowledge-bases-writeback-rag-en","Training Knowledge Bases with WriteBack-RAG","2026-03-28T14:54:45.643433+00:00",{"id":111,"slug":112,"title":113,"created_at":114},"71adc507-3c54-4605-bbe2-c966acd6187e","packforcing-long-video-generation-en","PackForcing: Efficient Long-Video Generation Method","2026-03-28T14:55:02.646943+00:00",{"id":116,"slug":117,"title":118,"created_at":119},"675942ef-b9ec-4c5f-a997-381250b6eacb","pixelsmile-facial-expression-editing-en","PixelSmile Framework Enhances Facial Expression Editing","2026-03-28T14:55:20.633463+00:00",{"id":121,"slug":122,"title":123,"created_at":124},"6954fa2b-8b66-4839-884b-e46f89fa1bc3","adaptive-block-scaled-data-types-en","IF4: Smarter 4-Bit Quantization That Adapts to Your Data","2026-03-31T06:00:36.65963+00:00"]