How to Linearize Transformers Without Losing Quality

Transformers were usually sped up with ad hoc linearization; this paper shows analysis-driven changes preserve quality better.

• Research org: Unspecified in arXiv abstract
• Core data: Up to 32B parameters
• Breakthrough: Frozen-backbone linearization with sink tokens, short convolutions, and fixed-budget cache routing

Long-context inference is where standard causal self-attention starts to hurt most: the compute cost grows quadratically, so every extra token makes the bill and latency worse. This paper is about making transformers more linear without treating the model like a black box and hoping the replacement pieces behave.

That matters to engineers because many “post hoc” linearization pipelines promise efficiency, but the hard part is knowing which substitutions keep model quality intact. The authors focus on that exact question in a strict frozen-backbone setting, which makes the tradeoff easier to study and harder to hand-wave away.

What problem this paper is trying to fix

The core bottleneck is causal self-attention. As context length grows, attention gets expensive fast, and that is especially painful for inference on long documents, retrieval-heavy workloads, and other settings where the model has to keep track of a lot of prior tokens.

There have already been many attempts to linearize transformers after training, but the paper argues that the field still lacks a clear way to tell which components are actually responsible for preserving quality. In other words, if a linearized model works, what part of the design made it work?

This paper tries to answer that by isolating the effect of state update design while keeping the backbone frozen. That is a useful constraint: it narrows the problem from “rebuild the model” to “replace the attention mechanism in a way that does the least damage.”

How the method works in plain English

The paper’s main analytical claim is that softmax depends on key-dependent, rank-1 orthogonal projections. The practical takeaway is that this helps explain why delta-style networks do better than simple gated accumulation methods. The authors are not just swapping modules; they are using that analysis to identify where approximation errors come from.

From there, they introduce structural interventions to close the gap. The abstract names three of them: sink tokens, short convolutions, and fixed-budget cache routing. The point is to patch the parts of the linearized system that lose information or route it too aggressively.

In developer terms, this is not a generic “make attention cheaper” idea. It is a recipe for preserving more of the behavior that full self-attention would have provided, while staying inside a linearized inference framework.

• Sink tokens appear to provide a stable place for information to accumulate.
• Short convolutions add local mixing that linear updates may miss.
• Fixed-budget cache routing controls what gets retained when memory is limited.

The paper keeps the backbone frozen, which is important. That means the gains are coming from the linearization design itself, not from retraining the whole model into a new operating regime.

What the paper actually shows

The abstract does include concrete evaluation claims, though not a full benchmark table. The authors say they scale the approach across LLaMA and Qwen models up to 32B parameters. They also say it outperforms prior post hoc baselines on MMLU.

For long-context behavior, the paper says the method matches the long-context retrieval of complex adaptive-caching frameworks. That is a meaningful claim because retrieval is often where linearized or compressed memory systems start to fall apart.

What the abstract does not give is the exact MMLU score, the retrieval metric, or the latency and memory numbers. So while the direction is clear, the magnitude of the gains is not fully specified in the source text here.

Still, the combination of claims matters. Beating prior post hoc baselines on MMLU suggests the method does not just save compute by sacrificing general capability. Matching adaptive-caching frameworks on retrieval suggests it can preserve long-context usefulness, at least at the level reported in the abstract.

Why developers should care

If you are building systems that need longer context windows, this paper points to a more disciplined way to linearize transformers. Instead of treating all replacements as equally good, it argues that the state update design is the lever that matters most.

That has practical implications for model deployment. If the backbone stays frozen, you may be able to explore efficiency upgrades without full retraining, but only if the replacement structure respects the behavior of the original attention mechanism.

The paper also suggests a broader lesson for inference optimization: analysis can be more valuable than brute-force architecture search. By understanding why softmax behaves the way it does, the authors identify specific interventions rather than piling on generic approximations.

Limitations and open questions

The biggest limitation in the source material is that the abstract is high-level. It tells us the method works across large LLaMA and Qwen models, but it does not provide the detailed numbers needed to judge how much headroom remains.

It also does not say whether the approach is easy to integrate into existing serving stacks, how sensitive it is to task mix, or what the compute and memory tradeoffs look like in production. Those are the questions practitioners will want answered before adopting it.

Another open question is how broadly the structural interventions transfer beyond the reported model families and evaluations. The abstract gives a promising signal, but not enough detail to assume the same results everywhere.

Even with those caveats, the paper is useful because it frames linearization as an engineering problem with identifiable failure modes, not just a speed hack. That makes it easier to reason about when a linear replacement is likely to hold up and when it is likely to degrade.

For teams working on long-context inference, the main takeaway is simple: if you want linear transformers that still behave like transformers, the design of the state update is not a minor detail. According to this paper, it is the key variable.