Linear attention

Standard transformer attention (the kind powering most LLMs) has a quadratic cost problem: double the context length and the compute and memory requirements roughly quadruple. Linear attention rewires that math by replacing the softmax scoring function with a simpler kernel, allowing the attention computation to be reformulated as a linear recurrence. The result is an architecture that grows linearly in cost with sequence length rather than quadratically, which matters enormously when context windows stretch to a million tokens or more.

In practice, linear attention historically traded accuracy for efficiency. Modern variants with gated linear attention, data-dependent multiplicative gates, and hybrid architectures that mix linear and softmax attention layers have largely closed that gap. By mid-2026, several open-weight frontier models including Qwen3.5 shipped gated linear-attention hybrids, and the trend appeared in engineering dispatches as one of the five architecture shifts defining the current generation of frontier models.

For builders, linear attention mostly shows up as an explanation for why a model handles very long contexts cheaply at inference time, or why a new model generation costs less per token without losing quality. The mechanism itself sits at the specialist layer; its effects surface at the pricing and capability layer that affects day-to-day routing decisions.