Executive Summary
An observation that surfaced in AI circles offers a useful starting point. It holds that China's open-weight frontier is converging on four techniques:
- (1) Linear attention. Most labs except DeepSeek use it: Kimi's KDA, Qwen's GDN, and KDA-style layers at GLM.
- (2) Sparse attention with an indexer or compression. Most labs except Kimi use it: DSA, QSA, LSA, IndexPool.
- (3) Upgraded residual connections are now standard: mHC and AttnRes.
- (4) Muon is close to universal on the training side.
This observation offers an interesting lens. We use each of its four points as a lens and read it against primary evidence: the config.json and modeling code each lab actually shipped on Hugging Face, plus its technical report. For each technique we explain the mechanism against DeepSeek V4 as the reference design. That is the architecture we decomposed in Research #15.
Core finding: Read through this lens, the four points describe one optimization target seen from four angles: minimize the HBM bytes moved per decoded token at long context, on bandwidth-poor hardware. In the landscape below, every lab's newest generation moves left, to fewer bytes. Their vertical position, attention FLOPs per byte, stays spread across a 35× range, because each lab picked a different attention mechanism to get there. Linear attention cuts KV capacity, but leaves decode bandwidth dominated by the remaining global layers. Sparse attention cuts bandwidth, but makes the indexer the new bottleneck. The two schools have therefore started to merge, in GLM-5.3-Flash and in Qwen3.8-Flash-Next (a Qwen4 preview).
Where the evidence sharpens the observation
- "LSA" is Meituan's, not Qwen's or GLM's. It is LongCat Sparse Attention, shipped in LongCat-2.0.
- "GLM uses KDA" is true only of GLM-5.3-Flash. The flagship GLM-5 to 5.3 line is MLA + DSA with no linear layers.
- Qwen's sparse attention is not in the mainline. Qwen3.5, 3.6 and 3.8 are GDN-hybrid with no sparse attention. QSA appears only in the experimental Qwen3.8-Flash-Next.
- Kimi's exception holds, but Kimi K2.5, K2.6 and K2.7 were still plain MLA. KDA reached a Kimi flagship only with K3 in July 2026.
- DeepSeek is not the only lab without linear attention. MiniMax, Meituan, Tencent, Xiaomi and StepFun ship none either. They use different softmax designs, as the taxonomy below shows.
Continue reading with a subscription.
The remaining 13 sections carry the evidence. Four lenses separate what actually spread between labs from what only looks shared: a common layer layout for linear attention but four different mixers; DSA as the sparse-attention template, with the indexer rather than the attention as the next thing to optimise; mHC copied parameter-for-parameter from DeepSeek to Zhipu; and Muon shared at the frontier but diverging in the details.
Then the numbers. A roofline model built from each model's
published config.json places 19 models on two axes and shows
decode attention to be bandwidth-bound on every H100-class or newer part,
which is why all four mechanisms optimise the same denominator. A second
chart carries that through to intelligence versus cost per task,
where the ranking changes: the lowest bytes/token does not win it.
The closing sections name the three candidate designs the next flagships will choose between, the investment implications of architecture ceasing to be a moat, and the observations that would falsify the convergence thesis.