滑动窗口注意力（SWA）将每个 token 的注意力限制在固定大小的局部邻域内，从而降低长上下文模型的注意力和 KV 缓存内存占用。一项新研究表明，在训练后切换到带注意力汇聚（attention sinks）的滑动窗口掩码，可以在不增加成本的情况下超越线性注意力。
RT @NielsRogge: For folks wondering what Sliding Window Attention is, there's a method for it on Papers with Code Sliding Window Attention (SWA): A local attention pattern that restricts each token to attending only within a fixed-size neighborhood instead of the full sequence. This reduces attention and KV-cache memory for long-context models, while periodic global-attention layers can preserve broader context. Find it here: https://t.co/K1MhZVasL8
Simple beats complicated: We show that switching to a sliding-window attention mask with attention sinks (at no cost) beats linear attention post-training. Huge thanks to my collaborators @RheaSukthanker, @CameronPashmina, and @Emy_Aze. Paper: https://t.co/h8DIc223Su
