Sliding-Window Attention with Attention Sinks Outperforms Linear Attention Post-Training
A new research result shows that switching to a sliding-window attention mask with attention sinks—at no additional cost—beats linear attention during post-trai…