Raven Extrapolates to 16× Its Training Context Length

A new paper from Arshia Afzal, Aviv Bick, Eric Xing, Volkan Cevher and Albert Gu proposes Raven, a linear-time sequence model that keeps a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned input-dependent routing. It targets a familiar trade-off: state-based models like SSMs update memory densely and suffer interference, while sliding-window attention keeps tokens explicitly but loses everything that exits the window. Raven keeps a fixed notebook and decides which few pages to rewrite. The reported results claim strong long-context recall where both approaches sharply degrade, gains that transfer into hybrid architectures, and — the number worth noting — successful extrapolation to context lengths 16× longer than training. That last result suggests the routing learns an actual retention policy rather than memorizing a length. No code has been released, so nothing is independently reproduced yet.