We're excited to share LoLCATs, a new method to convert existing Transformers like Llamas & Mistrals into state-of-the-art subquadratic LLMs. LoLCATs does two things: Attention Transfer: We replace the softmax attentions of an existing Transformer with linear attention analogs, but first train these linear layers to approximate their softmax ...

Oct 14, 2024 · We thus propose Low-rank Linear Conversion via Attention Transfer (LoLCATs), a simple two-step method that improves LLM linearizing quality with orders of magnitudes less memory and compute. We base these steps on two findings.

Oct 14, 2024 · We're excited to introduce LoLCATs (Low-rank Linear Conversion via Attention Transfer), a new approach for quickly creating subquadratic LLMs from existing Transformers. Beyond simply accelerating models, our focus is on creating fast models more efficiently, pushing the boundaries of AI development.

Oct 14, 2024 · However, we developed LoLCATs to make linearizing even more painless and quality-preserving. As our own test, LoLCATS let us create linear versions of the complete Llama 3.1 family (8B, 70B, and 405B) for the first time, doing so no less on the same budget of a parameter-efficient finetune .

LoLCATs presents the first viable approach to linearizing larger LLMs. We create the first linearized 70B LLM, taking only 18 hours on one 8×80GB H100 node, and the first linearized 405B LLM with a

Oct 14, 2024 · We now share some of our results, where LoLCATs improves the quality, training efficiency, and scalability of linearizing LLMs. Closing the linearizing quality gap. As a first test, we evaluated how LoLCATS compared to other linearizing methods at the popular 7B+ LLM scale.

Nov 1, 2024 · A team of Stanford University researchers has presented LoLCATs, a new method that linearizes standard Transformer LLMs, drastically reducing compute requirements while retaining most...

Oct 16, 2024 · This blog explores the use of learnable linear attention, low-rank adaptation (LoRA), and layer-wise optimization to make LLMs more efficient, scalable, and accessible. Learn how LoLCATs enable models to handle larger sequences with reduced computational costs, while maintaining performance.

Oct 14, 2024 · Researchers from Stanford University, Together AI, California Institute of Technology, and MIT introduced LoLCATS (Low-rank Linear Conversion via Attention Transfer). LoLCATS is a two-step method designed to efficiently improve the quality of linearized large language models without the need for expensive retraining on billions of tokens.

Oct 15, 2024 · Together.ai has unveiled a groundbreaking approach to linearizing large language models (LLMs) through a method known as LoLCATs, which stands for Low-rank Linear Conversion via Attention Transfer.