We're excited to share LoLCATs, a new method to convert existing Transformers like Llamas & Mistrals into state-of-the-art subquadratic LLMs. LoLCATs does two things: Attention Transfer: …

Oct 14, 2024 · We thus propose Low-rank Linear Conversion via Attention Transfer (LoLCATs), a simple two-step method that improves LLM linearizing quality with orders of magnitudes less …

Oct 14, 2024 · We're excited to introduce LoLCATs (Low-rank Linear Conversion via Attention Transfer), a new approach for quickly creating subquadratic LLMs from existing Transformers. …

Oct 14, 2024 · However, we developed LoLCATs to make linearizing even more painless and quality-preserving. As our own test, LoLCATS let us create linear versions of the complete …

LoLCATs presents the first viable approach to linearizing larger LLMs. We create the first linearized 70B LLM, taking only 18 hours on one 8×80GB H100 node, and the first linearized …

Oct 14, 2024 · We now share some of our results, where LoLCATs improves the quality, training efficiency, and scalability of linearizing LLMs. Closing the linearizing quality gap. As a first test, …

Nov 1, 2024 · A team of Stanford University researchers has presented LoLCATs, a new method that linearizes standard Transformer LLMs, drastically reducing compute requirements while …

Oct 16, 2024 · This blog explores the use of learnable linear attention, low-rank adaptation (LoRA), and layer-wise optimization to make LLMs more efficient, scalable, and accessible. …

Oct 14, 2024 · Researchers from Stanford University, Together AI, California Institute of Technology, and MIT introduced LoLCATS (Low-rank Linear Conversion via Attention …

Oct 15, 2024 · Together.ai has unveiled a groundbreaking approach to linearizing large language models (LLMs) through a method known as LoLCATs, which stands for Low-rank Linear …