Google TurboQuant: 6x Memory Reduction for AI Models

Google Research has launched TurboQuant, a compression algorithm that reduces memory usage in large language models by more than six times, without any loss in model accuracy and without requiring additional training or calibration.

The algorithm targets one of today's biggest bottlenecks in AI infrastructure: the KV cache, the memory-intensive buffer that models use during inference. TurboQuant compresses this cache down to 3 bits per element through a two-stage process called PolarQuant and Quantized Johnson-Lindenstrauss.

The results are striking. On NVIDIA H100 accelerators, internal tests show up to eight times faster attention computation. Combined with the memory reduction, that means the same hardware can run much larger models, support more concurrent users, or unlock longer context windows.

The practical implications are broad. Smartphones and laptops can now run far more powerful AI models locally, reducing dependency on cloud-based inference. Apple, which recently partnered with Google to bring Gemini into Siri, is among the companies positioned to benefit directly.

For enterprise IT leaders, this matters for two reasons. First, AI inference costs are a growing budget concern. TurboQuant promises to cut those costs in half. Second, local AI processing reduces data privacy risk by keeping sensitive data on-device.

The algorithm has been accepted at ICLR 2026 and AISTATS 2026 and is now available to developers.