Google’s TurboQuant Slashes LLM Memory Usage by 6x with Zero Accuracy Loss

Google Research has unveiled TurboQuant, a training-free compression algorithm that can reduce LLM key-value cache memory requirements by up to 6x while delivering an 8x performance increase in attention computation—all with zero accuracy loss. The breakthrough, formally introduced on March 24, targets the KV cache bottleneck that has long been one of the biggest practical constraints on LLM inference.

TurboQuant works through a two-stage compression pipeline. First, PolarQuant converts data vectors into polar coordinates, separating magnitude and direction for efficient quantization down to 3 bits. Second, a 1-bit error correction layer using the Quantized Johnson-Lindenstrauss (QJL) algorithm eliminates systematic bias in attention score calculations. The result: 100% recall in needle-in-a-haystack tests up to 104k tokens, matching or beating baselines on long-context benchmarks.

Critically, TurboQuant is training-free and data-oblivious—no dataset-specific tuning or calibration required. Benchmarks on Nvidia H100 GPUs showed that enterprises could see a 50%+ cost reduction in inference workloads. The papers will be presented at ICLR 2026 and AISTATS 2026.

Source

Google Research Blog | VentureBeat | Tom’s Hardware

Why This Matters

KV cache memory has been the silent bottleneck holding back practical LLM deployment, especially for long-context workloads. Every token in the context window adds to the cache, and at 128k+ context lengths, you’re often GPU memory-bound before you’re compute-bound. A 6x reduction in that cache fundamentally changes the economics of serving large models.

The “training-free” aspect is the real kicker. Most quantization approaches require careful calibration on representative data, making them fragile and deployment-specific. TurboQuant’s data-oblivious design means you can drop it into existing inference pipelines without retraining or tuning. For the broader industry, this could mean running frontier-class models on significantly less hardware—or fitting dramatically longer contexts on existing infrastructure. This is the kind of algorithmic breakthrough that compounds: it makes everything else cheaper.