Google's TurboQuant reduces AI LLM cache memory capacity requirements by at least six times — up to 8x performance boost on Nvidia H100 GPUs, compresses KV caches to 3 bits with no accuracy loss

Google TurboQuant.

Google Research published TurboQuant on Tuesday, a training-free compression algorithm that quantizes LLM KV caches down to 3 bits without any loss in model accuracy. In benchmarks on Nvidia H100 GPUs, 4-bit TurboQuant delivered up to an eight-times performance increase in computing attention logits compared to unquantized 32-bit keys, while reducing KV cache memory by at least six times.

Go deeper with TH Premium: AI and data centers

(Image credit: Microsoft)

• Photonics and high-speed data movement is the next big AI bottleneck
• The data center cooling state of play
• Massive AI data center buildouts are squeezing energy supplies
• Ultra Ethernet: The data center interconnection of tomorrow