SpectralQuant Achieves Up to 6.62x KV Cache Compression for LLMs via Three-Line Integration
SpectralQuant offers up to 6.62x KV cache compression for Mistral 7B Instruct and other HuggingFace models, with faster decoding and same outputs. It auto-calibrates from a bundled corpus and integrates in three lines of code, providing presets from 5.95x to 6.68x compression.
pip install spectralquant ✂️ Up to 6.62x KV cache compression for LLMs and transformers. Same model. Faster outputs. Smaller KV cache. Try now (2 mins): https://t.co/7qHRy0DDug - KV cache integration via @huggingface's DynamicCache - Three presets: 5.95x (paper), 6.55x (validated), 6.68x (edge)
Side-by-side on Mistral 7B Instruct, 6.62x compression: [FP16] 52.7 tok/s, 120 new tokens [SQ] 55.9 tok/s, 120 new tokens (actual 6.62x) Same outputs. Smaller cache. Faster decode.
Three lines for any @huggingface model: engine = sq.SpectralQuant(compression="high") out = engine.generate(model, tok, prompt) print(out["text"]) Drop-in. No model surgery. No manual calibration. SpectralQuant auto-calibrates from a bundled corpus, every call after that reuses it. Returns text