Why did a 4-bit KV cache use 35% more energy than a BF16 cache?
KV-cache compression sounds like a straightforward systems win. Store less data, move fewer bytes through GPU memory, and use less energy to generate each token. That was the assumption I wanted to test.
I built kvcache-bench to
compare several KV-cache compression methods under one measurement protocol.
Using Llama 3.1 8B on an RTX A6000, it measured answer quality, peak memory,
latency, and GPU energy for each method. The full sweep covered 345 runs.
The compression methods did make the cache smaller. With a roughly 16,000-token input, keeping only one in every ten cached positions reduced peak GPU memory from 22.08 GB to 18.37 GB. Storing the cache at four bits instead of BF16 reduced it to 19.12 GB.
But lower memory use did not lead to an equally large energy saving. Eviction reduced the GPU energy needed to generate each new token by only about 8%, and generation speed barely changed. The 4-bit cache performed worse: each token took about 16 milliseconds longer to generate and used 35% more energy than the BF16 cache.
That left the central question for this experiment: why did the GPU hold less data but spend more time and energy processing it?
The short answer is that compression added work the smaller cache did not pay for. At batch 1, each generated token reads about 16 GB of model weights and only about 2 GiB of cache, so even removing 90% of the cache saves little. The 4-bit cache also has to be compressed and converted back before attention can use it, and that extra work appears to have cost more time and energy than the smaller cache saved. I did not profile individual GPU operations, so that last part is what the measurements point to rather than something I traced.
Why I expected compression to save energy
During generation, the model reads its weights and the stored key-value tensors for earlier tokens. As the context grows, the KV cache grows with it. Compression reduces that stored history in one of two ways.
Eviction removes cache positions considered less useful, so the model attends to fewer full-precision keys and values.
Unlike eviction, quantization does not remove token entries from the KV cache. It keeps the keys and values for every token, but stores each number using fewer bits. In this experiment, the model weights stayed in BF16 while Transformers’ quantized-cache implementation used the quanto backend to store the cached keys and values at four or two bits.
Both approaches reduce cache memory. But energy depends on the whole generation path, not only the number of bytes left in the cache. Selecting positions, rearranging tensors, quantizing values, and reconstructing them for attention all require work. Compression saves energy only if the avoided memory traffic is more expensive than that extra work.
What I compared
The uncompressed cache was the baseline. I compared it with StreamingLLM, SnapKV, PyramidKV, TOVA, and quanto-backed 4-bit and 2-bit caches. The eviction methods were tested while retaining 50%, 25%, and 10% of the prompt cache.
Every run used Llama 3.1 8B on the same RTX A6000 with batch size 1. That matters because a single request reads roughly 16 GB of bfloat16 model weights during each decode step. At 16,384 tokens, the uncompressed KV cache is roughly 2 GiB. Even a large percentage reduction in the cache removes a much smaller quantity than the weights that remain untouched.
I separated quality measurement from performance measurement. Quality used 50 examples for each setting across six LongBench tasks and two RULER retrieval tasks. The performance pass used five retrieval prompts at 4k, 8k, and 16k context, with three repeats per setting. Every performance run generated exactly 128 tokens so that one method could not appear cheaper simply by stopping early.
GPU board power was sampled at 10 Hz and integrated over the generation window. The resulting joules-per-token measurement includes prompt processing, decoding, and surrounding input/output handling during that window. It excludes model loading and warmup. I did not subtract the card’s idle power.
The 16k result
The clearest comparison came from the longest performance setting. These values are means over three repeats:
| Cache setting | Peak memory (GB) | GPU J/token | Time between tokens (ms) |
|---|---|---|---|
| Full cache | 22.08 | 16.69 | 40.44 |
| Eviction methods, 10% kept | 18.37 | 15.25 to 15.42 | 39.18 to 40.15 |
| Quantized cache, 4-bit | 19.12 | 22.59 | 56.47 |
| Quantized cache, 2-bit | 18.61 | 21.22 | 53.57 |
The four eviction methods behaved alike. Keeping 10% of the prompt cache saved 3.71 GB and 8% to 9% of energy, while the time between tokens stayed within about one millisecond of the full cache. Both approaches reduced memory, but only quantization made generation much slower.
The 4-bit cache saved memory, but used more energy
The 4-bit cache used 2.96 GB less GPU memory than the full BF16 cache. But that memory saving came with a cost. Generating each token used 22.59 joules instead of 16.69, and the average wait between tokens increased from 40.44 to 56.47 milliseconds.
This was not limited to the longest test. Compared with the BF16 cache, the 4-bit cache used 15% more energy at a 4k context, 32% more at 8k, and 35% more at 16k. The 2-bit cache also used more energy: 17%, 29%, and 27% more at the same context lengths.
Why would a smaller cache use more energy? The quanto implementation has to compress cache values into fewer bits and convert them back into a form the attention calculation can use. That extra work appears to have cost more time and energy than the smaller cache saved in this experiment.
I did not profile the individual GPU operations, so these results cannot show exactly where that extra cost came from. They also do not mean that KV-cache quantization is always inefficient. They show that this Transformers and quanto implementation was slower and used more energy than a BF16 cache for this workload on one RTX A6000.
Quality was not the source of this tradeoff. Across six LongBench tasks, the 4-bit cache scored 48.2 compared with 48.4 for BF16, and both scored 100 on the RULER retrieval tests. The 4-bit cache largely preserved answer quality; its cost was higher latency and energy.
These results come from one model running at batch size 1 on a single RTX A6000. Larger batches, longer contexts, or a different quantization implementation could produce a different result, so the measurements should not be treated as evidence that KV-cache quantization always uses more energy.
What I took from it
On this hardware and software stack, the 4-bit cache traded memory for time and energy. It saved 2.96 GB and preserved quality close to BF16, but added about 16 milliseconds between tokens and used 35% more energy at 16k. The smaller cache did not pay back the extra work needed to compress and reconstruct its values.
The performance summary and task-level quality scores cover all 345 records. The source snapshot contains the raw results.