Why did SnapKV lose two-thirds of its answer quality when I batched prompts?

When a language model reads a long document, it stores a key and a value for every token so it does not have to process the document again while writing its answer. That stored history is the KV cache. For a full research paper it takes up a lot of GPU memory, so a family of methods tries to shrink it by keeping only the parts of the document the model is likely to need.

SnapKV is one of those methods. I built kvcache-bench to measure how much answer quality such methods give up. In this test, Llama 3.1 8B answered questions from Qasper, a dataset of questions about research papers. Each answer gets a score out of 100 based on how closely it matches a reference answer. SnapKV was allowed to keep half of the cache.

The first result was reassuring. With the full cache, the model scored 49.9. With SnapKV keeping half, it scored 48.3. Throwing away half the stored history cost less than two points.

Then I made the benchmark faster by batching, which means giving the GPU four questions at once instead of one. Batching should change how quickly the answers come back and nothing else. The model, the questions, and SnapKV’s half-cache limit all stayed the same. The full-cache model scored 50.6, about where it was before. SnapKV scored 15.7.

Running four questions together should not have changed any answer. So why did it wreck SnapKV’s answers and leave the full model’s alone?

The short answer is padding. Batching fills shorter prompts with placeholder tokens so every prompt in the batch has the same length, and the kvpress version of SnapKV ranked those tokens alongside the real text when it chose what to keep. Running one prompt at a time closed the gap, and the benchmark now refuses to combine SnapKV with batching.

Why the result was suspicious

The full-cache run showed that the model itself was fine. Its score barely moved when the batch size changed, so Llama could still read the batched prompts and answer the questions. The failure appeared only when SnapKV chose which parts of the prompt to discard.

A 32-point drop from a setting I had changed to save time was hard to believe. It told me where to look, though. Something that batching changed was reaching SnapKV before it made its choice.

How SnapKV decides what to keep

Each token of the prompt takes one position in the cache. For a research paper, that is thousands of positions. With a budget of 50%, SnapKV has to drop half of them before the model starts writing its answer.

It makes that decision once, right after the model reads the prompt. It scores every position by how much attention the last few tokens of the prompt pay to it. The highest-scoring positions stay in the cache, and SnapKV deletes the rest. Anything it deletes here is gone for the whole answer.

I used the SnapKV implementation from kvpress 0.5.4, NVIDIA’s library of KV-cache compression methods.

Reproducing the collapse

I compared the first ten Qasper examples under both batch sizes. The score measures word overlap between the generated answer and the reference answer. Its absolute value matters less here than the change between two runs that used the same model and questions.

SnapKV kept 50% of the prompt cache in both runs.

Method	Batch 1	Batch 4
Full cache	49.9	50.6
SnapKV, keeping 50%	48.3	15.7

The full-cache score moved by less than one point. SnapKV lost 32.6 points. Ten examples are too few to rank compression methods, but enough to reproduce a failure caused by one configuration change.

What batching adds to the input

The GPU processes a batch as one rectangular block, so every prompt in it must be the same length. Papers are not. The benchmark fills out the shorter prompts with placeholder tokens, called padding. Llama writes its answer at the end of the prompt, so the padding goes at the start.

long prompt:   A B C D E F
short prompt:  _ _ _ A B C

The underscores are padding. They are not part of any paper. Alongside the batch, the model receives an attention mask that marks which positions are real text and which are padding. Llama used that mask and ignored the padding while writing its answer, which is why ordinary generation still worked.

Where SnapKV went wrong

SnapKV’s choice of which positions to keep also needs to know which ones are padding. In the kvpress 0.5.4 code I inspected, SnapKV’s scoring step did not use the padding mask.

So SnapKV scored padding alongside the real text, and the padding competed with the paper for the limited slots in the cache. The model applied its attention mask later, when it generated the answer. By then, SnapKV could already have deleted useful parts of the paper.

This also explains why the full-cache run survived. Both runs used the same attention mask during generation. Only the SnapKV run made an extra selection before generation, and that selection was the step that ignored the mask.

Checking the explanation

The simplest way to remove padding is to process one prompt at a time. With no other prompt to match in length, every position belongs to the actual input.

I reran the same ten examples at batch 1. The full cache scored 49.93 and SnapKV scored 48.34. The 32-point gap disappeared without changing the model, questions, or cache budget.

StreamingLLM, another method in the benchmark, has the same weakness, and it is easier to see. It always keeps the first few positions of the prompt, because models tend to direct a large share of attention to them. In a left-padded batch, the first positions of a shorter prompt can be padding rather than text. Running at batch 1 removes that problem too.

Preventing the benchmark from repeating the mistake

The batched scores did not measure the comparison I intended, so I deleted them. Keeping them with a warning would still leave invalid numbers around for a future chart or summary to pick up.

I also added a configuration check. The benchmark now rejects quality or performance runs that combine a cache-eviction method with a batch size above one. A full-cache baseline may still use batching, because it never ranks positions or throws part of the cache away.

With the check in place, I reran the full quality sweep at batch 1. Across 50 Qasper examples, the full cache scored 49.57 and SnapKV scored 46.93 while keeping half of the prompt cache. That is the result I now report for this setup.

What this result does and does not show

The ten-example comparison isolates a batch-size failure. It is too small to rank SnapKV against other compression methods. The 50-example rerun also covers one model, dataset, cache budget, and implementation. It is not a general claim about SnapKV.

The explanation covers only the code I inspected, which is Llama 3.1 8B with left-padded batches and kvpress 0.5.4. I confirmed that the SnapKV scoring step did not use the padding mask. I did not record every position SnapKV kept during the failed run, so I cannot say how many padding positions survived in each example. The score collapse, the stable control, and the missing mask all point to the same failure, but I have not measured that last number.

Conclusion

SnapKV’s compression was never the problem. Batching changed its input. Grouping prompts of different lengths added padding to the shorter ones, and the kvpress 0.5.4 scoring step treated that padding as part of the prompt. Padding competed with the paper for half the cache, and the Qasper score fell from 48.3 to 15.7. The full-cache control stayed near 50 because it never made that selection, and the gap disappeared as soon as each prompt ran on its own.

For this implementation, the benchmark now runs cache-eviction methods only at batch 1. The result I report is 46.93 for SnapKV against 49.57 for the full cache across 50 examples. I think the lesson reaches beyond SnapKV. Any compression method that runs on batched input has to exclude padding in every step that scores, selects, or removes cache positions, and the model’s attention is only one of those steps. If one step uses a different definition of which tokens are real, the benchmark ends up measuring the padding instead of the method.

GitHub repository

The benchmark code, configuration check, and regression tests are available in the kvcache-bench repository.