Writing

• Why did a 4-bit KV cache use 35% more energy than a BF16 cache?

  A 4-bit KV cache saved 2.96 GB on an RTX A6000, but each token took 16 ms longer and used 35% more energy. Compressing and decompressing the cache appears to cost more than the smaller cache saves.

• Why did SnapKV lose two-thirds of its answer quality when I batched prompts?

  Answering questions one at a time, SnapKV scored 48.3 out of 100. Four at a time, it scored 15.7. The kvpress implementation let the filler tokens added by batching compete with the actual paper for space in the cache.

• Why did a GPU with twice the memory bandwidth decode at less than half the speed?

  An RTX 3070 Ti with twice the memory bandwidth of an RTX 4060 Ti decoded at less than half the speed. On a single machine, cutting the CPU's work of launching GPU kernels made decoding 1.69x faster.