KV cache eviction for million-token contexts
Design an eviction policy for serving very long contexts on limited GPU memory, comparing sliding windows, attention sinks, heavy-hitter eviction, quantization and tiered or offloaded storage. Reported on 1point3acres' Hugging Face bank; reconstructed in our own words.
Hugging FaceAI system designNo answer yet. Write the first one; the best answer gets endorsed.