KV Cache Eviction for Efficient LLM Inference
May 2026 — Present
Developing a vLLM KV-cache eviction policy to reduce memory pressure during large language model inference.
PythonvLLMCUDALLM InferenceHigh-Performance Computing
Funded through the Department of Energy Argonne National Laboratory Research Assistant Program, this project develops a novel KV-cache eviction method for vLLM to reduce memory pressure and improve the efficiency of large language model inference.
My work includes designing the eviction policy, modifying vLLM internals, benchmarking latency and throughput, and running large-scale experiments on Argonne systems, including the Aurora supercomputer.
Highlights
- Designing an eviction policy that preserves useful context while lowering GPU-memory demand.
- Benchmarking latency and throughput at scale on Argonne computing systems.