vLLM introduces push-based KV transfer, up to 3x better TTFT for disaggregated inference

The push mode has three stages: the decode instance sends block locations to the prefill instance via NIXL notification as soon as it allocates target blocks; after prefill completes, requests are paired by request ID, handling cases where registration and prefill arrival orders differ; then a dedicated background thread on the prefill side issues NIXL WRITE to write KV blocks directly into the decode instance's registered memory, sending completion notifications per tensor-parallel rank, with the decode side starting token generation only after all notifications arrive. Since descriptor preparation and transfer submission no longer occupy either engine's main loop, this per-request overhead no longer slows model execution. Benchmarks used Qwen3-32B in a 1P1D topology on Amazon EC2 P5en instances with EFA interconnect and prefix caching disabled, with input lengths of 1k to 16k and request rates of 1 to 8 QPS. At 8 QPS with 16k input, pull mode showed ~1744 ms TTFT versus ~540 ms in push mode; TPOT dropped from ~10.4 ms to ~7 ms. Under low load with short prompts, the gap between modes is small. For teams running vLLM disaggregated prefill/decode deployments, enabling it means configuring both prefill and decode servers as NixlPushConnector, with the biggest gains at high request rates and long contexts; but push mode requires choosing the decode instance in advance, so teams relying on load-aware routing for QoS management must weigh this trade-off. The feature was first validated on Trainium clusters, then merged into vLLM in an accelerator-agnostic form.