Encoder disaggregation in vLLM: how EPD splits multimodal serving

Encoder disaggregation moves the stage that turns images, video and audio into embeddings onto its own vLLM instances, separate from the GPUs that run the language model. On November 11, 2026, the vLLM project described the latest state of this design, called EPD (Encoder / Prefill / Decode), in a post by contributors from Inferact, InferMatrix and IBM. In its headline test on DeepSeek V4.1 Flash, adding four encoder GPUs to an eight-GPU setup raised throughput per GPU by 6.6–9.5% and cut 90th-percentile time to first token by 44–47% at high load, but lost per-GPU throughput at low load.

How a multimodal request flows: encode, prefill, decode

A text-only request has two phases, covered in our guide to how LLM inference serving works: prefill reads the prompt and builds the KV cache, and decode produces the answer token by token.

A request to a multimodal model adds a step in front. The vLLM post names the three stages:

• Encode (E): the media is preprocessed (fetched, decoded, resized, split into patches) and run through a vision or audio encoder that outputs embeddings.
• Prefill (P): the text prompt and those embeddings are processed together to build the state needed for generation.
• Decode (D): output tokens are generated from that state.

vLLM’s December 2025 architecture post describes the encoder as one-shot and compute-bound, and decode as heavily memory-bound and long-lived. Encoder demand also varies with media size, modality and request mix, independently of prefill and decode demand.

Why separating the encoder helps

When all three stages share one instance, an image-heavy request can stall a batch, and every stage must use the same hardware and parallelism plan. The vLLM documentation lists three benefits of putting the encoder in its own process:

1. Independent scaling. Vision encoders are small next to the language model, so encoder nodes can be added or removed without touching the language-model fleet.
2. Lower time to first token (TTFT). Text-only requests skip the encoder entirely, and encoder work for one request can run while another is already in prefill or decode.
3. Shared reuse. A shared cache lets any instance reuse an embedding another one already computed.

vLLM supports two layouts. E+PD puts the encoder on separate instances and keeps prefill and decode together; E/P/D separates all three. Encoder outputs travel to the language-model side through “EC connectors” (encoder cache), while the KV cache moves from prefill to decode through the existing KV connectors.

What the November update adds

The core pieces ship in vLLM v0.32.0. The biggest is metadata-only requests: rather than forwarding the original media to the prefill side, which may fetch and preprocess it again, the router sends only the layout data the encoder already computed, such as image grid sizes or audio token counts. In an audio test on Qwen3-Omni-30B-A3B this raised throughput from 62.21 to 73.13 requests per second and cut mean latency from 980 to 853 ms; accuracy moved from 75.90% to 75.65%, a gap the authors are still investigating. The update also adds a dedicated encoder-only execution path, embedding transfer over NIXL and Mooncake, shared output reuse through Mooncake Store, and routing support in vLLM Router, SMG and llm-d.

How it relates to prefill/decode disaggregation

EPD extends an older idea. The DistServe paper (arXiv 2401.09670) argued that colocating prefill and decode causes interference and forces one resource plan on both. By putting them on different GPUs, each tuned to its own latency target, DistServe reported serving up to 7.4 times more requests, or a 12.6 times tighter latency target, than the systems it was compared with.

vLLM’s experimental prefill/decode feature gives two reasons for splitting: tuning TTFT and inter-token latency separately, and controlling tail latency caused by prefill jobs interrupting decoding. The same page states that disaggregated prefill does not improve throughput. EPD applies the same logic one step earlier. vLLM’s TPU backend, covered in our open-source roundup on vllm-torchtpu, already supports disaggregated prefill and decode.

When encoder disaggregation pays off: a worked example

In the post’s DeepSeek V4.1 Flash test, the workload was 512 document-question pairs from MP-DocVQA, each with four to six document pages, averaging 4,905 input tokens and only 5.2 output tokens. That is image-heavy, prefill-heavy traffic with almost no decoding.

The baseline used eight GB200 GPUs, four for prefill and four for decode. The EPD setup kept exactly that and added four single-GPU encoder instances, for 12 GPUs. Throughput per GPU counts all 12, so the encoders are not free.

• At concurrency 64 to 128: EPD delivered 6.6–9.5% more input-plus-output throughput per GPU, and P90 TTFT was 44–47% lower.
• At concurrency 32: roughly break-even per GPU.
• At concurrency 16: EPD had lower per-GPU throughput than the baseline.

Our own arithmetic: with 1.5 times the GPUs and 6.6–9.5% more work per GPU, the EPD cluster handled roughly 1.6 times the baseline’s total traffic at high load (12 × 1.066 ÷ 8 ≈ 1.60; 12 × 1.095 ÷ 8 ≈ 1.64). At low load, the extra four GPUs were not earning their keep. As the authors put it, the benefit depends on load and resource allocation, not on separation alone.

The lesson for operators is to size the encoder pool from the media mix. A separate test in the post showed the other side: when GPU preprocessing and hardware video decoding lifted end-to-end throughput by about 94–115% on a 30-frame video benchmark, the prefill/decode tier became the bottleneck.

How to try it

The vLLM post recommends starting with E+PD to check the encoder-to-consumer handoff before adding a separate decode tier. With a vLLM checkout, Mooncake installed and two GPUs, the docs give this integration script, run from the repository root:

GPU_E=0 GPU_PD=1 MOONCAKE_EC_PROTOCOL=tcp \
  bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh

By default it uses Qwen/Qwen2.5-VL-3B-Instruct and compares the disaggregated outputs against a single-GPU baseline; MODEL selects another model and MOONCAKE_EC_PROTOCOL=rdma switches to RDMA on supported hardware. The docs also list shared-filesystem examples, disagg_1e1pd_example.sh and disagg_1e1p1d_example.sh, under examples/disaggregated/disaggregated_encoder/.

When it is not worth it

• Low or mostly text traffic. At concurrency 16 the DeepSeek test lost per-GPU throughput, and text-only requests gain nothing from an encoder tier.
• Small deployments. EPD needs at least one extra GPU or instance plus a transfer layer such as Mooncake or NIXL.
• Counting on best-case cache reuse. The zero-re-encode result was a single best-case run; the authors say real savings depend on how often outputs are reused against the Store’s memory and I/O costs, and concurrent cold requests can still duplicate work.
• Feature limits. Cross-encoder reuse initially supports images only, encoder tensor parallelism of 1 and no dynamic LoRA, and the updated NIXL EC protocol requires producers and consumers to be upgraded together.

What to watch next

• Observability that separates embedding compute, cache-load and transfer time, which vLLM lists as next work.
• A design discussion on letting prefill-side cache hits stop unnecessary encoder work upstream.
• Whether DeepSeek V4.1 Flash metadata-only support moves from vLLM’s main branch into a tagged release.
• Kubernetes deployments through llm-d’s multimodal serving guide.

Key takeaways

• EPD runs the vision or audio encoder of a multimodal model on separate vLLM instances from prefill and decode.
• Operators can scale encoder capacity to the media mix, and text-only requests skip the encoder.
• In vLLM’s DeepSeek V4.1 Flash test, EPD gave 6.6–9.5% more throughput per GPU and 44–47% lower P90 TTFT at concurrency 64–128.
• At low concurrency the extra encoder GPUs cost more than they saved.
• The idea mirrors prefill/decode disaggregation, as set out in the DistServe paper.

Frequently asked questions

What is encoder disaggregation in vLLM?

It is a deployment mode in which a separate vLLM instance runs the multimodal encoder and sends the resulting embeddings to the instances that run prefill and decode. vLLM calls the design EPD, for Encoder / Prefill / Decode.

Does encoder disaggregation always make serving faster?

No. In vLLM’s own DeepSeek V4.1 Flash test it improved throughput per GPU and tail TTFT at high concurrency, broke even at concurrency 32 and was worse per GPU at concurrency 16.

What is the difference between E+PD and E/P/D?

E+PD separates only the encoder and keeps prefill and decode on one instance. E/P/D separates all three stages, so the KV cache also has to be transferred from prefill to decode instances.