Your single-node vLLM deployment ran fine until you tried loading a 70B-parameter model. Or a Mixture-of-Experts model. Or you pushed context length and concurrency high enough that the KV cache alone consumed the headroom you had left. CUDA OOM at startup is the clearest signal that you have hit the single-node wall.

This post is the migration guide for what comes next. Red Hat engineer Ravindra Patil is presenting the multi-node path at KubeCon India on June 18, 2026, and the momentum behind it is real. We will cover the parallelism tradeoffs you need to understand before writing a single YAML file, the native multi-node options in vLLM itself, the Kubernetes primitive that makes multi-node replicas possible, and when the production overhead of llm-d is worth adding on top.

If you are looking for the full CNCF inference stack picture (GPU DRA, KAI Scheduler, Grove, and the Endpoint Picker), that is covered in Kubernetes LLM Inference Stack 2026. This post starts one level below that: vLLM, parallelism strategies, and the Kubernetes primitives that hold a multi-node replica together.

You Hit the Single-Node Wall. Now What?

A model needs to fit three things in aggregate GPU memory to serve requests: the model weights, the KV cache for your target context length and batch size, and activation and runtime overhead. When you can no longer fit all three on a single node, even with all GPUs on that node working together, you have hit the wall.

vLLM’s official documentation lays out the decision path cleanly:

  • If the model fits on a single GPU, distributed inference is unnecessary.
  • If the model is too large for a single GPU but fits on a single node with multiple GPUs, use tensor parallelism.
  • If the model is too large for a single node, combine tensor parallelism with pipeline parallelism.

Dense models in the 70B+ range and large MoE models like DeepSeek-R1 (671B parameters) typically land in that third bucket. Google’s multi-host GKE tutorial targets DeepSeek-R1-671B and Llama 3.1 405B as the canonical examples of models that cannot fit on one node.

flowchart TD
    A[Model to serve] --> B{Fits on 1 GPU?}
    B -->|Yes| C[Single GPU\nNo distribution needed]
    B -->|No| D{Fits on 1 node\nmulti-GPU?}
    D -->|Yes| E[Tensor parallelism\nTP within node via NVLink]
    D -->|No| F[TP within nodes\n+ PP across nodes]
    F --> G{Need P/D disaggregation,\ncache routing, or MoE\nexpert parallelism?}
    G -->|No| H[Native multi-node vLLM\nRay or multiprocessing]
    G -->|Yes| I[llm-d on Kubernetes\nLWS + wide-EP]

The decision path from single-GPU serving to multi-node llm-d. Each rung adds operational complexity; stop at the lowest rung that meets your requirements.

Tensor Parallelism vs Pipeline Parallelism: Get This Right First

Before running any multi-node command, you need a clear mental model of the two parallelism axes. Choosing wrong wastes GPUs or adds unnecessary latency.

Tensor parallelism: sharding weight matrices within a node

Tensor parallelism (TP) splits individual weight matrices across multiple GPUs. The two patterns are column parallelism (split the weight matrix along columns, each GPU computes a partial output) and row parallelism (split along rows, GPUs sum their partial results). The key property: all GPUs process every token together, synchronizing after each matrix operation.

This synchronization is constant and all-to-all, which means tensor parallelism requires high-bandwidth interconnects. NVLink between GPUs within a node provides the bandwidth needed. TP can actually reduce latency because each GPU only handles a fraction of the computation per token, but it is expensive to run across a slower inter-node link.

Keep tensor parallelism within a node.

Pipeline parallelism: splitting layers across nodes

Pipeline parallelism (PP) assigns different transformer layers to different GPUs or nodes. GPU 0 runs layers 1-N, GPU 1 runs layers N+1 to 2N, and so on. Each GPU processes its layer group and passes the intermediate activations to the next stage.

Communication happens exactly once per pipeline stage boundary, which is far less than the constant TP synchronization. The tradeoff: pipeline parallelism does not reduce per-token latency. A token still passes through all N stages sequentially. PP is a capacity and memory solution, not a speed one.

Use pipeline parallelism across nodes, especially when the inter-node link is slower than intra-node NVLink.

The rule

vLLM’s documentation states this directly: “The common practice is to set the tensor parallel size to the number of GPUs in each node, and the pipeline parallel size to the number of nodes.”

graph LR
    subgraph "Node A — Tensor Parallel (TP=4)"
        direction TB
        G1[GPU 0\nLayers 1–24\nCol shard] <-->|NVLink\nall-to-all| G2[GPU 1\nLayers 1–24\nCol shard]
        G2 <-->|NVLink| G3[GPU 2\nLayers 1–24\nRow shard]
        G3 <-->|NVLink| G4[GPU 3\nLayers 1–24\nRow shard]
    end

    subgraph "Node B — Tensor Parallel (TP=4)"
        direction TB
        G5[GPU 0\nLayers 25–48\nCol shard] <-->|NVLink| G6[GPU 1\nLayers 25–48\nCol shard]
        G6 <-->|NVLink| G7[GPU 2\nLayers 25–48\nRow shard]
        G7 <-->|NVLink| G8[GPU 3\nLayers 25–48\nRow shard]
    end

    G1 -->|InfiniBand / RoCE\nactivations once per stage| G5

    style G1 fill:#1e3a5f,color:#fff
    style G2 fill:#1e3a5f,color:#fff
    style G3 fill:#1e3a5f,color:#fff
    style G4 fill:#1e3a5f,color:#fff
    style G5 fill:#1a4a2a,color:#fff
    style G6 fill:#1a4a2a,color:#fff
    style G7 fill:#1a4a2a,color:#fff
    style G8 fill:#1a4a2a,color:#fff

TP shards weight matrices within a node across NVLink. PP passes activations between nodes once per pipeline stage over InfiniBand or RoCE. The intra-node communication (all-to-all) is far more frequent than the inter-node communication (one pass per stage).

The Minimal Step: Native Multi-Node vLLM

Before reaching for llm-d, native multi-node vLLM is the right first step. It is stable, requires no additional Kubernetes controllers, and gets you across the single-node wall. The cost: you get exactly one large logical replica with no prefill/decode separation, no cache-aware routing, and no independent phase scaling.

The parameter mapping follows the rule above. For a 2-node cluster with 8 GPUs per node, tensor_parallel_size equals 8 (GPUs per node) and pipeline_parallel_size equals 2 (number of nodes):

Option A: Ray backend (default for multi-node)

# vLLM v0.23.0 — 2 nodes, 8 GPUs each.
# Ray must be running across both nodes before launch.
vllm serve <model> \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --distributed-executor-backend ray

Option B: Multiprocessing backend (no Ray required)

# Head node (rank 0) — runs the API server.
vllm serve <model> \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr <HEAD_NODE_IP>

# Worker node (rank 1) — headless, joins the head.
vllm serve <model> \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --nnodes 2 \
  --node-rank 1 \
  --master-addr <HEAD_NODE_IP> \
  --headless

Both options give you a single logical endpoint serving a model too large for one node. What you do not get: the ability to scale prefill workers independently from decode workers, routing requests to the pod that already cached a prefix, or the expert parallelism that wide MoE models need. If your workload is relatively uniform and a single large replica meets your throughput requirements, stop here.

How Kubernetes Schedules a Multi-Node Replica: LeaderWorkerSet

The next question most teams hit: how do you express “this one model replica spans 4 pods on 4 nodes that must all come up together, know each other’s addresses, and die together if any one fails” in Kubernetes? A Deployment cannot express this. A StatefulSet gets you indexed names but not coordinated lifecycle or gang scheduling.

LeaderWorkerSet (LWS) is the answer. It is a kubernetes-sigs API designed for exactly this: “deploying a group of pods as a unit of replication” for “multi-host inference workloads where the LLM will be sharded and run across multiple devices on multiple nodes.”

Each LWS replica group is one leader pod (index 0, typically runs the vLLM API server and its own model shard) plus N worker pods (indices 1..N-1, each running a model shard and connecting to the leader). All pods in the group are created concurrently, share a lifecycle, and can be gang-scheduled all-or-nothing. If any pod in the group fails, the group restarts together.

Install LWS (v0.8.0, current as of June 2026)

VERSION=v0.8.0
kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/lws/releases/download/${VERSION}/manifests.yaml

# Verify the controller is running
kubectl get pods -n lws-system

LeaderWorkerSet shape for a 2-node model replica

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: vllm-multinode
spec:
  replicas: 1                          # number of replica groups (model copies)
  leaderWorkerTemplate:
    size: 2                            # 1 leader + 1 worker = 2-node replica
    leaderTemplate:
      metadata:
        labels:
          role: leader
      spec:
        containers:
          - name: vllm-leader
            image: vllm/vllm-openai:v0.23.0
            args:
              - "--tensor-parallel-size=8"
              - "--pipeline-parallel-size=2"
              - "--node-rank=0"
              - "--nnodes=2"
            resources:
              limits:
                nvidia.com/gpu: "8"
    workerTemplate:
      spec:
        containers:
          - name: vllm-worker
            image: vllm/vllm-openai:v0.23.0
            args:
              - "--tensor-parallel-size=8"
              - "--pipeline-parallel-size=2"
              - "--node-rank=1"
              - "--nnodes=2"
              - "--headless"
            resources:
              limits:
                nvidia.com/gpu: "8"

This skeleton illustrates the structure. For a production-ready manifest, including the environment variables LWS injects for leader address discovery, pull from the llm-d wide-ep-lws guide or the LWS examples directory — the guide ships tested overlays for GKE, CoreWeave, and DGX Cloud GB200.

graph TD
    subgraph "LWS Replica Group 0"
        L0[Leader Pod\nindex 0\nAPI server + shard A]
        W1[Worker Pod\nindex 1\nshard B]
        W2[Worker Pod\nindex 2\nshard C]
        L0 -.gang scheduled.-> W1
        L0 -.gang scheduled.-> W2
    end

    subgraph "LWS Replica Group 1"
        L1[Leader Pod\nindex 0\nAPI server + shard A]
        W3[Worker Pod\nindex 1\nshard B]
        W4[Worker Pod\nindex 2\nshard C]
        L1 -.gang scheduled.-> W3
        L1 -.gang scheduled.-> W4
    end

    K8s[Kubernetes Scheduler] -->|all-or-nothing| L0
    K8s -->|all-or-nothing| L1
    LB[Load Balancer / llm-d Router] --> L0
    LB --> L1

Two LWS replica groups, each running a full copy of the model sharded across 3 pods. The scheduler places each group all-or-nothing. The llm-d router or a standard load balancer distributes requests across groups.

When You Outgrow Raw Multi-Node vLLM: llm-d

Native multi-node vLLM gives you one large logical replica. llm-d gives you a production serving system built on top of vLLM. It does not replace vLLM — it wraps vLLM as the model server, so your weights, quantization configuration, and serving parameters carry over.

llm-d v0.7.0 (released May 12, 2026, CNCF Sandbox, donated jointly by IBM Research, Red Hat, and Google Cloud) adds four layers on top of the vLLM foundation:

  • Intelligent Routing - prefix-cache-aware request routing so requests land on the pod that already holds the relevant KV prefix. This is covered in depth in the Kubernetes LLM Inference Stack 2026 post.
  • Advanced KV-Cache Management - tiered KV-cache offloading across GPU, CPU, and disk, extending effective cache capacity without adding hardware.
  • Serving Large Models - prefill/decode (P/D) disaggregation and wide expert parallelism across nodes. This is the multi-node heart of this post.
  • Operational Excellence - predicted-latency scheduling and workload-variant autoscaling.

Wide expert parallelism for MoE models

For large Mixture-of-Experts models, llm-d ships a concrete “well-lit path” in its wide-ep-lws guide: serving DeepSeek-R1-0528 using vLLM P/D disaggregation with a wide expert parallel pattern across LeaderWorkerSets.

The topology in the guide: Prefill Data Parallelism 16, Decode Data Parallelism 16, across 32 total GPUs (validated on H200 and B200 clusters).

The deployment has three components:

1. The llm-d Router (choose standalone or gateway mode)

# Standalone router
helm install llm-d-router \
  oci://ghcr.io/llm-d/charts/llm-d-router-standalone-dev \
  --version v0 \
  --namespace ${NAMESPACE}

# Or gateway-integrated router
helm install llm-d-router \
  oci://ghcr.io/llm-d/charts/llm-d-router-gateway-dev \
  --version v0 \
  --namespace ${NAMESPACE}

2. The model server (vLLM, deployed via Kustomize overlay for your infrastructure)

# Apply the overlay for your provider: gke, coreweave, or dgx-cloud-gb200
kubectl apply -n ${NAMESPACE} \
  -k guides/wide-ep-lws/modelserver/gpu/vllm/${INFRA_PROVIDER}

3. Optional monitoring stack for latency and throughput observability.

Check the wide-ep-lws guide README for the current invocations — this is a versioned guide and exact flags may change between llm-d releases.

The network fabric is the real prerequisite

Before deploying anything, confirm your data-center networking. Multi-node LLM inference has two distinct fabric requirements that most teams underestimate.

Within a node: NVLink for tensor-parallel shards. This is standard on H100/H200/B200 nodes and requires no configuration.

Across nodes: RDMA, either InfiniBand or RoCE. This is required for two distinct data paths:

  1. Pipeline-parallel activation passing between nodes (lower bandwidth but latency-sensitive).
  2. KV-cache transfer between disaggregated prefill and decode workers over NIXL, llm-d’s transport layer, which supports InfiniBand, RoCE, and TPU ICI. Standard Ethernet negates the benefit of disaggregation.

For the wide-EP MoE path, there is a third and stricter constraint: full-mesh all-to-all RDMA connectivity. The wide-ep-lws guide is explicit: “Networks restricted to communicating only between matching NIC IDs (rail-only connectivity) will fail.” Many enterprise clusters ship with rail-only topologies where each NIC only communicates with its matching NIC ID counterpart on other hosts. That topology works for some HPC workloads but breaks expert-parallel MoE serving. Confirm with your network team before provisioning the GPU cluster.

sequenceDiagram
    participant Client
    participant Router as llm-d Router
    participant PF as Prefill Workers (DP 16)
    participant DC as Decode Workers (DP 16)
    participant Cache as KV Cache (NIXL/RDMA)

    Client->>Router: Request (prompt tokens)
    Router->>Router: Prefix-cache lookup
    Router->>PF: Route to prefill group\n(cache-miss or new prefix)
    PF->>Cache: Write KV cache over NIXL/RDMA
    PF->>DC: Signal ready + transfer reference
    Cache->>DC: Stream KV tensors over InfiniBand/RoCE
    DC->>Client: Generated tokens

    note over PF,DC: All-to-all RDMA required\nRail-only topology will fail
    note over Router: LeaderWorkerSet groups\neach multi-node worker set

Request flow through llm-d’s disaggregated wide-EP architecture. The prefill and decode phases run on separate GPU groups, each internally parallelized. KV cache moves over RDMA between phases. The router makes routing decisions based on prefix-cache affinity.

The Decision Framework: Which Rung Are You On?

Most teams should start at the lowest rung that solves their immediate problem, then climb only when they hit the next constraint. Each rung adds real operational cost.

RungWhatWhen to stop here
Single-node vLLMTP across GPUs within one nodeModel fits (weights + KV cache + overhead) on one node. Stay here — multi-node adds real complexity.
Multi-node vLLMTP within nodes + PP across nodes, Ray or multiprocessingModel doesn’t fit on one node, but traffic is relatively uniform, a single large replica is sufficient, and you don’t need disaggregated phase scaling or cache-aware routing.
llm-d distributedP/D disaggregation, prefix-cache routing, wide expert parallelismYou need disaggregated phase scaling, prefix-cache routing for repeated-prefix workloads (RAG pipelines, long system prompts), MoE expert parallelism, or multi-replica scale on shared GPU clusters.

Maturity note: llm-d is CNCF Sandbox as of this writing, following its donation in March 2026 by IBM Research, Red Hat, and Google Cloud. Sandbox status means the project is early-stage; API stability is not guaranteed before the project reaches Incubating. llm-d is appropriate for platform teams building toward production, for staging environments, and for infra experimentation. Treat it accordingly — the primitives are sound, but version pinning and a tested upgrade path matter.

Once you have scaled your serving layer, the security posture of what you have just built matters. See Securing AI Inference Servers on Kubernetes for the hardening playbook and Securing LLM Gateways: LiteLLM, vLLM Proxy, and the Attack Surface You’re Ignoring for the gateway layer.

Frequently Asked Questions

When do I actually need multi-node inference instead of single-node vLLM?

When the model’s weights plus its KV cache no longer fit in the aggregate GPU memory of one node, even with tensor parallelism across all GPUs on that node. This typically means large dense models (70B+ at higher precision, 405B/671B class) or wide MoE models. vLLM’s own guidance: fits on one GPU — single GPU; fits on one node — tensor parallelism; too large for one node — combine tensor and pipeline parallelism.

What’s the difference between tensor parallelism and pipeline parallelism, and which do I use across nodes?

Tensor parallelism shards individual weight matrices across GPUs and is communication-heavy, requiring NVLink or InfiniBand bandwidth. Keep TP within a node where that bandwidth exists. Pipeline parallelism splits the model by transformer layers with much lower communication (one transfer per stage) but does not reduce per-token latency. Use PP across nodes, especially when the inter-node link is slower than intra-node NVLink. The vLLM rule of thumb: “set the tensor parallel size to the number of GPUs in each node, and the pipeline parallel size to the number of nodes.”

How do I set tensor-parallel-size and pipeline-parallel-size for a 2-node, 8-GPU-per-node cluster?

Set --tensor-parallel-size 8 (GPUs per node) and --pipeline-parallel-size 2 (number of nodes). Multi-node vLLM defaults to the Ray distributed executor backend. Override with --distributed-executor-backend ray (explicit) or mp (multiprocessing, no Ray required, uses --nnodes, --node-rank, --master-addr, and --headless).

How does Kubernetes schedule a single model replica that spans multiple nodes?

With a LeaderWorkerSet (LWS), the kubernetes-sigs API for deploying “a group of pods as a unit of replication.” Each replica is one leader pod plus indexed worker pods created together with a shared lifecycle. LWS supports all-or-nothing gang scheduling so a multi-node replica never comes up half-scheduled. A standard Deployment cannot express this constraint, and a StatefulSet does not provide coordinated lifecycle or gang scheduling. LWS v0.8.0 is the current release; llm-d uses it as the underlying orchestration primitive for both multi-node and wide-EP deployments.

Does llm-d replace vLLM, and what does it add for multi-node serving?

No — llm-d wraps vLLM as its model server. Your weights, quantization settings, and serving parameters carry over; llm-d adds the distributed-serving infrastructure on top. For multi-node specifically it adds: prefill/decode disaggregation across node groups, KV-cache transfer over NIXL (InfiniBand/RoCE/TPU ICI), prefix-cache-aware routing, and the wide expert parallel path for MoE models. The wide-EP guide deploys DeepSeek-R1-0528 across 32 GPUs (prefill DP 16 / decode DP 16) using LeaderWorkerSets, but it requires full-mesh all-to-all RDMA — rail-only topologies will fail outright.

Version Reference

ComponentCurrent VersionRelease DateSource
vLLMv0.23.0Jun 13, 2026github.com/vllm-project/vllm/releases
llm-dv0.7.0May 12, 2026github.com/llm-d/llm-d/releases
LeaderWorkerSetv0.8.0Jan 26, 2026github.com/kubernetes-sigs/lws/releases

Note that our April post covering the full CNCF inference stack referenced an earlier llm-d version. This post uses the current v0.7.0. Re-verify these versions against the GitHub releases pages before implementing — vLLM in particular releases frequently.