MIG vs time-slicing: which should I use to share a GPU?

MIG partitions the GPU in hardware with memory and fault isolation (Ampere and newer: A100, H100, and later), best for multi-tenant isolation. Time-slicing packs more workloads by interleaving them but has no memory or fault isolation, so it suits trusted, bursty, or development workloads.

Kubernetes GPU Scheduling: DRA, Gang Scheduling, KAI, Kueue, and Volcano

Q: What is the difference between Kueue and Volcano?

Kueue handles queueing, quota, and admission (deciding when a job may create pods) but leaves pod placement to kube-scheduler. Volcano is a full batch scheduler that places pods and enforces gang scheduling via PodGroups. They are independent systems with separate quota models.

Q: When do I actually need gang scheduling?

When a job only makes progress if all of its pods run together, such as distributed training or multi-node inference. Without gang scheduling, some pods start and hold GPUs while waiting for the rest, wasting capacity and risking deadlock between two half-scheduled jobs.

Q: What is NVIDIA Grove and is it production-ready?

Grove is a declarative API (PodCliqueSet, PodClique, PodCliqueScalingGroup CRDs) for gang-scheduled multi-component model serving with startup ordering and scaling policies, consumed by a gang-aware scheduler such as KAI. As of August 2026 it is alpha (v0.1.0-alpha.11), not production-hardened.

GPU clusters are expensive. A single H100 SXM5 node costs over $30,000, and cloud rental runs $30 per hour per GPU. Teams running these clusters often find their hardware at 40-60% utilization not because there is no work waiting, but because the scheduler allocates GPUs wrong: a job claims resources it cannot fully use, another job deadlocks waiting for the first to release, and node capacity fragments across enough partial allocations that nothing large can schedule at all.

Two changes in 2025 and 2026 make this solvable at the Kubernetes layer. Dynamic Resource Allocation (DRA) graduated to GA in Kubernetes 1.34, replacing the opaque integer-counting model of the device plugin with a structured API that expresses exactly what you need and gives the autoscaler enough information to act. And a set of scheduler tools - KAI Scheduler, Grove, Kueue, and Volcano - matured into layered options with clear division of responsibility.

This post covers how DRA works under the hood, why gang scheduling matters and when it deadlocks, how these four projects compose, and the concrete failure modes in production multi-tenant clusters.

Why GPU scheduling changed: DRA is GA and on by default

The legacy model and its limits

The classic path installs the NVIDIA Kubernetes Device Plugin. The plugin discovers GPUs on each node and advertises them as extended resources under the nvidia.com/gpu key. Pods request GPUs with resources.limits["nvidia.com/gpu"]: "1" and the scheduler places the pod on a node that has at least one GPU slot available.

This model has three structural problems. First, it is opaque: the scheduler sees a count, not a type. You cannot express “I need a GPU with at least 40 GB of memory” or “I need two GPUs on the same NVLink fabric.” Second, it is static: device plugin resources cannot be shared or fractionally allocated at the scheduling layer without vendor-specific workarounds. Third, it breaks autoscaling: because the allocation logic lives in the vendor plugin rather than the scheduler, Cluster Autoscaler cannot simulate GPU allocation when deciding whether adding a node would unblock a pending pod. Scale-out events are unreliable for GPU pods.

What DRA replaces it with: ResourceSlices, ResourceClaims, DeviceClasses, and CEL

DRA graduated to GA in Kubernetes 1.34 (released 2025-08-27) with a stable resource.k8s.io/v1 API group. The DynamicResourceAllocation feature gate is stable and locked on from 1.34 through the current 1.36. You cannot disable DRA on a modern control plane; any cluster still on the device plugin path is using the legacy allocation model by definition.

The DRA object model has four kinds:

ResourceSlice: the driver publishes one ResourceSlice per node (or per set of devices) listing available devices with structured attributes: memory capacity, MIG profile support, NVLink connectivity, and driver-specific properties. The scheduler reads these directly without calling into the vendor driver.

DeviceClass: a named selector that says which devices qualify for a class of workload. Selectors use CEL (Common Expression Language) expressions evaluated against device attributes:

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: example-device-class
spec:
  selectors:
  - cel:
      expression: |-
        device.driver == "driver.example.com"

ResourceClaim: a user’s request for one or more devices matching a DeviceClass, with optional per-device CEL constraints. This is where you express the structured requirements the device plugin could never handle:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: example-resource-claim
spec:
  devices:
    requests:
    - name: single-gpu-claim
      exactly:
        deviceClassName: example-device-class
        allocationMode: All
        selectors:
        - cel:
            expression: |-
              device.attributes["driver.example.com"].type == "gpu" &&
              device.capacity["driver.example.com"].memory == quantity("64Gi")

ResourceClaimTemplate: when a Deployment or Job needs per-pod claims, a ResourceClaimTemplate creates a fresh ResourceClaim for each pod from a common spec. This is the typical path for training jobs where every pod needs its own GPU allocation.

Why DRA is autoscaler-compatible

With the device plugin model, the allocation decision is opaque to the Kubernetes core. Cluster Autoscaler cannot reason about whether a pending pod’s GPU request can be satisfied by an additional node because it cannot simulate the plugin’s allocation logic.

DRA inverts this. Because ResourceSlices are structured Kubernetes objects and DeviceClass selectors are CEL expressions in the API server, the scheduler and Cluster Autoscaler can evaluate allocation in the control plane without vendor-specific logic. Autoscaler can simulate a claim against a hypothetical new node’s ResourceSlice and decide whether provisioning that node would unblock a pending pod. This is what makes GPU scale-out reliable rather than approximate.

DRA allocation flow

sequenceDiagram
    participant D as NVIDIA DRA Driver
    participant A as API Server
    participant S as kube-scheduler
    participant K as kubelet
    participant P as Pod/Container

    D->>A: Publish ResourceSlice (GPUs + structured attributes)
    P->>A: Submit Pod referencing ResourceClaimTemplate
    A->>A: Expand template into per-pod ResourceClaim
    S->>A: Read ResourceSlice list for all nodes
    S->>S: Evaluate DeviceClass CEL selectors
    S->>A: Write device allocation into ResourceClaim
    S->>A: Bind Pod to node
    K->>D: Signal: prepare device for Pod
    D->>K: Return CDI device spec
    K->>P: Start container with GPU access

The scheduler holds the full ResourceSlice inventory and allocates devices in the control plane before binding the pod. Because no opaque vendor logic is involved in that allocation step, Cluster Autoscaler can simulate it - this is what makes DRA autoscaler-compatible.

The NVIDIA DRA driver in practice

The NVIDIA DRA Driver for GPUs (canonical repo: kubernetes-sigs/dra-driver-nvidia-gpu, current v0.4.1) replaces the device plugin and exposes GPUs, MIG instances, and ComputeDomains through DeviceClasses. It installs two kubelet plugins and publishes gpu.nvidia.com and mig.nvidia.com DeviceClasses. Requirements: Kubernetes 1.34.2 or later, NVIDIA GPU Driver 565 or later.

Install via the GPU Operator with the device plugin explicitly disabled:

helm upgrade -i gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --set driver.enabled=true \
  --set devicePlugin.enabled=false \
  --set dra.enabled=true

With devicePlugin.enabled=false, GPU Operator stops installing the legacy device plugin and switches to the DRA driver. Pods that use nvidia.com/gpu resource limits will no longer work on nodes managed by this configuration; they must migrate to ResourceClaims.

Full device vs MIG vs time-slicing

The DRA driver supports three GPU sharing modes with different isolation characteristics:

Mode	Isolation	Best for	Hardware
Full device (default)	Exclusive GPU	Training, memory-heavy inference	Any NVIDIA GPU
MIG (Multi-Instance GPU)	Hardware memory + fault isolation, up to 7 instances	Production multi-tenant inference	Ampere+: A100, H100, H200, B200
Time-slicing	Interleaved execution, no memory or fault isolation	Dev workloads, bursty jobs, trusted tenants	Any NVIDIA GPU

NVIDIA’s documentation states the time-slicing tradeoff plainly: “there is no memory or fault-isolation between replicas.” A GPU doing time-slicing can have one tenant’s workload see another’s memory artifacts or be terminated when a co-tenant exhausts memory. For production multi-tenant environments, MIG is the right choice when the hardware supports it.

Requesting a specific MIG profile via DRA uses a ResourceClaim with a CEL expression against the mig.nvidia.com DeviceClass:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: mig-gpu-claim
spec:
  devices:
    requests:
    - name: mig-request
      exactly:
        deviceClassName: mig.nvidia.com
        selectors:
        - cel:
            expression: |-
              device.attributes["mig.nvidia.com"].profile == "1g.10gb"

Verify the exact attribute key and profile name format against the dra-driver-nvidia-gpu driver documentation for your GPU model; attribute paths are driver-specific and may differ across Ampere generations.

Gang scheduling: the failure that wastes GPUs

The partial-schedule deadlock problem

Distributed training runs across many pods simultaneously. An 8-node training job needs 8 pods, each holding 8 GPUs, all running at the same time. If you let the default scheduler place pods as nodes become available, you will often end up with 7 of 8 pods running, holding 56 GPUs idle while the 8th pod waits for a free node.

Kueue’s documentation describes the outcome directly: “Partial scheduling is wasteful at best: Pods that start hold onto resources while waiting for the rest, and two such jobs can deadlock by each holding a fraction of the resources the other needs.” The deadlock scenario is not hypothetical. Two concurrent training jobs can reach a state where job A holds enough nodes to block job B from completing its allocation, and job B holds enough to block job A. Both jobs stall indefinitely. The only resolution is manual intervention.

All-or-nothing vs partial-schedule deadlock

graph TD
    subgraph "Without gang scheduling"
        A1[8-pod training job submitted] --> B1[7 pods scheduled, GPUs allocated]
        B1 --> C1[Pod 8 pending: no node available]
        C1 --> D1[56 GPUs held idle, no forward progress]
        E1[Second 8-pod job submitted] --> F1[4 pods scheduled]
        F1 --> G1[4 pods pending: blocked by job 1]
        D1 --> H1[Deadlock: each job holds what the other needs]
        G1 --> H1
    end

    subgraph "With gang scheduling / minMember=8"
        A2[8-pod training job submitted] --> B2{8 free GPU nodes available?}
        B2 -->|No| C2[All 8 pods held in queue, zero GPU capacity consumed]
        B2 -->|Yes| E2[All 8 pods admitted and scheduled together]
        E2 --> F2[Training runs, GPUs fully utilized]
        C2 --> G2[Second job queues behind first, no deadlock possible]
    end

Gang scheduling requires all pods in a group to be admitted together or not at all. The all-or-nothing constraint prevents partial allocations from blocking other jobs.

PodGroup with minMember

Volcano enforces the gang constraint via PodGroup.minMember. A PodGroup object describes the minimum number of pods that must schedule together; the scheduler will not place any pod in the group until the full minimum can be satisfied simultaneously:

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: training-job-group
  namespace: team-a
spec:
  minMember: 8
  queue: default
  priorityClassName: high-priority

Pods reference the group with the scheduling.volcano.sh/pod-group annotation. When a Job creates 8 pods and all reference the same PodGroup with minMember: 8, Volcano holds all 8 in its scheduling queue until it can place all 8 simultaneously. If the cluster has only 7 free GPU nodes, nothing gets allocated.

When you need gang scheduling

Gang scheduling is necessary for distributed training (PyTorch DDP, DeepSpeed, Megatron-LM) and multi-node inference serving where all replicas must be running before the first request can be served. Single-pod inference replicas, embarrassingly parallel batch jobs where each pod is fully independent, and data-parallel jobs where partial progress is still useful do not need it. Adding gang scheduling to jobs that do not require it adds scheduling latency without benefit.

The scheduler stack: Kueue, Volcano, KAI, and Grove

What each layer actually does

These four projects are not interchangeable, and treating them as alternatives to one another is the most common architecture mistake in GPU cluster design. They occupy different layers of the scheduling stack:

graph LR
    subgraph "Layer 4: Declarative workload API"
        G[Grove\nPodCliqueSet / PodClique\nStartup ordering, scaling groups]
    end

    subgraph "Layer 3: Gang placement"
        K[KAI Scheduler\nGang scheduling, GPU sharing\nTopology-aware, fair-share queues]
        V[Volcano\nPodGroup gang scheduling\nBatch scheduler]
    end

    subgraph "Layer 2: Quota and admission"
        Q[Kueue\nClusterQueue, LocalQueue\nFair-share, preemption, DRA quota]
    end

    subgraph "Layer 1: Base scheduling"
        S[kube-scheduler\nNode affinity, pod placement]
    end

    G --> K
    Q --> K
    Q --> V
    Q --> S
    K -->|gang-aware placement| S
    V -->|replaces kube-scheduler\nfor batch jobs| S

Layer	Project	What it decides	What it does NOT decide
Declarative workload API	Grove (alpha)	Multi-component startup order, scaling groups, gang constraints	Pod placement, quota enforcement
Gang placement / pod-to-node	KAI Scheduler	Which node each pod goes to, GPU topology, fair-share	Quota enforcement
Gang placement / pod-to-node	Volcano	Which node each pod goes to, PodGroup all-or-nothing	Kueue ClusterQueue quotas
Quota and admission	Kueue	When a job is allowed to create pods, fair-share across teams	Pod placement, gang scheduling
Base scheduling	kube-scheduler	Pod-to-node binding	GPU topology, gang constraints

The critical point: Kueue does not schedule pods to nodes. It controls admission: when a job’s pods are allowed to exist at all. Once Kueue admits a job, pod placement passes to whichever scheduler is downstream (kube-scheduler, KAI, or Volcano).

Layered pattern A: Kueue (quota/admission) + Volcano (gang placement)

Kueue manages ClusterQueues with GPU quotas, fair sharing between teams, and preemption. Volcano handles gang placement. A team submits a Volcano Job; Kueue gates when it runs; Volcano places all pods simultaneously once admitted:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: h100-sxm5
spec:
  nodeLabels:
    nvidia.com/gpu.product: H100-SXM5-80GB
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: training-cluster-queue
spec:
  namespaceSelector:
    matchLabels:
      team: training
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: h100-sxm5
      resources:
      - name: nvidia.com/gpu
        nominalQuota: "32"
      - name: cpu
        nominalQuota: "256"
      - name: memory
        nominalQuota: 1Ti

See the Kueue DRA concepts documentation for ClusterQueue device class quota syntax when running the DRA driver instead of the traditional nvidia.com/gpu resource.

Layered pattern B: KAI Scheduler + Grove (declarative multi-node serving)

KAI Scheduler (open-sourced from NVIDIA’s Run:ai platform as Apache 2.0 on 2025-04-01, now a CNCF Sandbox project, current v0.17.0) is an AI-workload-aware scheduler with gang scheduling, hierarchical fair-share queues, GPU sharing, bin-packing, and topology-aware placement built in.

Grove (repo: ai-dynamo/grove, available within NVIDIA Dynamo, current v0.1.0-alpha.11) is a declarative API for multi-component model serving. A PodCliqueSet describes the full topology of a serving deployment: which roles exist (prefill, decode, router), how many replicas each role needs, which roles scale together, and startup ordering. Grove emits a PodGang object that KAI consumes to enforce the gang constraint:

apiVersion: grove.io/v1alpha1
kind: PodCliqueSet
metadata:
  name: simple1
  labels:
    app: simple1
spec:
  replicas: 1
  template:
    cliques:
      - name: pca
        spec:
          roleName: rolea
          replicas: 3
          podSpec:
            containers:
              - name: pca
                image: nginx:latest
                resources:
                  requests:
                    cpu: 10m
    podCliqueScalingGroups:
      - name: sga
        cliqueNames: [pcb, pcc]

Grove is alpha as of August 2026 (v0.1.0-alpha.11, released 2026-07-03). The API may change before GA and the project has not been hardened across diverse production workloads. Evaluate KAI + Grove for teams building new inference infrastructure who can tolerate early-stage software; do not treat it as production-hardened today.

Multi-tenant GPU pools: training, batch, and inference together

Running training and inference on the same GPU pool creates a conflict between latency goals and throughput goals. Inference jobs need low-latency preemption rights. Training jobs need sustained throughput without interruption. A cluster that gives everything equal priority will evict a training job mid-run when a sudden inference load spike hits, wasting the entire run and the GPUs consumed up to that point.

Kueue’s fair-share model addresses this through Cohorts: multiple ClusterQueues sharing a pool of capacity, with borrowing limits and directional preemption. Training teams borrow from inference capacity when inference is idle; inference preempts borrowed capacity on demand. Configure preemption.reclaimWithinCohort on each ClusterQueue to control the direction and priority of preemption rather than leaving it symmetric.

The quota-model gotcha

Kueue and Volcano maintain independent quota models. This is by design - they are separate projects with separate governance - but it creates a real operational risk in mixed deployments.

If your team runs Kueue for training jobs and Volcano standalone for batch jobs, Volcano jobs are not subject to Kueue’s ClusterQueue quotas. A Volcano batch job can consume GPU capacity that Kueue has reserved for a training queue, and Kueue will see those GPUs as unavailable without knowing why.

The remedy: route all jobs through Kueue admission, even Volcano batch jobs. Let Kueue be the single point of quota enforcement; Volcano then handles only placement. This requires configuring your Volcano Jobs to declare Kueue Workloads and ensuring that Kueue’s admission controller is the gatekeeper for all job creation.

Fragmentation and bin-packing

GPU fragmentation happens when jobs of different sizes run on the same cluster over time. A cluster of 8-GPU nodes that alternates between 1-GPU inference pods and 8-GPU training runs will accumulate nodes with 3, 5, or 7 GPUs allocated: not enough for the training job, too many slots consumed for the inference pod. The default scheduler spreads load (the LeastAllocated scoring strategy), which maximizes fragmentation.

Bin-packing with MostAllocated

kube-scheduler’s NodeResourcesFit plugin supports three scoring strategies. LeastAllocated (default) spreads pods across nodes, maximizing fragmentation. MostAllocated packs pods onto the most-utilized nodes first, consolidating workloads and leaving whole nodes free for large jobs.

A KubeSchedulerConfiguration enabling bin-packing with GPU weighting:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: default-scheduler
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: MostAllocated
        resources:
        - name: nvidia.com/gpu
          weight: 10
        - name: cpu
          weight: 1
        - name: memory
          weight: 1

Weighting GPUs at 10x relative to CPU focuses the bin-packing score on GPU consolidation. Inference pods pack onto fewer nodes; whole nodes stay free for large training jobs.

MIG and time-slicing to reduce stranded capacity

The single largest source of stranded GPU capacity in inference clusters is single-GPU inference pods consuming a whole GPU when they only need a fraction of one. MIG partitions an A100 or H100 into up to seven independent instances with hardware-isolated memory, so seven 1g.10gb MIG instances replace what was previously seven whole-GPU allocations.

For development clusters and trusted tenants that do not need memory isolation, time-slicing allows more than one workload per GPU. The tradeoff is the absence of fault or memory isolation: one tenant crashing may affect GPU state that another tenant is using, and NVIDIA’s documentation explicitly calls this out.

Migrating off the device plugin

The migration from the legacy device plugin to the NVIDIA DRA driver is a workload-by-workload process across isolated node pools, not a cluster-wide flip:

Check version requirements. Kubernetes 1.34.2 or later, NVIDIA GPU Driver 565 or later.
Install the GPU Operator with DRA enabled on a separate node pool. Set dra.enabled=true and devicePlugin.enabled=false. The DRA driver and the device plugin cannot both manage GPUs on the same node.
Create DeviceClass objects for your GPU types (gpu.nvidia.com, mig.nvidia.com) and any CEL constraints your workloads need (memory thresholds, NVLink topology, MIG profile).
Migrate workloads one at a time. Replace resources.limits["nvidia.com/gpu"] with a resourceClaims reference in the Pod spec. Run DRA pods on the DRA node pool; keep device-plugin pods on the legacy pool during migration.
Update Kueue ClusterQueues. With DRA active, Kueue quotas on GPU resources use device class quota rather than the nvidia.com/gpu extended resource. See the Kueue DRA concepts docs for the current ClusterQueue syntax.
Validate autoscaling. Confirm that a pending pod with a ResourceClaim triggers Cluster Autoscaler to provision a matching node. This is the primary functional validation for the migration: if autoscaling is working correctly, DRA’s structured-parameter design is doing its job.
Remove the device plugin from the node pool once all workloads on that pool use ResourceClaims.

Frequently asked questions

Do I still need the NVIDIA device plugin now that DRA is GA?

On Kubernetes 1.34 and later, DRA is the default GPU allocation model and the NVIDIA DRA driver replaces the device plugin for GPU allocation. You install the DRA driver via the GPU Operator with devicePlugin.enabled=false for the GPU node pool. The device plugin still works on clusters that have not migrated, but it is the legacy path and cannot express the structured resource requirements (memory thresholds, topology constraints, MIG profiles) that DRA enables.

What is the difference between Kueue and Volcano? Do I use both?

Kueue handles queueing, quota, and admission: it decides when a job is allowed to create pods, enforces fair-share between teams, and supports preemption. It does not schedule pods to nodes. Volcano is a full batch scheduler that places pods and enforces gang scheduling via PodGroups. They are independent systems with separate quota models. Running Volcano standalone means Kueue’s ClusterQueue quotas do not apply to Volcano jobs. Use both when you want Kueue’s quota model and Volcano’s gang placement, but route all jobs through Kueue admission so quota enforcement stays centralized.

When do I actually need gang scheduling?

When a job only makes progress if all of its pods run simultaneously: distributed training with parameter servers, pipeline-parallel inference where prefill and decode stages must communicate in real time, and multi-node model serving where all replicas must be available before the first request arrives. Without gang scheduling, pods that start hold GPUs while waiting for the rest of the group, and two half-scheduled jobs can deadlock by each holding what the other needs. Single-pod jobs and embarrassingly parallel batch jobs where each pod is fully independent do not need it.

MIG partitions the GPU in hardware: memory paths are isolated, faults are contained to the instance, and up to seven instances can run on a single A100 or H100. This is the right choice for production multi-tenant inference where two tenants must not interfere with each other’s memory or execution. Time-slicing interleaves multiple workloads on one GPU but, as NVIDIA’s documentation states, “there is no memory or fault-isolation between replicas.” Use time-slicing for development environments, bursty workloads, or when all tenants on a GPU are equally trusted.

What is NVIDIA Grove and is it production-ready?

Grove is a declarative Kubernetes API (PodCliqueSet, PodClique, and PodCliqueScalingGroup CRDs) for describing multi-component model serving deployments with startup ordering, scaling groups, and gang scheduling constraints. It emits PodGang objects consumed by a gang-aware scheduler such as KAI. Grove is part of NVIDIA Dynamo. As of August 2026 it is alpha (v0.1.0-alpha.11, released 2026-07-03): the API may change before GA, and the project has not been through broad production hardening. Teams building new inference infrastructure should track it closely but not depend on it for availability-sensitive workloads today.