Version

Qwen2.5-72B

On this page

A 72B dense chat model served from an AWQ INT4 quantization on a single 80GB GPU per replica: one Standalone engine fed by a ModelCache. The platform side comes in two shapes - an A100 on AKS and an H100 on Nebius - and the ML side is the same manifest for both. The deployment carries no clusterSelector and two replicas, and each pool holds exactly one GPU, so with both platforms applied one replica lands on each. The service then splits traffic between the two GPUs by weight, which the last section uses to compare them. To serve on just one platform, apply one tab and drop replicas to 1.

These manifests mirror the repository’s AKS and Nebius demos. Apply the platform side first, then the ML side.

Platform

# InferenceClass for the A100 shape on AKS. One NVIDIA A100 80GB on a
# Standard_NC24ads_A100_v4. The GPU is declared as a DRA device: the scheduler
# matches a ModelDeployment's nodeSelector against this capacity, then DRA
# binds the physical GPU to the serving pod.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: aks-a100-1x-nc24ads
spec:
  description: "AKS Standard_NC24ads_A100_v4, 1x NVIDIA A100 80GB"
  provisioning:
    provider: AKS
    aks:
      vmSize: Standard_NC24ads_A100_v4
      diskSizeGb: 200
      accelerator:
        type: nvidia-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.0.0" }
    capacity:
      # The H100's usable VRAM as the NVIDIA DRA driver reports it, not the
      # nominal 80GB.
      memory: { value: "81920Mi" }

# AKS InferenceCluster with one A100 node pool in swedencentral. No
# clusterSelector targets it; the ModelDeployment matches on device capacity
# alone, so it lands here or on any other compatible cluster in the fleet.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: aks-a100-single
  labels:
    modelplane.ai/region: swedencentral
spec:
  cluster:
    source: AKS
    aks:
      location: swedencentral
  nodePools:
  - name: gpua100
    className: aks-a100-1x-nc24ads
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

# InferenceClass for the H100 shape on Nebius. One NVIDIA H100 80GB on the
# gpu-h100-sxm platform, with the cuda13.0 driver preset mk8s preinstalls on
# the pool's nodes. The GPU is declared as a DRA device: the scheduler matches
# a ModelDeployment's nodeSelector against this capacity, then DRA binds the
# physical GPU to the serving pod.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: nebius-h100-1x-sxm
spec:
  description: "Nebius gpu-h100-sxm, 1x NVIDIA H100 80GB"
  provisioning:
    provider: Nebius
    nebius:
      platform: gpu-h100-sxm
      preset: 1gpu-16vcpu-200gb
      diskSizeGb: 200
      driversPreset: cuda13.0
      accelerator:
        type: nvidia-h100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Hopper }
      cudaComputeCapability: { version: "9.0.0" }
    capacity:
      # The H100's usable VRAM as the NVIDIA DRA driver reports it, not the
      # nominal 80GB.
      memory: { value: "81559Mi" }

# Nebius InferenceCluster with one H100 node pool that scales from zero: no
# minNodeCount, so the autoscaling floor defaults to nodeCount (0) and the GPU
# node only appears when a deployment's pod goes Pending.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: nebius-h100-single
  labels:
    modelplane.ai/region: eu-north
spec:
  cluster:
    source: Nebius
    nebius: {}
  nodePools:
  - name: gpu-h100
    className: nebius-h100-1x-sxm
    nodeCount: 0
    maxNodeCount: 1

Deployment

# The cache the engine serves from, hydrated once per matched cluster. The AWQ
# quant repo is public, so no authSecret is needed.
apiVersion: modelplane.ai/v1alpha1
kind: ModelCache
metadata:
  name: qwen-72b-awq
  namespace: ml-team
spec:
  source: HuggingFace
  huggingFace:
    repo: Qwen/Qwen2.5-72B-Instruct-AWQ
    sizeGiB: 150

# Qwen2.5-72B Instruct (AWQ INT4) served on a single 80GB GPU per replica by
# vLLM. The model layer is cloud-agnostic: the >=70Gi CEL selector matches the
# A100 class on AKS (81920Mi) and the H100 class on Nebius (81559Mi) alike.
# Replicas are fleet-wide, not per-cluster, and each pool holds exactly one
# such GPU, so with both platforms applied the two replicas land one on the
# A100 and one on the H100. With a single platform applied, drop replicas to 1.
#
# 72B in AWQ INT4 is ~40Gi of weights, leaving room for the KV cache on an
# 80GB GPU. --max-model-len caps the context at 8192 - raise it only as far as
# the leftover VRAM allows.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-72b
  namespace: ml-team
spec:
  replicas: 2
  template:
    spec:
      modelCacheRef:
        name: qwen-72b-awq
      engines:
      - name: qwen-72b
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("70Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=/mnt/models
                - --served-model-name=qwen-72b
                - --max-model-len=8192

# Exposes the qwen-72b deployment's endpoints as a single OpenAI-compatible
# URL. Modelplane labels each composed ModelEndpoint with the deployment name,
# so this selector reaches every replica. Read the public address from
# status.address:
#   kubectl get ms qwen-72b -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen-72b
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen-72b

Compare the A100 and the H100

Replicas are fleet-wide, not per-cluster: the deployment’s replicas: 2 means two complete serving instances, and because each pool holds a single 80GB GPU they land one on the A100 and one on the H100. Modelplane labels each replica’s endpoint with the cluster it runs on, so the service can split traffic between the platforms by weight. This service pairs the deployment label with each cluster label and gives each GPU half of the live traffic behind the same URL:

# Splits the service's traffic between the two platforms. Each entry pairs the
# deployment label with the cluster label Modelplane puts on every endpoint, and
# the weights fix each side's share of requests regardless of how many replicas
# match. 50/50 sends both GPUs the same workload; shift the ratio to favor one.
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen-72b
  namespace: ml-team
spec:
  endpoints:
  - weight: 50
    selector:
      matchLabels:
        modelplane.ai/deployment: qwen-72b
        modelplane.ai/cluster: aks-a100-single
  - weight: 50
    selector:
      matchLabels:
        modelplane.ai/deployment: qwen-72b
        modelplane.ai/cluster: nebius-h100-single

Both GPUs now serve the same workload, so their engine metrics give a direct performance comparison: scrape each replica’s latency and throughput as in Collecting engine metrics and read the two side by side. Weights are relative, so once one platform wins, shift the 50/50 toward it - 80/20, and as far as 100/0 - without touching the deployment.