When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM

Nov 13, 2026

By Fatima Fall

Discuss (0)

AI-Generated Summary

  • GLM-5.3 NVFP4, a quantized mixture-of-experts model with 753 billion parameters and 40 billion activated per token, requires about 464 GB of GPU memory, exceeding the 252 GB HBM3e on a single GB300 GPU.
  • Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve the model through one OpenAI-compatible endpoint using tensor parallelism (TP=2), with each GPU holding and computing roughly half the weights.
  • vLLM launches a distributed service across both Stations, enabling multi-token prediction, FP8 KV-cache storage, and prefix caching to maximize throughput under concurrent load.
  • AIPerf benchmarks on the SPEED-Bench Coding dataset show overall output throughput peaking at 607.8 tokens per second at concurrency 16, then falling to 385.0 tokens per second at concurrency 32 with mean request latency of 453.5 seconds.