{"article":{"slug":"when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm","title":"When One GPU Isn't Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM","subtitle":null,"summary":"NVIDIA shows how to serve the frontier-scale GLM-5.3 model locally across two DGX Stations linked by ConnectX-8 RDMA with vLLM, reaching about 607.8 tokens per second.","content_type":"tutorial","language":"en","canonical_url":"https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/","author":{"name":"Fatima Fall","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"NVIDIA Technical Blog","url":"https://developer.nvidia.com/blog/","listing_slug":null,"listing":null},"topics":[{"name":"llms","slug":"llms","url":"https://listedarticles.com/topics/llms"},{"name":"infrastructure","slug":"infrastructure","url":"https://listedarticles.com/topics/infrastructure"},{"name":"hardware","slug":"hardware","url":"https://listedarticles.com/topics/hardware"},{"name":"performance","slug":"performance","url":"https://listedarticles.com/topics/performance"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":4077,"reading_minutes":18,"published_at":"2026-11-13T17:00:00.000Z","added_at":"2026-11-15T01:49:16.302Z","updated_at":"2026-11-15T01:49:16.302Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm","markdown_url":"https://listedarticles.com/articles/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm.md","example":false,"citation":"Fatima Fall, NVIDIA Technical Blog. \"When One GPU Isn't Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM.\" 13 Nov 2026. https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/ (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/"},"body_markdown":"# When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM\n\n![](https://developer-blogs.nvidia.com/wp-content/uploads/2026/11/image3-3-1024x576-png.webp)\n\nNov 13, 2026\n\nBy [Fatima Fall](https://developer.nvidia.com/blog/author/ffall/ \"Posts by Fatima Fall\")\n\n[Discuss (0)](#entry-content-comments)\n\n- [T](https://twitter.com/intent/tweet?text=When+One+GPU+Isn%E2%80%99t+Enough%3A+Serving+GLM-5.3+Across+Two+NVIDIA+DGX+Stations+with+vLLM+%7C+NVIDIA+Technical+Blog+https%3A%2F%2Fdeveloper.nvidia.com%2Fblog%2Fwhen-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm%2F)\n- [R](https://www.reddit.com/submit?url=https%3A%2F%2Fdeveloper.nvidia.com%2Fblog%2Fwhen-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm%2F&title=When+One+GPU+Isn%E2%80%99t+Enough%3A+Serving+GLM-5.3+Across+Two+NVIDIA+DGX+Stations+with+vLLM+%7C+NVIDIA+Technical+Blog)\n- [E](mailto:?subject=I'd like to share a link with you&body=https%3A%2F%2Fdeveloper.nvidia.com%2Fblog%2Fwhen-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm%2F)\n\n## AI-Generated Summary\n\n- [GLM-5.3 NVFP4](https://huggingface.co/nvidia/GLM-5.3-NVFP4), a quantized mixture-of-experts model with 753 billion parameters and 40 billion activated per token, requires about 464 GB of GPU memory, exceeding the 252 GB HBM3e on a single GB300 GPU.\n- Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve the model through one OpenAI-compatible endpoint using tensor parallelism (TP=2), with each GPU holding and computing roughly half the weights.\n- vLLM launches a distributed service across both Stations, enabling multi-token prediction, FP8 KV-cache storage, and prefix caching to maximize throughput under concurrent load.\n- AIPerf benchmarks on the SPEED-Bench Coding dataset show overall output throughput peaking at 607.8 tokens per second at concurrency 16, then falling to 385.0 tokens per second at concurrency 32 with mean request latency of 453.5 seconds.\n\n### Next Steps\n\n- Follow the [Connect Two DGX Stations playbook](https://build.nvidia.com/station/connect-two-stations) to link and validate the hardware before deployment.\n- Review the [NVIDIA Local AI recipe repository](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-station-2gpu/index.html) for the latest recommended configurations.\n- Connect an application using the [vLLM serving documentation](https://docs.vllm.ai/en/v0.30.0/serving/online_serving/) and rerun AIPerf with custom prompts to find the optimal concurrency for the workload.\n\nPowered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. [Learn more](https://www.nvidia.com/en-us/agreements/trustworthy-ai/terms/)\n\nRunning a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve one model through a single endpoint, with tensor parallelism splitting the model weights and computation across both GPUs.\n\nThe NVIDIA [GLM-5.3 NVFP4](https://huggingface.co/nvidia/GLM-5.3-NVFP4) model, a quantized version of [Z.ai](http://z.ai)’s mixture-of-experts (MoE) model, has 753 billion parameters, with 40 billion activated per token. Sparse activation reduces computation, but all model weights still need to remain available because each token can be routed to different experts. At about 464 GB, the checkpoint exceeds the 252 GB of HBM3e on a single GB300 GPU. This tutorial uses two DGX Stations to keep the weights in GPU memory, with CPU weight offload disabled.\n\nBefore starting, connect the two Stations with the [Connect Two DGX Stations playbook](https://build.nvidia.com/station/connect-two-stations).\n\nThis post shows how to serve GLM-5.3 locally across two NVIDIA DGX Stations with vLLM, send a request, and measure behavior under concurrent load. Tensor parallelism (TP=2) splits the model’s tensors across the two GB300 GPUs. Both Stations compute every request, and Station A returns the answer through one OpenAI-compatible API. By the end, you’ll have a working endpoint and a baseline for how it behaves under load.\n\n*Figure 1. Two Stations serve one model: each GPU holds and computes part of the model over the selected RDMA connection. Their memories remain distinct*\n\n## Prerequisites\n\n- Two GB300 DGX Stations running Linux, connected and validated with the [Connect Two DGX Stations playbook](https://build.nvidia.com/station/connect-two-stations), including its GPU-memory RDMA test on both rails.\n- SSH and sudo access on both Stations with the same Linux username. Docker and the NVIDIA Container Toolkit come preinstalled with the [DGX Station software stack](https://docs.nvidia.com/dgx/dgx-station-development-guide/porting/software-requirements.html). Allow outbound access to the container registry, Hugging Face, and package repositories.\n- At least 464 GB of free disk space on each Station for its copy of the checkpoint, plus room for the vLLM container image and temporary files.\n- Acceptance of the NVIDIA Open Model Agreement and the original GLM-5.3 license terms, listed on the [model card](https://huggingface.co/nvidia/GLM-5.3-NVFP4#licenseterms-of-use). SPEED-Bench is governed by the [NVIDIA Evaluation Dataset License Agreement](https://huggingface.co/datasets/nvidia/SPEED-Bench/blob/main/License.pdf).\n- A Hugging Face account and read access token, with access to the gated [cais/hle](https://huggingface.co/datasets/cais/hle) dataset: sign in, open its page, and select **Agree and access repository**. Step 5 downloads SPEED-Bench prompts from this dataset.\n\nThe companion video follows the same workflow.\n\n*Video 1. Walkthrough of preparation, inference, benchmarking, and shutdown*\n\n## Step 1. Prepare the two Stations\n\nThis tutorial uses a separate computer, such as your laptop, to manage both Stations. Start by opening an SSH session to each Station from that computer. Alternatively, work from a local terminal on each Station. Run every command in this step on both Stations unless it starts with “On A.”\n\nThe two Stations run the same container and checkpoint in different roles: Station A (rank 0) hosts the API, and Station B (rank 1) runs only a worker, with no API server.\n\nFirst, check Docker access and collect the network values you’ll use later:\n\n```\ndocker version\nibdev2netdev\nip -br addr\n```\n\n| **Note:** If the `docker version` output has no `Server` section, the client cannot reach the Docker daemon. If the error is `permission denied`, follow the [Docker post-installation procedure](https://docs.docker.com/engine/install/linux-postinstall/): add your user to the `docker` group with `sudo usermod -aG docker \"$USER\"`, run `exit`, then reconnect and rerun `docker version`. The `docker` group grants root-level privileges. |\n| --- |\n\nThe `ibdev2netdev` output lists the two direct rails, `cx8r0` and `cx8r1`. Pick one for serving traffic, for example `cx8r1`, and note two things for it on each Station:\n\n- **Its RDMA device**, on the `ibdev2netdev` line that ends with the rail’s name. In `mlx5_1 port 1 ==> cx8r1 (Up)`, the device is `mlx5_1`.\n- **Its IPv4 address**, on the `ip -br addr` line that starts with the rail’s name. In `cx8r1 UP 192.168.101.1/24`, the address is `192.168.101.1`.\n\nThe rail name and device are the same on both Stations; only the IPv4 address differs.\n\nNext, create `~/glm53.env` on each Station. This file stores the values that later commands use:\n\n```\nnano ~/glm53.env\n```\n\nPaste the following block and replace each `<...>` placeholder, keeping the quotes, with the values you noted.\n\n```\nRANK=\"<NODE_RANK>\"                    # RANK=\"0\" on Station A, RANK=\"1\" on Station B \nHEAD_IP=\"<STATION_A_IPV4_ADDRESS>\"    # IPv4 address you noted on Station A; same in both files\nWORKER_IP=\"<STATION_B_IPV4_ADDRESS>\"  # IPv4 address you noted on Station B; same in both files\nRAIL_IFACE=\"<RAIL_NAME>\"              # Rail you picked, for example cx8r1; same in both files\nRAIL_HCA=\"<RDMA_DEVICE>\"              # RDMA device you noted, for example mlx5_1; same in both files\n\nIMG=vllm/vllm-openai:v0.30.0-aarch64\nIMG+=@sha256:4864d46625cbc3307623e29ac742030655e27249feba7b97ec925ce4cc4dfb56\nMODEL_REPO=nvidia/GLM-5.3-NVFP4\nMODEL_REV=e3b8f9f2fad29727dfa95e766c5cac18ea1414bf\nMODEL_HOST=\"$HOME/models/GLM-5.3-NVFP4\"\nCONTAINER=glm53-full\nPORT=8000\n\nif [ \"$RANK\" = 0 ]; then\n  NODE_IP=$HEAD_IP; PEER_IP=$WORKER_IP\nelse\n  NODE_IP=$WORKER_IP; PEER_IP=$HEAD_IP\nfi\n\nGPU_UUID=$(nvidia-smi --query-gpu=uuid,name --format=csv,noheader |\n  awk -F, '$2 ~ /GB300/ {gsub(/ /,\"\",$1); print $1; exit}')\n```\n\nFor example, with the default addresses on `cx8r1`, Station A’s file starts like this. Station B’s file is identical except for `RANK=\"1\"`:\n\n```\nRANK=\"0\"\nHEAD_IP=\"192.168.101.1\"\nWORKER_IP=\"192.168.101.2\"\nRAIL_IFACE=\"cx8r1\"\nRAIL_HCA=\"mlx5_1\"\n```\n\n`MODEL_HOST` is built from your home directory, and Step 2 copies the checkpoint over SSH to the same path on Station B.\n\n| **Note:** Later steps start with `source ~/glm53.env` to load these values. **Run this command again whenever you open a new SSH session.** |\n| --- |\n\nOn both Stations, load the values and check storage, connectivity, and baseline memory:\n\n```\nsource ~/glm53.env\nmkdir -p \"$MODEL_HOST\" && df -h \"$MODEL_HOST\"\nethtool \"$RAIL_IFACE\"\nip link show dev \"$RAIL_IFACE\"\nping -I \"$RAIL_IFACE\" -M do -s 8972 -c 3 \"$PEER_IP\"\ntimedatectl show -p NTPSynchronized --value\nnvidia-smi\n```\n\nThese checks catch cabling and addressing mistakes before they surface as NCCL timeouts. Expect a `400000Mb/s` link, MTU 9000, successful jumbo-frame pings, and synchronized time. `nvidia-smi` should show no compute processes on the GB300. On a Station with a display GPU, Xorg can appear on both GPUs with a few MiB.\n\n| **Tip:** Note the GPU memory use shown by `nvidia-smi`; Step 4 compares it with the loaded model. |\n| --- |\n\nStep 2 copies the checkpoint from A to B over SSH, so A needs passwordless access to B. On A, create a key without a passphrase (see [SSH keys](https://ubuntu.com/server/docs/how-to/security/openssh-server/#ssh-keys) for details), copy its public key to B after checking B’s host fingerprint, and confirm the login:\n\n```\nssh-keygen -t ed25519\nssh-copy-id \"$WORKER_IP\"\nssh -o BatchMode=yes \"$WORKER_IP\" hostname\n```\n\n## Step 2. Download and copy the model\n\nTensor parallelism requires both ranks to run the same software against the same weights. Start by pulling the pinned image on both Stations:\n\n```\ndocker pull \"$IMG\" &&\ndocker image inspect \"$IMG\" --format '{{.Architecture}} {{.RepoDigests}}'\n```\n\nThe pull can take several minutes. Matching arm64 digests mean both ranks run identical software.\n\nOn A, download the pinned checkpoint. At about 464 GB, it takes roughly an hour at 1 Gb/s, or about six minutes at 10 Gb/s:\n\n```\nmkdir -p \"$MODEL_HOST\"\ndocker run --rm --network host -v \"$MODEL_HOST:/model\" \\\n  --entrypoint bash \"$IMG\" -lc \\\n  \"exec hf download '$MODEL_REPO' --revision '$MODEL_REV' --local-dir /model\"\n```\n\nWhen `hf download` prints `Downloaded`, copy the files to B. On A:\n\n```\nssh \"$WORKER_IP\" \"mkdir -p '$MODEL_HOST'\" &&\nrsync -aH --info=progress2 --exclude '.cache/' \\\n  \"$MODEL_HOST/\" \"$WORKER_IP:$MODEL_HOST/\"\n```\n\nEach disk keeps the full checkpoint; vLLM partitions the weights across the GPUs at load time, with no further quantization.\n\nOn A, compare both copies (the check reads B’s file list over SSH), then validate the checkpoint:\n\n```\n(\n  set -euo pipefail\n  L=$(find \"$MODEL_HOST\" -path '*/.cache' -prune -o \\\n    -type f -printf '%P %s\\n' | LC_ALL=C sort | sha256sum)\n  R=$(ssh \"$WORKER_IP\" \"set -o pipefail; find '$MODEL_HOST' \\\n    -path '*/.cache' -prune -o -type f -printf '%P %s\\\\n' | \\\n    LC_ALL=C sort | sha256sum\")\n  test \"$L\" = \"$R\"\n  test \"$(find \"$MODEL_HOST\" -maxdepth 1 \\\n    -name 'model-*-of-00050.safetensors' | wc -l)\" = 50\n  python3 -c \"import json; c=json.load(open('$MODEL_HOST/config.json')); \\\nassert c['architectures']==['GlmMoeDsaForCausalLM']; \\\nassert c['quantization_config']['quant_algo']=='MIXED_PRECISION'\"\n  echo MODEL_READY\n)\n```\n\n`MODEL_READY` means both Stations hold the same file names and sizes in the expected layout. `MIXED_PRECISION` is expected: NVFP4 quantization targets MoE-block linear weights and activations, not every tensor.\n\n## Step 3. Launch the distributed service\n\nThis step turns two machines into one service: the same launch script runs on both Stations, reads `~/glm53.env`, and starts one vLLM rank on each. Two settings matter most if you adapt it: `--max-model-len 98304`, the context ceiling in tokens, which covers input plus output, and `--gpu-memory-utilization 0.93`, which gives vLLM 93% of GPU memory. If you change other flags, change one at a time and test again.\n\nFor the latest recommended configurations, see [NVIDIA’s Local AI recipe repository](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-station-2gpu/index.html).\n\n| **Security note:** `--host 0.0.0.0` with `--network host` exposes the unauthenticated API on every Station A interface, including management. Firewall the port to intended clients, and add authentication and TLS before wider access. |\n| --- |\n\nOn both Stations, create a file named `~/launch-glm53.sh`:\n\n```\nnano ~/launch-glm53.sh\n```\n\nThe script enables multi-token prediction (MTP), a form of speculative decoding: the model’s MTP layer drafts four tokens per step, and vLLM verifies them with the full model before accepting them, so each step can produce several tokens. The gain depends on how often drafts are accepted, which varies with the prompt.\n\nFP8 KV-cache storage halves cache memory compared with 16-bit storage, leaving room for more tokens. Prefix caching is enabled to avoid reprocessing prompt prefixes the server has already seen. Paste the script below; it checks your configuration, finds the RoCE GID index for the selected rail, and starts this Station’s vLLM rank in a container:\n\n```\nset -euo pipefail\nsource \"$HOME/glm53.env\"\ncase \"$RANK\" in 0|1) ;; *) echo \"Invalid RANK\" >&2; exit 1;; esac\ntest -n \"$GPU_UUID\"\ntest -f \"$MODEL_HOST/config.json\"\nGID_INDEX=$(/usr/sbin/show_gids 2>/dev/null |\n  awk -v d=\"$RAIL_HCA\" -v ip=\"$NODE_IP\" \\\n  '$1==d && $5==ip && $6==\"v2\" {g=$3} END {if(g!=\"\") print g}')\ntest -n \"$GID_INDEX\"\nHEADLESS=\nif [ \"$RANK\" = 1 ]; then HEADLESS=--headless; fi\nSPEC_CONFIG='{\"method\":\"mtp\",\"num_speculative_tokens\":4}'\nKERNEL_CONFIG='{\"enable_flashinfer_autotune\":false,\"moe_backend\":\"flashinfer_cutlass\"}'\nCOMPILE_CONFIG='{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[5,10,20,40,80],\"pass_config\":{\"fuse_allreduce_rms\":false}}'\n\ndocker run -d --name \"$CONTAINER\" \\\n  --gpus device=\"$GPU_UUID\" --network host --ipc host --shm-size 32g \\\n  --device /dev/infiniband --cap-add IPC_LOCK \\\n  --ulimit memlock=-1 --ulimit stack=67108864 \\\n  -v \"$MODEL_HOST:/model:ro\" \\\n  -e RANK=\"$RANK\" -e HEADLESS=\"$HEADLESS\" \\\n  -e MASTER_ADDR=\"$HEAD_IP\" -e MODEL_REPO=\"$MODEL_REPO\" -e PORT=\"$PORT\" \\\n  -e SPEC_CONFIG=\"$SPEC_CONFIG\" -e KERNEL_CONFIG=\"$KERNEL_CONFIG\" \\\n  -e COMPILE_CONFIG=\"$COMPILE_CONFIG\" \\\n  -e NCCL_CUMEM_ENABLE=1 -e NCCL_IB_HCA=\"$RAIL_HCA\" \\\n  -e NCCL_IB_GID_INDEX=\"$GID_INDEX\" \\\n  -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 \\\n  -e VLLM_ALLREDUCE_USE_FLASHINFER=0 -e VLLM_ALLREDUCE_USE_SYMM_MEM=0 \\\n  -e NCCL_SOCKET_IFNAME=\"$RAIL_IFACE\" -e GLOO_SOCKET_IFNAME=\"$RAIL_IFACE\" \\\n  -e NCCL_NET_GDR_LEVEL=5 -e NCCL_DEBUG=INFO -e NCCL_DEBUG_SUBSYS=INIT,NET \\\n  -e NCCL_DEBUG_FILE='/tmp/nccl_%h_%p.log' -e VLLM_HOST_IP=\"$NODE_IP\" \\\n  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \\\n  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \\\n  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e FLASHINFER_DISABLE_VERSION_CHECK=1 \\\n  --entrypoint bash \"$IMG\" -lc '\n    set -euo pipefail  \n    # Use NCCL 2.30.7 from the image: older versions can deadlock\n    # cross-node CUDA graph replays over RoCE (vLLM issue #52504).\n    export VLLM_NCCL_SO_PATH=$(python3 -c \"import nvidia.nccl,os;\nprint(os.path.join(list(nvidia.nccl.__path__)[0], \\\"lib\\\", \\\"libnccl.so.2\\\"))\")\n    test -f \"$VLLM_NCCL_SO_PATH\"\n    EXTRA=()\n    if [ -n \"$HEADLESS\" ]; then EXTRA+=(\"$HEADLESS\"); fi\n    exec vllm serve /model --served-model-name \"$MODEL_REPO\" \\\n      --host 0.0.0.0 --port \"$PORT\" \\\n      --nnodes 2 --node-rank \"$RANK\" --master-addr \"$MASTER_ADDR\" \\\n      --master-port 29501 --tensor-parallel-size 2 \\\n      --speculative-config \"$SPEC_CONFIG\" \\\n      --max-model-len 98304  --max-num-batched-tokens 8192 \\\n      --block-size 128 --max-num-seqs 16 \\\n      --trust-remote-code --disable-custom-all-reduce \\\n      --kv-cache-dtype fp8 --enable-prefix-caching \\\n      --gpu-memory-utilization 0.93 \\\n      --enable-prompt-tokens-details \\\n      --kernel-config \"$KERNEL_CONFIG\" --compilation-config \"$COMPILE_CONFIG\" \\\n      --distributed-timeout-seconds 1800 \\\n      --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 \\\n      \"${EXTRA[@]}\"\n  '\n```\n\nRun the script on B first, then on A:\n\n```\nbash ~/launch-glm53.sh\n```\n\nEach command prints a container ID and returns immediately; the service is not ready yet. B’s worker waits for A to join, then each Station loads its share of the weights from local storage, prepares kernels, and captures CUDA graphs. This takes several minutes.\n\nOn A, follow the log to see when the service is ready (`--since` hides output from earlier runs):\n\n```\ndocker logs --follow --since \"$(docker inspect \\\n  --format '{{.State.StartedAt}}' \"$CONTAINER\")\" \"$CONTAINER\"\n```\n\nWhen the log shows `Application startup complete`, press Ctrl+C to stop following it; the service keeps running.\n\nAt this point one model spans two machines. Neither GPU holds a complete copy, and neither can answer a request alone: every token needs both ranks to exchange partial results over the RoCE rail.\n\nThe remaining GPU memory holds the KV cache, which stores the context of in-flight requests. The startup log reports its [capacity](https://docs.vllm.ai/en/v0.30.0/serving/parallelism_scaling/) (GPU KV cache size). All requests in flight share it, so longer prompts or more users need more of it. If you run out, lower `--max-model-len` or send fewer requests at once.\n\n## Step 4. Verify serving and send a request\n\nOn A, check the API (`127.0.0.1` is A itself):\n\n```\ndocker exec \"$CONTAINER\" curl -fsS --max-time 10 \\\n  \"http://127.0.0.1:$PORT/health\" >/dev/null && echo HEALTH_OK\n```\n\n`HEALTH_OK` proves that Station A answers. Next, on both Stations, check that the ranks communicate over GPU RDMA rather than falling back to TCP sockets. The check reads the NCCL logs that the launcher writes through `NCCL_DEBUG=INFO`, and the same block shows the loaded memory:\n\n```\ndocker exec \"$CONTAINER\" bash -lc '\n  set -euo pipefail\n  files=(/tmp/nccl_*.log)\n  test -e \"${files[0]}\"\n  grep -hE \"NET/IB.*GDRDMA|GPU Direct RDMA Enabled\" \"${files[@]}\" >/dev/null\n  grep -hF \"$NCCL_IB_HCA\" \"${files[@]}\" >/dev/null\n  if grep -hE \"NET/Socket|NCCL WARN|NCCL ERROR\" \"${files[@]}\"; then\n    exit 1\n  fi\n  echo RDMA_CHECK_OK\n'\nnvidia-smi\n```\n\n`RDMA_CHECK_OK` means the check passed with no warnings. In `nvidia-smi`, A shows the `VLLM::Worker_TP0` process and B shows `VLLM::Worker_TP1`, one rank per GPU. The startup log reports about 218 GiB of weights per GPU, roughly half the checkpoint, which confirms that the model is split across the two GPUs.\n\n| Measurement | Station A | Station B |\n| --- | --- | --- |\n| GB300 GPU used (GiB) | 0.01 → 231.35 | 0.00 → 231.34 |\n\n*Table 1. GPU memory before and after loading the model, with no requests running. Values include weights, KV cache, and runtime buffers*\n\nOn A, send a chat request through the OpenAI-compatible API:\n\n```\ndocker exec \"$CONTAINER\" curl -fsS --max-time 120 \\\n  \"http://127.0.0.1:$PORT/v1/chat/completions\" \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"model\":\"nvidia/GLM-5.3-NVFP4\",\n    \"messages\":[{\"role\":\"user\",\"content\":\"Quarterly units sold: Q1 = 120; Q2 = 180; Q3 = 150. Summarize the trend in one sentence and calculate the percentage change from Q1 to Q2.\"}],\n    \"temperature\":0,\"max_tokens\":4096,\n    \"chat_template_kwargs\":{\"reasoning_effort\":\"low\"}}' |\npython3 -c 'import json,sys; c=json.load(sys.stdin)[\"choices\"][0];\nprint(c[\"message\"][\"content\"]); print(\"finish_reason:\",c[\"finish_reason\"])'\n```\n\nThe response should look like this:\n\n```\n**Trend:** Units sold rose sharply in Q2 before partially declining in Q3, remaining above the Q1 level.\n\n**Q1 → Q2 percentage change:** (180 - 120) / 120 × 100 = **50% increase**\nfinish_reason: stop\n```\n\n`finish_reason: stop` means the model finished its answer instead of hitting the token limit. The request sets `reasoning_effort` to `low` in `chat_template_kwargs`. GLM-5.3 also accepts `high` and `max`, which let it reason longer before answering, at the cost of latency. Because the launcher enables tool calling (`--enable-auto-tool-choice`), agent frameworks that use the OpenAI tools API can call the same endpoint.\n\n## Step 5. Concurrency and release resources\n\n[AIPerf](https://docs.nvidia.com/aiperf/latest/welcome-to-ai-perf-documentation), the NVIDIA benchmarking tool for inference endpoints, measures throughput and latency. Installing it inside the serving container can change vLLM’s dependencies, so on A, install it in its own virtual environment:\n\n```\nsudo apt update &&\nsudo apt install -y python3.12-venv python3.12-dev build-essential &&\npython3.12 -m venv \"$HOME/.venvs/aiperf-0.12.0\" &&\n\"$HOME/.venvs/aiperf-0.12.0/bin/pip\" install 'aiperf==0.12.0' &&\n\"$HOME/.venvs/aiperf-0.12.0/bin/pip\" check\n```\n\nThe install should end with `No broken requirements found`.\n\nAIPerf can send synthetic prompts or replay an existing dataset. This tutorial uses the 80 Coding conversations from SPEED-Bench, a set of real prompts built to evaluate speculative decoding, so MTP is measured on realistic text. SPEED-Bench’s preparation script fetches these prompts from several Hugging Face datasets, including the gated [cais/hle](https://huggingface.co/datasets/cais/hle): if you haven’t already, open its page and select **Agree and access repository** before you continue. Then, on A, log in to Hugging Face:\n\n```\n\"$HOME/.venvs/aiperf-0.12.0/bin/hf\" auth login\n```\n\nNext, download and prepare the dataset. See the [official AIPerf guide](https://docs.nvidia.com/aiperf/v0.12.0/tutorials/datasets-inputs/profile-with-speed-bench-dataset#prepare-the-dataset) for details:\n\n```\nSPEED_BENCH_DIR=\"$HOME/datasets/speed-bench\"\nmkdir -p \"$SPEED_BENCH_DIR\"\ncurl -fLsS https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py | \\\n  \"$HOME/.venvs/aiperf-0.12.0/bin/python\" - --config qualitative --output_dir \"$SPEED_BENCH_DIR\"\n```\n\nOn A, check the dataset and service. Continue when the check prints `BENCHMARK_READY`:\n\n```\n(\n  set -euo pipefail\n  source ~/glm53.env\n  curl -fsS \"http://$HEAD_IP:$PORT/health\" >/dev/null\n  \"$HOME/.venvs/aiperf-0.12.0/bin/python\" - <<'PY_CHECK'\nimport json\nfrom pathlib import Path\nroot = Path.home()\npath = root / \"datasets/speed-bench/qualitative.jsonl\"\nassert path.exists(), \"Dataset missing: rerun the SPEED-Bench preparation\"\nrows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]\ncoding = [row for row in rows if row[\"category\"] == \"coding\"]\nassert len(coding) == 80, \"Expected 80 Coding conversations\"\nassert sum(len(row[\"messages\"]) for row in coding) == 89\nassert all(\"FULL BENCHMARK DATA SHOULD BE FETCHED\" not in message[\"content\"]\n           for row in coding for message in row[\"messages\"])\nprint(\"BENCHMARK_READY: dataset has 80 Coding conversations, 89 turns in total.\")\n\nPY_CHECK\n)\n```\n\nThe full sweep covers concurrency 1, 8, 16, and 32. At each level, AIPerf sends 3×C warmup requests, then measures 10×C requests, where C is the concurrency. Some conversations are sent more than once at higher concurrency.\n\nBenchmark requests use the model’s default reasoning effort (max) and an 8,192-token output cap; responses can finish before reaching it. Full sweeps can take several hours.\n\n A two-hour request timeout leaves room for long responses, `--server-metrics` collects server counters used to measure MTP acceptance, and `--export-level raw` keeps each response for inspection.\n\nFor a quick test, run the block below on A with both containers running and no other API traffic. It tests C1 and C8 with prefix caching enabled, as in the reference measurements in Table 2, below.\n\n```\n(\n  set -euo pipefail\n  source ~/glm53.env\n  CODING_DATASET=\"$HOME/datasets/speed-bench/qualitative.jsonl\"\n  test -f \"$CODING_DATASET\"\n export CONCURRENCIES=${CONCURRENCIES:-\"1 8\"} # Quick run: C1 and C8.\n  # Optional full benchmark: follow the reproduction instructions below.\n  # Step 3: --max-num-seqs 16 -> export CONCURRENCIES=\"1 8 16\"\n  # Step 3: --max-num-seqs 32 -> export CONCURRENCIES=32\n\n  RUN_DIR=\"$HOME/aiperf-results/blog-coding-$(date +%Y%m%d-%H%M%S)\"\n  for C in $CONCURRENCIES;  do\n    ART=\"$RUN_DIR/c$C\"\n    env -u HF_HUB_OFFLINE -u TRANSFORMERS_OFFLINE \\\n      \"$HOME/.venvs/aiperf-0.12.0/bin/aiperf\" profile \\\n      -m \"$MODEL_REPO\" --url \"http://$HEAD_IP:$PORT\" \\\n      --endpoint-type chat --streaming --tokenizer \"$MODEL_HOST\" \\\n      --custom-dataset-type speed_bench_coding --input-file \"$CODING_DATASET\" \\\n      --request-count \"$((10 * C))\" --warmup-request-count \"$((3 * C))\" \\\n      --osl 8192 \\\n      --use-server-token-count --concurrency \"$C\" --random-seed 42 \\\n      --request-timeout-seconds 7200 \\\n      --server-metrics \"http://$HEAD_IP:$PORT/metrics\" \\\n      --export-level raw --artifact-dir \"$ART\"\n    printf \"Results: %s\\n\" \"$ART\"\n  done\n  printf \"Results directory: %s\\n\" \"$RUN_DIR\"\n)\n```\n\nAIPerf saves metrics, raw responses, and logs in the results directory. Warmup requests are excluded from the reported metrics.\n\n**Optional: Reproduce Table 2**. In `~/launch-glm53.sh` from Step 3 on both Stations, keep `--enable-prefix-caching` and set `--max-num-seqs` to 16 or 32 for the corresponding group shown in the comments above. Before each group, stop and remove the current containers on A, then B (`docker stop --timeout 60 \"$CONTAINER\"`, followed by `docker rm \"$CONTAINER\"`). Relaunch the updated script on B, then A with bash `~/launch-glm53.sh` and wait for `Application startup complete`. On A, run the matching `export CONCURRENCIES` command from the comments, then rerun the AIPerf block.\n\nTo test without prefix caching, replace `--enable-prefix-caching` with `--no-enable-prefix-caching` in the Step 3 script on both Stations, then recreate the containers as above. Performance may differ from Table 2.\n\nFigure 2 compares overall output throughput across measured requests with mean per-user throughput. Token counts include reasoning.\n\n*Figure 2. Overall output throughput versus mean per-user throughput on SPEED-Bench Coding, with prefix caching enabled. C1, C8, C16, and C32 denote configured concurrency*\n\nTable 2, below, shows how throughput and latency change as more requests share the service.\n\n|  |  |  |  |  |\n| --- | --- | --- | --- | --- |\n| **Configured** **concurrency** | **Overall throughput**  **(tok/s)** | **Mean request latency**  **(s)** | **Mean TTFT** **(ms)** | **Mean inter-token latency** **(ms/token)** |\n| 1 | 107.4 | 49.84 | 479.75 | 8.78 |\n| 8 | 386.0 | 114.17 | 433.79 | 19.01 |\n| 16 | 607.8 | 136.46 | 482.90 | 23.77 |\n| 32 | 385.0 | 453.54 | 899.36 | 77.90 |\n\n*Table 2. SPEED-Bench Coding results with prefix caching enabled, four MTP draft tokens per step, and an FP8 KV cache. Latency metrics are means over measured requests; warmup is excluded. TTFT measures time to the first token, which may be reasoning*\n\nOverall throughput peaks at 607.8 tok/s at C16. It falls to 385.0 tok/s at C32, where mean request latency is 453.5 s. Higher concurrency also reduces mean per-user throughput, from 115.3 tok/s at C1 to 13.0 tok/s at C32.\n\nAfter the requests finish, stop the vLLM container on A, then on B:\n\n```\ndocker stop --timeout 60 \"$CONTAINER\"\ndocker inspect --format \\\n  'Status={{.State.Status}} Running={{.State.Running}}' \"$CONTAINER\"\nnvidia-smi\n```\n\nThe status should read `Status=exited Running=false`, with no vLLM processes in `nvidia-smi`; the checkpoint, containers, and results stay on disk. To resume, run `docker start \"$CONTAINER\"` on B, then A, and repeat the Step 4 checks.\n\n## Troubleshoot common blockers\n\n- Missing variables or SSH failure: `source ~/glm53.env` after reconnecting; verify A-to-B key authentication and B’s fingerprint.\n- SPEED-Bench preparation fails with a gated-dataset error: sign in to Hugging Face with the same account, open the [cais/hle](https://huggingface.co/datasets/cais/hle) page, select **Agree and access repository**, then rerun the preparation.\n- Startup or RDMA check failure: inspect `docker logs \"$CONTAINER\"` on both nodes; verify the checkpoint and the GID, and rerun the playbook’s validation (script 07).\n- [CUDA out of memory](https://docs.vllm.ai/en/v0.30.0/usage/troubleshooting/): lower `--max-model-len` or concurrency, then retest.\n\n## Put the endpoint to work\n\nYou now have a 753-billion-parameter model running behind one local API on two NVIDIA DGX Stations, with every weight in GPU memory. That opens up work that’s hard to do on a hosted endpoint: running inference on sensitive or proprietary data, prototyping agents locally, and iterating on prompts without per-token costs.\n\nConnect your own application using the [vLLM serving documentation](https://docs.vllm.ai/en/v0.30.0/serving/online_serving/), then rerun AIPerf with your prompts to find the concurrency that fits your workload. A frontier-scale model is now ready to build on, right from your desk.\n\n[Discuss (0)](#entry-content-comments)\n\n## Tags\n\n[Agentic AI / Generative AI](https://developer.nvidia.com/blog/category/generative-ai/) | [Edge Computing](https://developer.nvidia.com/blog/category/edge-computing/) | [Networking / Communications](https://developer.nvidia.com/blog/category/networking-communications/) | [General](https://developer.nvidia.com/blog/recent-posts/?industry=General) | [Blackwell](https://developer.nvidia.com/blog/recent-posts/?products=Blackwell) | [DGX](https://developer.nvidia.com/blog/recent-posts/?products=DGX) | [Intermediate Technical](https://developer.nvidia.com/blog/recent-posts/?learning_levels=Intermediate+Technical) | [Tutorial](https://developer.nvidia.com/blog/recent-posts/?content_types=Tutorial) | [DGX Station](https://developer.nvidia.com/blog/tag/dgx-station/) | [Inference Performance](https://developer.nvidia.com/blog/tag/inference-performance/) | [LLM Benchmarking](https://developer.nvidia.com/blog/tag/llm-benchmarking/) | [LLMs](https://developer.nvidia.com/blog/tag/large-language-models/) | [Local AI](https://developer.nvidia.com/blog/tag/local-ai/) | [vLLM](https://developer.nvidia.com/blog/tag/vllm/)\n\n## About the Authors\n\n**About Fatima Fall**\n  \nFatima Fall is a technical marketing engineer with a deep interest in everything related to AI. She studied at Howard University, where she got a B.Sc. in applied mathematics.\n\n[View all posts by Fatima Fall](https://developer.nvidia.com/blog/author/ffall/)\n","body_html":"<h1 id=\"when-one-gpu-isn-t-enough-serving-glm-5-3-across-two-nvidia-dgx-\">When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM</h1>\n<figure><img src=\"https://developer-blogs.nvidia.com/wp-content/uploads/2026/11/image3-3-1024x576-png.webp\" alt=\"\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></figure>\n<p>Nov 13, 2026</p>\n<p>By <a href=\"https://developer.nvidia.com/blog/author/ffall/\" title=\"Posts by Fatima Fall\" rel=\"nofollow ugc noopener\">Fatima Fall</a></p>\n<p><a href=\"#entry-content-comments\">Discuss (0)</a></p>\n<ul><li><a href=\"https://twitter.com/intent/tweet?text=When+One+GPU+Isn%E2%80%99t+Enough%3A+Serving+GLM-5.3+Across+Two+NVIDIA+DGX+Stations+with+vLLM+%7C+NVIDIA+Technical+Blog+https%3A%2F%2Fdeveloper.nvidia.com%2Fblog%2Fwhen-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm%2F\" rel=\"nofollow ugc noopener\">T</a></li><li><a href=\"https://www.reddit.com/submit?url=https%3A%2F%2Fdeveloper.nvidia.com%2Fblog%2Fwhen-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm%2F&amp;title=When+One+GPU+Isn%E2%80%99t+Enough%3A+Serving+GLM-5.3+Across+Two+NVIDIA+DGX+Stations+with+vLLM+%7C+NVIDIA+Technical+Blog\" rel=\"nofollow ugc noopener\">R</a></li><li>E</li></ul>\n<h2 id=\"ai-generated-summary\">AI-Generated Summary</h2>\n<ul><li><a href=\"https://huggingface.co/nvidia/GLM-5.3-NVFP4\" rel=\"nofollow ugc noopener\">GLM-5.3 NVFP4</a>, a quantized mixture-of-experts model with 753 billion parameters and 40 billion activated per token, requires about 464 GB of GPU memory, exceeding the 252 GB HBM3e on a single GB300 GPU.</li><li>Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve the model through one OpenAI-compatible endpoint using tensor parallelism (TP=2), with each GPU holding and computing roughly half the weights.</li><li>vLLM launches a distributed service across both Stations, enabling multi-token prediction, FP8 KV-cache storage, and prefix caching to maximize throughput under concurrent load.</li><li>AIPerf benchmarks on the SPEED-Bench Coding dataset show overall output throughput peaking at 607.8 tokens per second at concurrency 16, then falling to 385.0 tokens per second at concurrency 32 with mean request latency of 453.5 seconds.</li></ul>\n<h3 id=\"next-steps\">Next Steps</h3>\n<ul><li>Follow the <a href=\"https://build.nvidia.com/station/connect-two-stations\" rel=\"nofollow ugc noopener\">Connect Two DGX Stations playbook</a> to link and validate the hardware before deployment.</li><li>Review the <a href=\"https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-station-2gpu/index.html\" rel=\"nofollow ugc noopener\">NVIDIA Local AI recipe repository</a> for the latest recommended configurations.</li><li>Connect an application using the <a href=\"https://docs.vllm.ai/en/v0.30.0/serving/online_serving/\" rel=\"nofollow ugc noopener\">vLLM serving documentation</a> and rerun AIPerf with custom prompts to find the optimal concurrency for the workload.</li></ul>\n<p>Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. <a href=\"https://www.nvidia.com/en-us/agreements/trustworthy-ai/terms/\" rel=\"nofollow ugc noopener\">Learn more</a></p>\n<p>Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve one model through a single endpoint, with tensor parallelism splitting the model weights and computation across both GPUs.</p>\n<p>The NVIDIA <a href=\"https://huggingface.co/nvidia/GLM-5.3-NVFP4\" rel=\"nofollow ugc noopener\">GLM-5.3 NVFP4</a> model, a quantized version of <a href=\"http://z.ai\" rel=\"nofollow ugc noopener\">Z.ai</a>’s mixture-of-experts (MoE) model, has 753 billion parameters, with 40 billion activated per token. Sparse activation reduces computation, but all model weights still need to remain available because each token can be routed to different experts. At about 464 GB, the checkpoint exceeds the 252 GB of HBM3e on a single GB300 GPU. This tutorial uses two DGX Stations to keep the weights in GPU memory, with CPU weight offload disabled.</p>\n<p>Before starting, connect the two Stations with the <a href=\"https://build.nvidia.com/station/connect-two-stations\" rel=\"nofollow ugc noopener\">Connect Two DGX Stations playbook</a>.</p>\n<p>This post shows how to serve GLM-5.3 locally across two NVIDIA DGX Stations with vLLM, send a request, and measure behavior under concurrent load. Tensor parallelism (TP=2) splits the model’s tensors across the two GB300 GPUs. Both Stations compute every request, and Station A returns the answer through one OpenAI-compatible API. By the end, you’ll have a working endpoint and a baseline for how it behaves under load.</p>\n<p><em>Figure 1. Two Stations serve one model: each GPU holds and computes part of the model over the selected RDMA connection. Their memories remain distinct</em></p>\n<h2 id=\"prerequisites\">Prerequisites</h2>\n<ul><li>Two GB300 DGX Stations running Linux, connected and validated with the <a href=\"https://build.nvidia.com/station/connect-two-stations\" rel=\"nofollow ugc noopener\">Connect Two DGX Stations playbook</a>, including its GPU-memory RDMA test on both rails.</li><li>SSH and sudo access on both Stations with the same Linux username. Docker and the NVIDIA Container Toolkit come preinstalled with the <a href=\"https://docs.nvidia.com/dgx/dgx-station-development-guide/porting/software-requirements.html\" rel=\"nofollow ugc noopener\">DGX Station software stack</a>. Allow outbound access to the container registry, Hugging Face, and package repositories.</li><li>At least 464 GB of free disk space on each Station for its copy of the checkpoint, plus room for the vLLM container image and temporary files.</li><li>Acceptance of the NVIDIA Open Model Agreement and the original GLM-5.3 license terms, listed on the <a href=\"https://huggingface.co/nvidia/GLM-5.3-NVFP4#licenseterms-of-use\" rel=\"nofollow ugc noopener\">model card</a>. SPEED-Bench is governed by the <a href=\"https://huggingface.co/datasets/nvidia/SPEED-Bench/blob/main/License.pdf\" rel=\"nofollow ugc noopener\">NVIDIA Evaluation Dataset License Agreement</a>.</li><li>A Hugging Face account and read access token, with access to the gated <a href=\"https://huggingface.co/datasets/cais/hle\" rel=\"nofollow ugc noopener\">cais/hle</a> dataset: sign in, open its page, and select <strong>Agree and access repository</strong>. Step 5 downloads SPEED-Bench prompts from this dataset.</li></ul>\n<p>The companion video follows the same workflow.</p>\n<p><em>Video 1. Walkthrough of preparation, inference, benchmarking, and shutdown</em></p>\n<h2 id=\"step-1-prepare-the-two-stations\">Step 1. Prepare the two Stations</h2>\n<p>This tutorial uses a separate computer, such as your laptop, to manage both Stations. Start by opening an SSH session to each Station from that computer. Alternatively, work from a local terminal on each Station. Run every command in this step on both Stations unless it starts with “On A.”</p>\n<p>The two Stations run the same container and checkpoint in different roles: Station A (rank 0) hosts the API, and Station B (rank 1) runs only a worker, with no API server.</p>\n<p>First, check Docker access and collect the network values you’ll use later:</p>\n<pre><code>docker version\nibdev2netdev\nip -br addr</code></pre>\n<div class=\"table-wrap\"><table><thead><tr><th><strong>Note:</strong> If the <code>docker version</code> output has no <code>Server</code> section, the client cannot reach the Docker daemon. If the error is <code>permission denied</code>, follow the <a href=\"https://docs.docker.com/engine/install/linux-postinstall/\" rel=\"nofollow ugc noopener\">Docker post-installation procedure</a>: add your user to the <code>docker</code> group with <code>sudo usermod -aG docker &quot;$USER&quot;</code>, run <code>exit</code>, then reconnect and rerun <code>docker version</code>. The <code>docker</code> group grants root-level privileges.</th></tr></thead></table></div>\n<p>The <code>ibdev2netdev</code> output lists the two direct rails, <code>cx8r0</code> and <code>cx8r1</code>. Pick one for serving traffic, for example <code>cx8r1</code>, and note two things for it on each Station:</p>\n<ul><li><strong>Its RDMA device</strong>, on the <code>ibdev2netdev</code> line that ends with the rail’s name. In <code>mlx5_1 port 1 ==&gt; cx8r1 (Up)</code>, the device is <code>mlx5_1</code>.</li><li><strong>Its IPv4 address</strong>, on the <code>ip -br addr</code> line that starts with the rail’s name. In <code>cx8r1 UP 192.168.101.1/24</code>, the address is <code>192.168.101.1</code>.</li></ul>\n<p>The rail name and device are the same on both Stations; only the IPv4 address differs.</p>\n<p>Next, create <code>~/glm53.env</code> on each Station. This file stores the values that later commands use:</p>\n<pre><code>nano ~/glm53.env</code></pre>\n<p>Paste the following block and replace each <code>&lt;...&gt;</code> placeholder, keeping the quotes, with the values you noted.</p>\n<pre><code>RANK=&quot;&lt;NODE_RANK&gt;&quot;                    # RANK=&quot;0&quot; on Station A, RANK=&quot;1&quot; on Station B \nHEAD_IP=&quot;&lt;STATION_A_IPV4_ADDRESS&gt;&quot;    # IPv4 address you noted on Station A; same in both files\nWORKER_IP=&quot;&lt;STATION_B_IPV4_ADDRESS&gt;&quot;  # IPv4 address you noted on Station B; same in both files\nRAIL_IFACE=&quot;&lt;RAIL_NAME&gt;&quot;              # Rail you picked, for example cx8r1; same in both files\nRAIL_HCA=&quot;&lt;RDMA_DEVICE&gt;&quot;              # RDMA device you noted, for example mlx5_1; same in both files\n\nIMG=vllm/vllm-openai:v0.30.0-aarch64\nIMG+=@sha256:4864d46625cbc3307623e29ac742030655e27249feba7b97ec925ce4cc4dfb56\nMODEL_REPO=nvidia/GLM-5.3-NVFP4\nMODEL_REV=e3b8f9f2fad29727dfa95e766c5cac18ea1414bf\nMODEL_HOST=&quot;$HOME/models/GLM-5.3-NVFP4&quot;\nCONTAINER=glm53-full\nPORT=8000\n\nif [ &quot;$RANK&quot; = 0 ]; then\n  NODE_IP=$HEAD_IP; PEER_IP=$WORKER_IP\nelse\n  NODE_IP=$WORKER_IP; PEER_IP=$HEAD_IP\nfi\n\nGPU_UUID=$(nvidia-smi --query-gpu=uuid,name --format=csv,noheader |\n  awk -F, &#39;$2 ~ /GB300/ {gsub(/ /,&quot;&quot;,$1); print $1; exit}&#39;)</code></pre>\n<p>For example, with the default addresses on <code>cx8r1</code>, Station A’s file starts like this. Station B’s file is identical except for <code>RANK=&quot;1&quot;</code>:</p>\n<pre><code>RANK=&quot;0&quot;\nHEAD_IP=&quot;192.168.101.1&quot;\nWORKER_IP=&quot;192.168.101.2&quot;\nRAIL_IFACE=&quot;cx8r1&quot;\nRAIL_HCA=&quot;mlx5_1&quot;</code></pre>\n<p><code>MODEL_HOST</code> is built from your home directory, and Step 2 copies the checkpoint over SSH to the same path on Station B.</p>\n<div class=\"table-wrap\"><table><thead><tr><th><strong>Note:</strong> Later steps start with <code>source ~/glm53.env</code> to load these values. <strong>Run this command again whenever you open a new SSH session.</strong></th></tr></thead></table></div>\n<p>On both Stations, load the values and check storage, connectivity, and baseline memory:</p>\n<pre><code>source ~/glm53.env\nmkdir -p &quot;$MODEL_HOST&quot; &amp;&amp; df -h &quot;$MODEL_HOST&quot;\nethtool &quot;$RAIL_IFACE&quot;\nip link show dev &quot;$RAIL_IFACE&quot;\nping -I &quot;$RAIL_IFACE&quot; -M do -s 8972 -c 3 &quot;$PEER_IP&quot;\ntimedatectl show -p NTPSynchronized --value\nnvidia-smi</code></pre>\n<p>These checks catch cabling and addressing mistakes before they surface as NCCL timeouts. Expect a <code>400000Mb/s</code> link, MTU 9000, successful jumbo-frame pings, and synchronized time. <code>nvidia-smi</code> should show no compute processes on the GB300. On a Station with a display GPU, Xorg can appear on both GPUs with a few MiB.</p>\n<div class=\"table-wrap\"><table><thead><tr><th><strong>Tip:</strong> Note the GPU memory use shown by <code>nvidia-smi</code>; Step 4 compares it with the loaded model.</th></tr></thead></table></div>\n<p>Step 2 copies the checkpoint from A to B over SSH, so A needs passwordless access to B. On A, create a key without a passphrase (see <a href=\"https://ubuntu.com/server/docs/how-to/security/openssh-server/#ssh-keys\" rel=\"nofollow ugc noopener\">SSH keys</a> for details), copy its public key to B after checking B’s host fingerprint, and confirm the login:</p>\n<pre><code>ssh-keygen -t ed25519\nssh-copy-id &quot;$WORKER_IP&quot;\nssh -o BatchMode=yes &quot;$WORKER_IP&quot; hostname</code></pre>\n<h2 id=\"step-2-download-and-copy-the-model\">Step 2. Download and copy the model</h2>\n<p>Tensor parallelism requires both ranks to run the same software against the same weights. Start by pulling the pinned image on both Stations:</p>\n<pre><code>docker pull &quot;$IMG&quot; &amp;&amp;\ndocker image inspect &quot;$IMG&quot; --format &#39;{{.Architecture}} {{.RepoDigests}}&#39;</code></pre>\n<p>The pull can take several minutes. Matching arm64 digests mean both ranks run identical software.</p>\n<p>On A, download the pinned checkpoint. At about 464 GB, it takes roughly an hour at 1 Gb/s, or about six minutes at 10 Gb/s:</p>\n<pre><code>mkdir -p &quot;$MODEL_HOST&quot;\ndocker run --rm --network host -v &quot;$MODEL_HOST:/model&quot; \\\n  --entrypoint bash &quot;$IMG&quot; -lc \\\n  &quot;exec hf download &#39;$MODEL_REPO&#39; --revision &#39;$MODEL_REV&#39; --local-dir /model&quot;</code></pre>\n<p>When <code>hf download</code> prints <code>Downloaded</code>, copy the files to B. On A:</p>\n<pre><code>ssh &quot;$WORKER_IP&quot; &quot;mkdir -p &#39;$MODEL_HOST&#39;&quot; &amp;&amp;\nrsync -aH --info=progress2 --exclude &#39;.cache/&#39; \\\n  &quot;$MODEL_HOST/&quot; &quot;$WORKER_IP:$MODEL_HOST/&quot;</code></pre>\n<p>Each disk keeps the full checkpoint; vLLM partitions the weights across the GPUs at load time, with no further quantization.</p>\n<p>On A, compare both copies (the check reads B’s file list over SSH), then validate the checkpoint:</p>\n<pre><code>(\n  set -euo pipefail\n  L=$(find &quot;$MODEL_HOST&quot; -path &#39;*/.cache&#39; -prune -o \\\n    -type f -printf &#39;%P %s\\n&#39; | LC_ALL=C sort | sha256sum)\n  R=$(ssh &quot;$WORKER_IP&quot; &quot;set -o pipefail; find &#39;$MODEL_HOST&#39; \\\n    -path &#39;*/.cache&#39; -prune -o -type f -printf &#39;%P %s\\\\n&#39; | \\\n    LC_ALL=C sort | sha256sum&quot;)\n  test &quot;$L&quot; = &quot;$R&quot;\n  test &quot;$(find &quot;$MODEL_HOST&quot; -maxdepth 1 \\\n    -name &#39;model-*-of-00050.safetensors&#39; | wc -l)&quot; = 50\n  python3 -c &quot;import json; c=json.load(open(&#39;$MODEL_HOST/config.json&#39;)); \\\nassert c[&#39;architectures&#39;]==[&#39;GlmMoeDsaForCausalLM&#39;]; \\\nassert c[&#39;quantization_config&#39;][&#39;quant_algo&#39;]==&#39;MIXED_PRECISION&#39;&quot;\n  echo MODEL_READY\n)</code></pre>\n<p><code>MODEL_READY</code> means both Stations hold the same file names and sizes in the expected layout. <code>MIXED_PRECISION</code> is expected: NVFP4 quantization targets MoE-block linear weights and activations, not every tensor.</p>\n<h2 id=\"step-3-launch-the-distributed-service\">Step 3. Launch the distributed service</h2>\n<p>This step turns two machines into one service: the same launch script runs on both Stations, reads <code>~/glm53.env</code>, and starts one vLLM rank on each. Two settings matter most if you adapt it: <code>--max-model-len 98304</code>, the context ceiling in tokens, which covers input plus output, and <code>--gpu-memory-utilization 0.93</code>, which gives vLLM 93% of GPU memory. If you change other flags, change one at a time and test again.</p>\n<p>For the latest recommended configurations, see <a href=\"https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-station-2gpu/index.html\" rel=\"nofollow ugc noopener\">NVIDIA’s Local AI recipe repository</a>.</p>\n<div class=\"table-wrap\"><table><thead><tr><th><strong>Security note:</strong> <code>--host 0.0.0.0</code> with <code>--network host</code> exposes the unauthenticated API on every Station A interface, including management. Firewall the port to intended clients, and add authentication and TLS before wider access.</th></tr></thead></table></div>\n<p>On both Stations, create a file named <code>~/launch-glm53.sh</code>:</p>\n<pre><code>nano ~/launch-glm53.sh</code></pre>\n<p>The script enables multi-token prediction (MTP), a form of speculative decoding: the model’s MTP layer drafts four tokens per step, and vLLM verifies them with the full model before accepting them, so each step can produce several tokens. The gain depends on how often drafts are accepted, which varies with the prompt.</p>\n<p>FP8 KV-cache storage halves cache memory compared with 16-bit storage, leaving room for more tokens. Prefix caching is enabled to avoid reprocessing prompt prefixes the server has already seen. Paste the script below; it checks your configuration, finds the RoCE GID index for the selected rail, and starts this Station’s vLLM rank in a container:</p>\n<pre><code>set -euo pipefail\nsource &quot;$HOME/glm53.env&quot;\ncase &quot;$RANK&quot; in 0|1) ;; *) echo &quot;Invalid RANK&quot; &gt;&amp;2; exit 1;; esac\ntest -n &quot;$GPU_UUID&quot;\ntest -f &quot;$MODEL_HOST/config.json&quot;\nGID_INDEX=$(/usr/sbin/show_gids 2&gt;/dev/null |\n  awk -v d=&quot;$RAIL_HCA&quot; -v ip=&quot;$NODE_IP&quot; \\\n  &#39;$1==d &amp;&amp; $5==ip &amp;&amp; $6==&quot;v2&quot; {g=$3} END {if(g!=&quot;&quot;) print g}&#39;)\ntest -n &quot;$GID_INDEX&quot;\nHEADLESS=\nif [ &quot;$RANK&quot; = 1 ]; then HEADLESS=--headless; fi\nSPEC_CONFIG=&#39;{&quot;method&quot;:&quot;mtp&quot;,&quot;num_speculative_tokens&quot;:4}&#39;\nKERNEL_CONFIG=&#39;{&quot;enable_flashinfer_autotune&quot;:false,&quot;moe_backend&quot;:&quot;flashinfer_cutlass&quot;}&#39;\nCOMPILE_CONFIG=&#39;{&quot;cudagraph_mode&quot;:&quot;FULL_DECODE_ONLY&quot;,&quot;cudagraph_capture_sizes&quot;:[5,10,20,40,80],&quot;pass_config&quot;:{&quot;fuse_allreduce_rms&quot;:false}}&#39;\n\ndocker run -d --name &quot;$CONTAINER&quot; \\\n  --gpus device=&quot;$GPU_UUID&quot; --network host --ipc host --shm-size 32g \\\n  --device /dev/infiniband --cap-add IPC_LOCK \\\n  --ulimit memlock=-1 --ulimit stack=67108864 \\\n  -v &quot;$MODEL_HOST:/model:ro&quot; \\\n  -e RANK=&quot;$RANK&quot; -e HEADLESS=&quot;$HEADLESS&quot; \\\n  -e MASTER_ADDR=&quot;$HEAD_IP&quot; -e MODEL_REPO=&quot;$MODEL_REPO&quot; -e PORT=&quot;$PORT&quot; \\\n  -e SPEC_CONFIG=&quot;$SPEC_CONFIG&quot; -e KERNEL_CONFIG=&quot;$KERNEL_CONFIG&quot; \\\n  -e COMPILE_CONFIG=&quot;$COMPILE_CONFIG&quot; \\\n  -e NCCL_CUMEM_ENABLE=1 -e NCCL_IB_HCA=&quot;$RAIL_HCA&quot; \\\n  -e NCCL_IB_GID_INDEX=&quot;$GID_INDEX&quot; \\\n  -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 \\\n  -e VLLM_ALLREDUCE_USE_FLASHINFER=0 -e VLLM_ALLREDUCE_USE_SYMM_MEM=0 \\\n  -e NCCL_SOCKET_IFNAME=&quot;$RAIL_IFACE&quot; -e GLOO_SOCKET_IFNAME=&quot;$RAIL_IFACE&quot; \\\n  -e NCCL_NET_GDR_LEVEL=5 -e NCCL_DEBUG=INFO -e NCCL_DEBUG_SUBSYS=INIT,NET \\\n  -e NCCL_DEBUG_FILE=&#39;/tmp/nccl_%h_%p.log&#39; -e VLLM_HOST_IP=&quot;$NODE_IP&quot; \\\n  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \\\n  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \\\n  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e FLASHINFER_DISABLE_VERSION_CHECK=1 \\\n  --entrypoint bash &quot;$IMG&quot; -lc &#39;\n    set -euo pipefail  \n    # Use NCCL 2.30.7 from the image: older versions can deadlock\n    # cross-node CUDA graph replays over RoCE (vLLM issue #52504).\n    export VLLM_NCCL_SO_PATH=$(python3 -c &quot;import nvidia.nccl,os;\nprint(os.path.join(list(nvidia.nccl.__path__)[0], \\&quot;lib\\&quot;, \\&quot;libnccl.so.2\\&quot;))&quot;)\n    test -f &quot;$VLLM_NCCL_SO_PATH&quot;\n    EXTRA=()\n    if [ -n &quot;$HEADLESS&quot; ]; then EXTRA+=(&quot;$HEADLESS&quot;); fi\n    exec vllm serve /model --served-model-name &quot;$MODEL_REPO&quot; \\\n      --host 0.0.0.0 --port &quot;$PORT&quot; \\\n      --nnodes 2 --node-rank &quot;$RANK&quot; --master-addr &quot;$MASTER_ADDR&quot; \\\n      --master-port 29501 --tensor-parallel-size 2 \\\n      --speculative-config &quot;$SPEC_CONFIG&quot; \\\n      --max-model-len 98304  --max-num-batched-tokens 8192 \\\n      --block-size 128 --max-num-seqs 16 \\\n      --trust-remote-code --disable-custom-all-reduce \\\n      --kv-cache-dtype fp8 --enable-prefix-caching \\\n      --gpu-memory-utilization 0.93 \\\n      --enable-prompt-tokens-details \\\n      --kernel-config &quot;$KERNEL_CONFIG&quot; --compilation-config &quot;$COMPILE_CONFIG&quot; \\\n      --distributed-timeout-seconds 1800 \\\n      --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 \\\n      &quot;${EXTRA[@]}&quot;\n  &#39;</code></pre>\n<p>Run the script on B first, then on A:</p>\n<pre><code>bash ~/launch-glm53.sh</code></pre>\n<p>Each command prints a container ID and returns immediately; the service is not ready yet. B’s worker waits for A to join, then each Station loads its share of the weights from local storage, prepares kernels, and captures CUDA graphs. This takes several minutes.</p>\n<p>On A, follow the log to see when the service is ready (<code>--since</code> hides output from earlier runs):</p>\n<pre><code>docker logs --follow --since &quot;$(docker inspect \\\n  --format &#39;{{.State.StartedAt}}&#39; &quot;$CONTAINER&quot;)&quot; &quot;$CONTAINER&quot;</code></pre>\n<p>When the log shows <code>Application startup complete</code>, press Ctrl+C to stop following it; the service keeps running.</p>\n<p>At this point one model spans two machines. Neither GPU holds a complete copy, and neither can answer a request alone: every token needs both ranks to exchange partial results over the RoCE rail.</p>\n<p>The remaining GPU memory holds the KV cache, which stores the context of in-flight requests. The startup log reports its <a href=\"https://docs.vllm.ai/en/v0.30.0/serving/parallelism_scaling/\" rel=\"nofollow ugc noopener\">capacity</a> (GPU KV cache size). All requests in flight share it, so longer prompts or more users need more of it. If you run out, lower <code>--max-model-len</code> or send fewer requests at once.</p>\n<h2 id=\"step-4-verify-serving-and-send-a-request\">Step 4. Verify serving and send a request</h2>\n<p>On A, check the API (<code>127.0.0.1</code> is A itself):</p>\n<pre><code>docker exec &quot;$CONTAINER&quot; curl -fsS --max-time 10 \\\n  &quot;http://127.0.0.1:$PORT/health&quot; &gt;/dev/null &amp;&amp; echo HEALTH_OK</code></pre>\n<p><code>HEALTH_OK</code> proves that Station A answers. Next, on both Stations, check that the ranks communicate over GPU RDMA rather than falling back to TCP sockets. The check reads the NCCL logs that the launcher writes through <code>NCCL_DEBUG=INFO</code>, and the same block shows the loaded memory:</p>\n<pre><code>docker exec &quot;$CONTAINER&quot; bash -lc &#39;\n  set -euo pipefail\n  files=(/tmp/nccl_*.log)\n  test -e &quot;${files[0]}&quot;\n  grep -hE &quot;NET/IB.*GDRDMA|GPU Direct RDMA Enabled&quot; &quot;${files[@]}&quot; &gt;/dev/null\n  grep -hF &quot;$NCCL_IB_HCA&quot; &quot;${files[@]}&quot; &gt;/dev/null\n  if grep -hE &quot;NET/Socket|NCCL WARN|NCCL ERROR&quot; &quot;${files[@]}&quot;; then\n    exit 1\n  fi\n  echo RDMA_CHECK_OK\n&#39;\nnvidia-smi</code></pre>\n<p><code>RDMA_CHECK_OK</code> means the check passed with no warnings. In <code>nvidia-smi</code>, A shows the <code>VLLM::Worker_TP0</code> process and B shows <code>VLLM::Worker_TP1</code>, one rank per GPU. The startup log reports about 218 GiB of weights per GPU, roughly half the checkpoint, which confirms that the model is split across the two GPUs.</p>\n<div class=\"table-wrap\"><table><thead><tr><th>Measurement</th><th>Station A</th><th>Station B</th></tr></thead><tbody><tr><td>GB300 GPU used (GiB)</td><td>0.01 → 231.35</td><td>0.00 → 231.34</td></tr></tbody></table></div>\n<p><em>Table 1. GPU memory before and after loading the model, with no requests running. Values include weights, KV cache, and runtime buffers</em></p>\n<p>On A, send a chat request through the OpenAI-compatible API:</p>\n<pre><code>docker exec &quot;$CONTAINER&quot; curl -fsS --max-time 120 \\\n  &quot;http://127.0.0.1:$PORT/v1/chat/completions&quot; \\\n  -H &#39;Content-Type: application/json&#39; \\\n  -d &#39;{&quot;model&quot;:&quot;nvidia/GLM-5.3-NVFP4&quot;,\n    &quot;messages&quot;:[{&quot;role&quot;:&quot;user&quot;,&quot;content&quot;:&quot;Quarterly units sold: Q1 = 120; Q2 = 180; Q3 = 150. Summarize the trend in one sentence and calculate the percentage change from Q1 to Q2.&quot;}],\n    &quot;temperature&quot;:0,&quot;max_tokens&quot;:4096,\n    &quot;chat_template_kwargs&quot;:{&quot;reasoning_effort&quot;:&quot;low&quot;}}&#39; |\npython3 -c &#39;import json,sys; c=json.load(sys.stdin)[&quot;choices&quot;][0];\nprint(c[&quot;message&quot;][&quot;content&quot;]); print(&quot;finish_reason:&quot;,c[&quot;finish_reason&quot;])&#39;</code></pre>\n<p>The response should look like this:</p>\n<pre><code>**Trend:** Units sold rose sharply in Q2 before partially declining in Q3, remaining above the Q1 level.\n\n**Q1 → Q2 percentage change:** (180 - 120) / 120 × 100 = **50% increase**\nfinish_reason: stop</code></pre>\n<p><code>finish_reason: stop</code> means the model finished its answer instead of hitting the token limit. The request sets <code>reasoning_effort</code> to <code>low</code> in <code>chat_template_kwargs</code>. GLM-5.3 also accepts <code>high</code> and <code>max</code>, which let it reason longer before answering, at the cost of latency. Because the launcher enables tool calling (<code>--enable-auto-tool-choice</code>), agent frameworks that use the OpenAI tools API can call the same endpoint.</p>\n<h2 id=\"step-5-concurrency-and-release-resources\">Step 5. Concurrency and release resources</h2>\n<p><a href=\"https://docs.nvidia.com/aiperf/latest/welcome-to-ai-perf-documentation\" rel=\"nofollow ugc noopener\">AIPerf</a>, the NVIDIA benchmarking tool for inference endpoints, measures throughput and latency. Installing it inside the serving container can change vLLM’s dependencies, so on A, install it in its own virtual environment:</p>\n<pre><code>sudo apt update &amp;&amp;\nsudo apt install -y python3.12-venv python3.12-dev build-essential &amp;&amp;\npython3.12 -m venv &quot;$HOME/.venvs/aiperf-0.12.0&quot; &amp;&amp;\n&quot;$HOME/.venvs/aiperf-0.12.0/bin/pip&quot; install &#39;aiperf==0.12.0&#39; &amp;&amp;\n&quot;$HOME/.venvs/aiperf-0.12.0/bin/pip&quot; check</code></pre>\n<p>The install should end with <code>No broken requirements found</code>.</p>\n<p>AIPerf can send synthetic prompts or replay an existing dataset. This tutorial uses the 80 Coding conversations from SPEED-Bench, a set of real prompts built to evaluate speculative decoding, so MTP is measured on realistic text. SPEED-Bench’s preparation script fetches these prompts from several Hugging Face datasets, including the gated <a href=\"https://huggingface.co/datasets/cais/hle\" rel=\"nofollow ugc noopener\">cais/hle</a>: if you haven’t already, open its page and select <strong>Agree and access repository</strong> before you continue. Then, on A, log in to Hugging Face:</p>\n<pre><code>&quot;$HOME/.venvs/aiperf-0.12.0/bin/hf&quot; auth login</code></pre>\n<p>Next, download and prepare the dataset. See the <a href=\"https://docs.nvidia.com/aiperf/v0.12.0/tutorials/datasets-inputs/profile-with-speed-bench-dataset#prepare-the-dataset\" rel=\"nofollow ugc noopener\">official AIPerf guide</a> for details:</p>\n<pre><code>SPEED_BENCH_DIR=&quot;$HOME/datasets/speed-bench&quot;\nmkdir -p &quot;$SPEED_BENCH_DIR&quot;\ncurl -fLsS https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py | \\\n  &quot;$HOME/.venvs/aiperf-0.12.0/bin/python&quot; - --config qualitative --output_dir &quot;$SPEED_BENCH_DIR&quot;</code></pre>\n<p>On A, check the dataset and service. Continue when the check prints <code>BENCHMARK_READY</code>:</p>\n<pre><code>(\n  set -euo pipefail\n  source ~/glm53.env\n  curl -fsS &quot;http://$HEAD_IP:$PORT/health&quot; &gt;/dev/null\n  &quot;$HOME/.venvs/aiperf-0.12.0/bin/python&quot; - &lt;&lt;&#39;PY_CHECK&#39;\nimport json\nfrom pathlib import Path\nroot = Path.home()\npath = root / &quot;datasets/speed-bench/qualitative.jsonl&quot;\nassert path.exists(), &quot;Dataset missing: rerun the SPEED-Bench preparation&quot;\nrows = [json.loads(line) for line in path.read_text().splitlines() if line.strip()]\ncoding = [row for row in rows if row[&quot;category&quot;] == &quot;coding&quot;]\nassert len(coding) == 80, &quot;Expected 80 Coding conversations&quot;\nassert sum(len(row[&quot;messages&quot;]) for row in coding) == 89\nassert all(&quot;FULL BENCHMARK DATA SHOULD BE FETCHED&quot; not in message[&quot;content&quot;]\n           for row in coding for message in row[&quot;messages&quot;])\nprint(&quot;BENCHMARK_READY: dataset has 80 Coding conversations, 89 turns in total.&quot;)\n\nPY_CHECK\n)</code></pre>\n<p>The full sweep covers concurrency 1, 8, 16, and 32. At each level, AIPerf sends 3×C warmup requests, then measures 10×C requests, where C is the concurrency. Some conversations are sent more than once at higher concurrency.</p>\n<p>Benchmark requests use the model’s default reasoning effort (max) and an 8,192-token output cap; responses can finish before reaching it. Full sweeps can take several hours.</p>\n<p> A two-hour request timeout leaves room for long responses, <code>--server-metrics</code> collects server counters used to measure MTP acceptance, and <code>--export-level raw</code> keeps each response for inspection.</p>\n<p>For a quick test, run the block below on A with both containers running and no other API traffic. It tests C1 and C8 with prefix caching enabled, as in the reference measurements in Table 2, below.</p>\n<pre><code>(\n  set -euo pipefail\n  source ~/glm53.env\n  CODING_DATASET=&quot;$HOME/datasets/speed-bench/qualitative.jsonl&quot;\n  test -f &quot;$CODING_DATASET&quot;\n export CONCURRENCIES=${CONCURRENCIES:-&quot;1 8&quot;} # Quick run: C1 and C8.\n  # Optional full benchmark: follow the reproduction instructions below.\n  # Step 3: --max-num-seqs 16 -&gt; export CONCURRENCIES=&quot;1 8 16&quot;\n  # Step 3: --max-num-seqs 32 -&gt; export CONCURRENCIES=32\n\n  RUN_DIR=&quot;$HOME/aiperf-results/blog-coding-$(date +%Y%m%d-%H%M%S)&quot;\n  for C in $CONCURRENCIES;  do\n    ART=&quot;$RUN_DIR/c$C&quot;\n    env -u HF_HUB_OFFLINE -u TRANSFORMERS_OFFLINE \\\n      &quot;$HOME/.venvs/aiperf-0.12.0/bin/aiperf&quot; profile \\\n      -m &quot;$MODEL_REPO&quot; --url &quot;http://$HEAD_IP:$PORT&quot; \\\n      --endpoint-type chat --streaming --tokenizer &quot;$MODEL_HOST&quot; \\\n      --custom-dataset-type speed_bench_coding --input-file &quot;$CODING_DATASET&quot; \\\n      --request-count &quot;$((10 * C))&quot; --warmup-request-count &quot;$((3 * C))&quot; \\\n      --osl 8192 \\\n      --use-server-token-count --concurrency &quot;$C&quot; --random-seed 42 \\\n      --request-timeout-seconds 7200 \\\n      --server-metrics &quot;http://$HEAD_IP:$PORT/metrics&quot; \\\n      --export-level raw --artifact-dir &quot;$ART&quot;\n    printf &quot;Results: %s\\n&quot; &quot;$ART&quot;\n  done\n  printf &quot;Results directory: %s\\n&quot; &quot;$RUN_DIR&quot;\n)</code></pre>\n<p>AIPerf saves metrics, raw responses, and logs in the results directory. Warmup requests are excluded from the reported metrics.</p>\n<p><strong>Optional: Reproduce Table 2</strong>. In <code>~/launch-glm53.sh</code> from Step 3 on both Stations, keep <code>--enable-prefix-caching</code> and set <code>--max-num-seqs</code> to 16 or 32 for the corresponding group shown in the comments above. Before each group, stop and remove the current containers on A, then B (<code>docker stop --timeout 60 &quot;$CONTAINER&quot;</code>, followed by <code>docker rm &quot;$CONTAINER&quot;</code>). Relaunch the updated script on B, then A with bash <code>~/launch-glm53.sh</code> and wait for <code>Application startup complete</code>. On A, run the matching <code>export CONCURRENCIES</code> command from the comments, then rerun the AIPerf block.</p>\n<p>To test without prefix caching, replace <code>--enable-prefix-caching</code> with <code>--no-enable-prefix-caching</code> in the Step 3 script on both Stations, then recreate the containers as above. Performance may differ from Table 2.</p>\n<p>Figure 2 compares overall output throughput across measured requests with mean per-user throughput. Token counts include reasoning.</p>\n<p><em>Figure 2. Overall output throughput versus mean per-user throughput on SPEED-Bench Coding, with prefix caching enabled. C1, C8, C16, and C32 denote configured concurrency</em></p>\n<p>Table 2, below, shows how throughput and latency change as more requests share the service.</p>\n<div class=\"table-wrap\"><table><thead><tr><th></th><th></th><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>Configured</strong> <strong>concurrency</strong></td><td><strong>Overall throughput</strong>  <strong>(tok/s)</strong></td><td><strong>Mean request latency</strong>  <strong>(s)</strong></td><td><strong>Mean TTFT</strong> <strong>(ms)</strong></td><td><strong>Mean inter-token latency</strong> <strong>(ms/token)</strong></td></tr><tr><td>1</td><td>107.4</td><td>49.84</td><td>479.75</td><td>8.78</td></tr><tr><td>8</td><td>386.0</td><td>114.17</td><td>433.79</td><td>19.01</td></tr><tr><td>16</td><td>607.8</td><td>136.46</td><td>482.90</td><td>23.77</td></tr><tr><td>32</td><td>385.0</td><td>453.54</td><td>899.36</td><td>77.90</td></tr></tbody></table></div>\n<p><em>Table 2. SPEED-Bench Coding results with prefix caching enabled, four MTP draft tokens per step, and an FP8 KV cache. Latency metrics are means over measured requests; warmup is excluded. TTFT measures time to the first token, which may be reasoning</em></p>\n<p>Overall throughput peaks at 607.8 tok/s at C16. It falls to 385.0 tok/s at C32, where mean request latency is 453.5 s. Higher concurrency also reduces mean per-user throughput, from 115.3 tok/s at C1 to 13.0 tok/s at C32.</p>\n<p>After the requests finish, stop the vLLM container on A, then on B:</p>\n<pre><code>docker stop --timeout 60 &quot;$CONTAINER&quot;\ndocker inspect --format \\\n  &#39;Status={{.State.Status}} Running={{.State.Running}}&#39; &quot;$CONTAINER&quot;\nnvidia-smi</code></pre>\n<p>The status should read <code>Status=exited Running=false</code>, with no vLLM processes in <code>nvidia-smi</code>; the checkpoint, containers, and results stay on disk. To resume, run <code>docker start &quot;$CONTAINER&quot;</code> on B, then A, and repeat the Step 4 checks.</p>\n<h2 id=\"troubleshoot-common-blockers\">Troubleshoot common blockers</h2>\n<ul><li>Missing variables or SSH failure: <code>source ~/glm53.env</code> after reconnecting; verify A-to-B key authentication and B’s fingerprint.</li><li>SPEED-Bench preparation fails with a gated-dataset error: sign in to Hugging Face with the same account, open the <a href=\"https://huggingface.co/datasets/cais/hle\" rel=\"nofollow ugc noopener\">cais/hle</a> page, select <strong>Agree and access repository</strong>, then rerun the preparation.</li><li>Startup or RDMA check failure: inspect <code>docker logs &quot;$CONTAINER&quot;</code> on both nodes; verify the checkpoint and the GID, and rerun the playbook’s validation (script 07).</li><li><a href=\"https://docs.vllm.ai/en/v0.30.0/usage/troubleshooting/\" rel=\"nofollow ugc noopener\">CUDA out of memory</a>: lower <code>--max-model-len</code> or concurrency, then retest.</li></ul>\n<h2 id=\"put-the-endpoint-to-work\">Put the endpoint to work</h2>\n<p>You now have a 753-billion-parameter model running behind one local API on two NVIDIA DGX Stations, with every weight in GPU memory. That opens up work that’s hard to do on a hosted endpoint: running inference on sensitive or proprietary data, prototyping agents locally, and iterating on prompts without per-token costs.</p>\n<p>Connect your own application using the <a href=\"https://docs.vllm.ai/en/v0.30.0/serving/online_serving/\" rel=\"nofollow ugc noopener\">vLLM serving documentation</a>, then rerun AIPerf with your prompts to find the concurrency that fits your workload. A frontier-scale model is now ready to build on, right from your desk.</p>\n<p><a href=\"#entry-content-comments\">Discuss (0)</a></p>\n<h2 id=\"tags\">Tags</h2>\n<p><a href=\"https://developer.nvidia.com/blog/category/generative-ai/\" rel=\"nofollow ugc noopener\">Agentic AI / Generative AI</a> | <a href=\"https://developer.nvidia.com/blog/category/edge-computing/\" rel=\"nofollow ugc noopener\">Edge Computing</a> | <a href=\"https://developer.nvidia.com/blog/category/networking-communications/\" rel=\"nofollow ugc noopener\">Networking / Communications</a> | <a href=\"https://developer.nvidia.com/blog/recent-posts/?industry=General\" rel=\"nofollow ugc noopener\">General</a> | <a href=\"https://developer.nvidia.com/blog/recent-posts/?products=Blackwell\" rel=\"nofollow ugc noopener\">Blackwell</a> | <a href=\"https://developer.nvidia.com/blog/recent-posts/?products=DGX\" rel=\"nofollow ugc noopener\">DGX</a> | <a href=\"https://developer.nvidia.com/blog/recent-posts/?learning_levels=Intermediate+Technical\" rel=\"nofollow ugc noopener\">Intermediate Technical</a> | <a href=\"https://developer.nvidia.com/blog/recent-posts/?content_types=Tutorial\" rel=\"nofollow ugc noopener\">Tutorial</a> | <a href=\"https://developer.nvidia.com/blog/tag/dgx-station/\" rel=\"nofollow ugc noopener\">DGX Station</a> | <a href=\"https://developer.nvidia.com/blog/tag/inference-performance/\" rel=\"nofollow ugc noopener\">Inference Performance</a> | <a href=\"https://developer.nvidia.com/blog/tag/llm-benchmarking/\" rel=\"nofollow ugc noopener\">LLM Benchmarking</a> | <a href=\"https://developer.nvidia.com/blog/tag/large-language-models/\" rel=\"nofollow ugc noopener\">LLMs</a> | <a href=\"https://developer.nvidia.com/blog/tag/local-ai/\" rel=\"nofollow ugc noopener\">Local AI</a> | <a href=\"https://developer.nvidia.com/blog/tag/vllm/\" rel=\"nofollow ugc noopener\">vLLM</a></p>\n<h2 id=\"about-the-authors\">About the Authors</h2>\n<p><strong>About Fatima Fall</strong></p>\n<p>Fatima Fall is a technical marketing engineer with a deep interest in everything related to AI. She studied at Howard University, where she got a B.Sc. in applied mathematics.</p>\n<p><a href=\"https://developer.nvidia.com/blog/author/ffall/\" rel=\"nofollow ugc noopener\">View all posts by Fatima Fall</a></p>","headings":[{"level":1,"text":"When One GPU Isn’t Enough: Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM","id":"when-one-gpu-isn-t-enough-serving-glm-5-3-across-two-nvidia-dgx-"},{"level":2,"text":"AI-Generated Summary","id":"ai-generated-summary"},{"level":3,"text":"Next Steps","id":"next-steps"},{"level":2,"text":"Prerequisites","id":"prerequisites"},{"level":2,"text":"Step 1. Prepare the two Stations","id":"step-1-prepare-the-two-stations"},{"level":2,"text":"Step 2. Download and copy the model","id":"step-2-download-and-copy-the-model"},{"level":2,"text":"Step 3. Launch the distributed service","id":"step-3-launch-the-distributed-service"},{"level":2,"text":"Step 4. Verify serving and send a request","id":"step-4-verify-serving-and-send-a-request"},{"level":2,"text":"Step 5. Concurrency and release resources","id":"step-5-concurrency-and-release-resources"},{"level":2,"text":"Troubleshoot common blockers","id":"troubleshoot-common-blockers"},{"level":2,"text":"Put the endpoint to work","id":"put-the-endpoint-to-work"},{"level":2,"text":"Tags","id":"tags"},{"level":2,"text":"About the Authors","id":"about-the-authors"}]}}