How do I choose DCGM, nvbandwidth, nccl-tests, gpu-burn, and NVIDIA Field Diagnostic for GPU delivery vs RMA in 2026?

Quick Take

Choose GPU stress tools by goal: DCGM for in-OS triage, nvbandwidth for link copy, nccl-tests for collectives, gpu-burn for facility soak, and NVIDIA Field Diagnostic for RMA. Delivery that skips the first four is incomplete; an RMA without fieldiag.log and ECC on is incomplete. Fit for HGX/B300-class Shanghai plant sign-off; unfit as a price quote or as treating a DCGM green run as NVIDIA RMA eligibility. B300 pass bars below are simulated pending confirmation vs your datasheet.

What is the five-tool GPU stress stack? Definition and fit/unfit

The GPU server stress-test stack is a complementary toolkit of five roles that Shanghai plants should freeze before CapEx sign-off on GPU servers and HGX-class chassis. DCGM / dcgmi answers “is this GPU sick in the OS?” nvbandwidth answers “how fast do the links copy?” nccl-tests answers “how fast do training-like collectives move?” gpu-burn answers “does full load hold for hours?” NVIDIA Field Diagnostic (fieldiag) is the RMA authority. They do not substitute for one another.

Fit this page to HGX / B300-class delivery acceptance, fleet inspect, fabric vs HBM vs thermal isolation, and assembling an RMA evidence pack. Unfit: quoting GPU prices from this page; using gpu-burn or DCGM Pass as Field Diagnostic; treating WeChat-class B300 numbers as your chassis SLA; running Field Diag with ECC off; failing a dual-socket box solely because SYS P2P is ~PCIe-class; skipping nccl-tests on a training cluster PO.

How do I choose a tool by goal and scenario?

Map the buyer goal to the first tool, then follow the scenario sequence. Do not start an RMA ticket from a DCGM Fail alone: a Fail can be host, driver, cooling, power, or interference—not only the GPU ASIC.

What does DCGM / dcgmi diag actually prove?

DCGM Diagnostics are an in-OS health suite. Official NVIDIA docs state they do not repair faults, do not replace NVIDIA Field Diagnostics, and do not determine RMA eligibility. A green run means the selected plugins did not fail on this host at this driver—not that NVIDIA will accept a board swap. Result tokens are Pass / Fail / Skip / Warning; treat Skip as “not a clean green.”

Suite selectors from the DCGM user guide: 1 / quick / short = software deployment checks; 2 / medium adds memory + pcie; 3 / long adds diagnostic, memory_bandwidth, targeted stress/power, and the nvbandwidth plugin plus nccl_tests plugin when present; 4 / xlong adds memtest + pulse_test on top of L3. Typical plant commands: dcgmi discovery -l, dcgmi diag -r 1, dcgmi diag -r 3, dcgmi diag -r memory -i <GPU_INDEX>, dcgmi diag -r 2 -t pcie, JSON via dcgmi diag -r 1 -j, plus dcgmi dmon / health / topo. If the nvbandwidth plugin Fails, install the binary and idle the GPUs (is_allowed) before blaming the ASIC. Shanghai TaoHan Industrial files DCGM JSON next to the PO sheet as triage, never as the RMA ticket.

When should I run nvbandwidth vs nccl-tests?

Run nvbandwidth when you need memcpy/link evidence: D2D (HBM), H2D/D2H (PCIe), P2P (NVLink or SYS). Official GitHub (retrieved 2026-09-19): CUDA 11.x+ (multinode 12.3+ + driver 550+), CMake 3.20+, default buffer 512 MiB (-b), default iterations 3 (-i), JSON via --format json (not blog -j). Always ./nvbandwidth -l before freezing -t names. Strict acceptance often uses -b 2048 -i 10. CE is Copy Engine (link ceiling, no SM compute); SM is kernel copy (closer to app cost). Run both where the family exists. Deep CLI—including exact testcase spellings—lives on the sibling nvbandwidth guide; this page only needs the path families and the topology lesson: same-socket NV18 should be high; cross-CPU SYS walking PCIe is expected low.

Run nccl-tests when the purchase order is a training cluster. Common tests: all_reduce_perf (most used), all_gather_perf, alltoall_perf, broadcast_perf, reduce_perf, reduce_scatter_perf, sendrecv_perf. Single-node 8-GPU example: ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8. Read algbw (algorithm BW), busbw (bus BW); wrong must stay 0. Too-small messages never hit the fabric ceiling. Multi-node MPI plus NCCL_IB_* knobs are site-specific / pending confirmation—publish as a pattern, not a copy-paste for every fabric. nvbandwidth healthy with nccl-tests slow usually means NCCL/IB/topo, not a dead ASIC.

When is gpu-burn the right soak — and when is it unsafe?

gpu-burn (community repo wilicc/gpu-burn, not NVIDIA Field Diagnostic) is the facility soak: full load, power, thermal, compute errors. Typical commands: ./gpu_burn 1800 (30 min), ./gpu_burn -d 3600 (1 h double precision), ./gpu_burn 7200 (2 h). Daily inspect 15–30 min; strict delivery 12–24 h with errors=0. Watch a second terminal watch -n 1 nvidia-smi. WeChat-class “check fans if ≥90 °C” is pending confirmation vs OEM thermal spec. Multi-GPU full load can trip PDU/OPP—step GPUs if the hall is not signed off. gpu-burn errors after two hours often mean CDU/fans first; they are not an automatic NVIDIA RMA. Do not treat a 30-minute green soak as Field Diagnostic.

Is NVIDIA Field Diagnostic required for GPU RMA?

Yes for an NVIDIA/vendor RMA packet in the usual plant path. Official names: NVIDIA Field Diagnostic / HW Field Diag. GPU Debug Guidelines call it the authoritative and comprehensive NVIDIA tool, usually required before an RMA; contact your system vendor for if/when/how. Binary is ./fieldiag (OEM packages may wrap fieldiag.sh --level1/--level2/--sit). ECC must be enabled; failures only with ECC off are not RMA-eligible (NVIDIA RMA Process). Artifact is fieldiag.log—missing log delays the ticket. Obtain the package from OEM / NVIDIA support; it is not a public GitHub clone like gpu-burn. HGX SKUs often have SKU-specific packages. WeChat module times (~2 h L1 / ~4.5–5.5 h L2, --sit ~34 min) are OEM-package class / pending confirmation. Follow the Field Diagnostic Quick Start in your package (DU-05711-001 / DU-05363-001 class docs)—do not assume every crate boots the same Rufus USB flow. Channel narratives of error codes (540 NVRM Fatal, 140 NvLink bus, 143 PCI Express, PEX ERROR) stay pending confirmation against your OEM log decoder; do not treat them as a public NVIDIA catalog.

B300 worked-example pass bars (simulated)

Label everywhere: illustrative B300-class 8-GPU dual-socket; pending confirmation vs OEM datasheet / your PO. Do not treat these rows as a contractual SLA.

nvbandwidth measures how fast the links copy; nccl-tests measures how fast training-like collectives move; gpu-burn measures whether full load holds; Field Diagnostic is the RMA authority; DCGM is the in-OS triage ruler. Optional after the GPU stack: stress-ng --vm / memtester for DRAM and stress-ng --cpu plus turbostat for CPU—pending confirmation vs OEM, not a sixth equal tool.

HowTo: delivery vs RMA sequence (Step 1–6)

Use this numbered checklist on the Shanghai plant floor. Keep exact nvbandwidth -t names from -l on the release you built.

1. Step 1 — Capture baseline. Save nvidia-smi -q (SKU, driver, CUDA, PCIe Gen/Width, clocks/temp), nvidia-smi topo -m, and nvidia-smi nvlink -s into a dated folder. Confirm ECC on if RMA is in scope.
2. Step 2 — Triage in OS. dcgmi discovery -l then dcgmi diag -r 1 (quick) and dcgmi diag -r 3 (long) on idle GPUs; save JSON (-j). Treat Skip/Fail as not signed off, not as an automatic RMA.
3. Step 3 — Measure links. Build nvbandwidth from GitHub; ./nvbandwidth -l; run default sweep plus D2D CE, H2D/D2H CE, P2P SM with -b 2048 -i 10 --format json. Score NV18 vs SYS cells (do not fail SYS-class cross-CPU as broken NVLink).
4. Step 4 — Measure collectives. all_reduce_perf single-node (-g 8 class) then multi-node only if the PO is a cluster; wrong must be 0; archive busbw peaks.
5. Step 5 — Soak the facility. gpu-burn 12–24 h for strict delivery (or 15–30 min smoke); errors=0; watch PDU/CDU/nvidia-smi. This does not replace Field Diagnostic.
6. Step 6 — RMA packet (only if hardware still suspect). Run the OEM NVIDIA Field Diagnostic with ECC enabled; keep fieldiag.log + nvidia-bug-report.log.gz; cross-slot if the fault follows the GPU. Submit via vendor/NVIDIA — DCGM/gpu-burn logs are attachments, not the ticket.

Real-World Deployment Scenarios

Map the five buyer pictures below to your workload, hall, and commercial constraints before freezing CapEx. The same five-tool order applies; only the emphasis changes.

Common pitfalls

FAQ

References

• NVIDIA DCGM Diagnostics — suite levels; not a Field Diag / RMA substitute
• dcgmi diag CLI reference
• NVIDIA nvbandwidth on GitHub — 512 MiB default, --format json
• NVIDIA nccl-tests
• gpu-burn (community)
• NVIDIA Field Diagnostic
• NVIDIA RMA Process — ECC on, fieldiag.log
• NVIDIA GPU Debug Guidelines — Field Diag as RMA authority
• NVIDIA NCCL
• nvidia-smi documentation

Key Takeaway

Freeze a Shanghai plant GPU packet around five complementary tools: DCGM for in-OS triage (not RMA), nvbandwidth with a 512 MiB default and --format json for links, nccl-tests for collectives, gpu-burn 12–24 h for soak, and NVIDIA Field Diagnostic with ECC on plus fieldiag.log for the RMA ticket. Keep B300 numeric bars labeled as worked examples pending confirmation. TAOHAN (上海滔涵实业有限公司) helps domestic buyers lock that checklist before CapEx sign-off—without publishing channel GPU prices.

Related reading