Quick Take
要点速览： Run NVIDIA nvbandwidth from GitHub to measure CE and SM memcpy bandwidth across D2D, H2D/D2H, and P2P before you sign GPU delivery. Fit for HGX and B300-class acceptance baselines; unfit as a substitute for NCCL or compute burn-in. Score NV18 cells separately from cross-CPU SYS pairs.
Buyers searching for a GPU acceptance bandwidth test or delivery inspection checklist need a repeatable CLI, not a marketing screenshot. This guide follows the official NVIDIA nvbandwidth README (retrieved 2026-09-19): default buffer 512 MiB, prefer --format json, and always list testcases with -l before freezing a script.
What is NVIDIA nvbandwidth?
速答： nvbandwidth is NVIDIA’s open-source tool that reports memcpy bandwidth on your host using Copy Engine (CE) hardware DMA or Streaming Multiprocessor (SM) kernels. It covers host↔device, local HBM device↔device, and peer GPU paths. Use it for delivery acceptance and topology diagnosis—not as a price quote or sole burn-in.
Definition: nvbandwidth measures bandwidth for various memcpy patterns using the copy engine or kernel copy, and reports measured results on your system—additional tuning may be required for peak. That makes it the right first-pass instrument when a server arrives and you need written evidence before CapEx sign-off.
CE vs SM (must teach both):
- Copy Engine (CE): hardware DMA; does not consume SM compute; measures link and path bandwidth.
- Streaming Multiprocessor (SM): kernels on SMs; consumes compute; closer to application memcpy cost.
- Acceptance rule: run both CE and SM where the testcase family exists.
Fit: GPU server / HGX / B300-class delivery acceptance; diagnosing PCIe vs NVLink; locking a written bandwidth baseline before CapEx sign-off.
Unfit: replacing NCCL or application training benchmarks; quoting channel GPU prices; accepting a box when the supplier refuses any bandwidth evidence; treating simulated worked-example numbers on this page as your chassis SLA.
Install & build from GitHub
速答： Build nvbandwidth from GitHub with CUDA 11.x+, CMake 3.20+ (prefer 3.24+), and GCC 7.x+ on Linux. Install libboost-program-options-dev, then run cmake . && make or sudo ./debian_install.sh. Multinode needs CUDA 12.3+, driver 550+, MPI, and cmake -DMULTINODE=1 before any acceptance script freezes.
| Item | Official value (GitHub) |
|---|---|
| Repo | https://github.com/NVIDIA/nvbandwidth |
| CUDA | 11.x+; multinode needs CUDA 12.3+ and driver 550+ |
| Compiler | C++17 — GCC 7.x+ (Linux) |
| CMake | 3.20+ (encourage 3.24+) |
| Linux deps | libboost-program-options-dev, build-essential, cmake |
| Single-node | cmake . && make |
| Ubuntu helper | sudo ./debian_install.sh |
| Multinode | cmake -DMULTINODE=1 . && make (needs MPI) |
After build, confirm help and the live test list before any acceptance script:
Testcase spellings change across releases—the README example uses names such as device_to_device_memcpy_write_ce. Freeze -t arguments only after -l on the binary you ship with the acceptance packet.
CLI flags that matter for acceptance
速答： Defaults are 512 MiB buffer (-b) and three samples (-i). List tests with -l, select with -t, and export with --format json (also text or perf). Prefer GitHub flags over blog shortcuts such as -j or a stale 256 MiB default. Use -d only when affinity must be disabled.
| Flag | Meaning | Acceptance note |
|---|---|---|
-b / --bufferSize | Buffer size in MiB | Default 512; strict runs often use 2048 |
-i / --testSamples | Iterations | Default 3; strict runs often use 10 |
-l / --list | List testcases | Mandatory before scripting |
-t / --testcase | Named tests (space-separated) | Verify spellings with -l |
--format | json, text, perf | Prefer json for archives |
-d / --disableAffinity | Disable affinity | Use only when diagnosing pinning |
-h | Help | Sanity-check the built binary |
Example strict acceptance invocation pattern (replace -t names after -l):
Core test modes: D2D / H2D-D2H / P2P
速答： D2D measures local HBM memcpy. H2D and D2H measure host↔device PCIe. P2P measures GPU↔GPU over NVLink (NV18) or PCIe/SYS. Always pair CE and SM variants when listed, and map every cell to nvidia-smi topo -m before declaring pass or fail on delivery.
- D2D — device local HBM memcpy (e.g.
device_to_device_memcpy_*_ce/_sm; exact names from./nvbandwidth -l). - H2D / D2H — host↔device PCIe (e.g.
host_to_device_memcpy_ce,device_to_host_memcpy_ce, bidirectional CE). - P2P — GPU↔GPU over NVLink or PCIe/SYS (e.g.
sm_copy_device_to_device_bidirectional, one-to-one, one-to-many families).
Topology lesson: same-socket NV18 pairs should land in the high band. Cross-CPU SYS pairs that walk PCIe are expected to land near PCIe class. That low SYS cell is not a “broken NVLink” unless an expected NV18 cell collapses to SYS-class numbers.
Pass/fail thresholds (B300 worked example — simulated)
速答： These B300 ×8 dual-socket bars are an illustrative simulation pending confirmation against your OEM datasheet and purchase order. Use them only to shape RFQ language, never as contractual proof. Re-derive every pass bar from your chassis sheet before CapEx sign-off.
| Test | Theory (worked example) | Pass bar (example) | Watch / fail signal | First checks |
|---|---|---|---|---|
| D2D HBM (CE) | ~8 TB/s class (B300 HBM3e) | > ~6.4 TB/s (≈80%) | < ~5 TB/s | clocks, thermal throttle, HBM health |
| H2D PCIe CE | ~64 GB/s Gen5 x16 | > ~50 GB/s | < ~40 GB/s | Link Gen/Width, slot, affinity |
| D2H PCIe CE | ~64 GB/s | > ~50 GB/s | < ~40 GB/s | same |
| H↔D bidirectional CE | ~55–64 GB/s class total | > ~40 GB/s total | < ~30 GB/s | half-duplex split is normal |
| P2P NV18 bidirectional SM | ~900 GB/s class | > ~800 GB/s | ~50 GB/s on expected NV18 pair | Bridge / nvidia-smi topo -m / nvlink -s |
| P2P NV18 unidirectional SM | ~450 GB/s class | > ~400 GB/s | ~25 GB/s | same |
| P2P SYS (cross-CPU) | ~PCIe class (~64 GB/s bidir) | expect ~45–55 GB/s class | treating SYS as NV18 fail | topology matrix |
Simulated matrix story (illustrative only — not PO proof):
- D2D CE ~6848 GB/s/GPU (~85.6% of 8000) — pass under the worked-example bar
- H2D CE ~55.8 GB/s — pass
- D2H CE ~54.9 GB/s — pass
- Bidirectional ~55.7 GB/s total with ~27.9 each way — normal half-duplex
- P2P bidir NV18 ~873 GB/s; SYS ~48.4 GB/s — expected dual-socket pattern
- P2P one-to-one NV18 ~448 GB/s; SYS ~24.2 GB/s
HowTo: delivery acceptance script (six steps)
速答： Capture nvidia-smi baselines, build nvbandwidth, run a default sweep, then a strict -b 2048 -i 10 JSON run covering D2D, H2D, D2H, bidirectional CE, and P2P SM. Score NV18 versus SYS, escalate Bridge or Gen/Width faults, and archive JSON as the regression baseline.
- Step 1 — Baseline capture: Run
nvidia-smi -q(product, driver, CUDA, PCIe Gen/Width, clocks/temp) plusnvidia-smi topo -mandnvidia-smi nvlink -s. Save outputs to a dated folder. - Step 2 — Build nvbandwidth: Clone the GitHub repo; install deps;
cmake . && makeorsudo ./debian_install.sh. Confirm./nvbandwidth -hand./nvbandwidth -l. - Step 3 — Quick sweep: Run
./nvbandwidthwith defaults for a one-to-two-minute health matrix. - Step 4 — Strict acceptance: Large buffer + more samples, e.g.
-b 2048 -i 10, plus--format jsoninto a dated results dir. Cover D2D CE, H2D CE, D2H CE, bidirectional CE, and P2P bidirectional SM using exact-tnames from-l. - Step 5 — Score vs topology: Compare NV18 vs SYS cells. Do not fail the box solely because cross-CPU SYS is ~PCIe-class.
- Step 6 — Escalate anomalies: NV18→~50 GB/s → reseat Bridge + nvlink status; H2D ~30 GB/s → Gen/Width; CE OK / SM low → other GPU load or SM clock; save JSON as baseline for regression.
Optional soak (not required on every RFQ): loop with thermal watch and nvidia-smi nvlink --errorcounters.
Reading the bandwidth matrix + JSON
速答： Read CE and SM columns against the topology map, not against a single global number. High NV18 with low SYS is normal on dual-socket HGX-class boxes. Archive --format json next to nvidia-smi topo so later regressions compare apples to apples.
When you open a JSON or text matrix:
- Ignore 0.00 on the diagonal (self-to-self).
- Annotate each GPU pair as NV18 or SYS using
nvidia-smi topo -mbefore scoring. - Expect half-duplex splits on bidirectional host↔device CE—per-direction figures near half of the total are normal.
- Keep driver version, CUDA toolkit version, and buffer size in the same folder as the JSON so a later retest is comparable.
If an expected NV18 pair prints ~50 GB/s while neighbors stay in the hundreds of GB/s, treat that as a Bridge / fabric path fault first—not as “the GPU is slow.”
Common failure patterns
速答： NV18 collapsing to ~50 GB/s usually means Bridge or SYS pathing. H2D near 30 GB/s usually means Gen4 or a narrow link. CE healthy with SM low usually means other GPU jobs or throttle. High variance means enlarge buffer and iterations under exclusive access.
| Symptom | Likely cause | Check |
|---|---|---|
| Expected NV18 ~50 GB/s | Bridge / SYS path | topo -m, nvlink -s |
| H2D ~30 GB/s | Gen4 or narrow link | nvidia-smi -q PCIe |
| CE OK, SM low | other GPU jobs / throttle | idle GPUs; CLOCK query |
| Build fails on boost | missing package | libboost-program-options-dev |
| High run-to-run variance | small buffer / few iters / noise | -b 2048 -i 10, exclusive access |
| 0.00 on diagonal | self-self | ignore |
真实应用场景
速答： Map the three scenarios below to your workload, facility, and commercial constraints before freezing CapEx. Equal-BOM RFQs should cite which scenario drives the shortlist.
Scenario 1: HGX H200 Node Incoming Inspection (Bandwidth Bound)
Workload Profile: 8-GPU HGX H200 servers arriving from overseas freight into a free-port staging hall before rack install.
Architectural Fit: Run nvbandwidth host-to-device and P2P suites against published class thresholds before signing the delivery note. Failures usually trace to PCIe bifurcation, BIOS NUMA, or NVLink fabric faults—not software alone.
Scenario 2: Mixed H100/H200 Pod Acceptance (Fleet Bound)
Workload Profile: Existing H100 racks receiving H200 expansion nodes that must share the same NCCL fabric assumptions.
Architectural Fit: Accept each SKU family on its own bandwidth floor; never average H100 and H200 results into one pass/fail number for CapEx close-out.
Scenario 3: Multi-Vendor Integrator Handoff (Contract Bound)
Workload Profile: SI delivers turnkey nodes; buyer must prove bandwidth before payment milestones.
Architectural Fit: Freeze the nvbandwidth command set, GPU count, and driver/CUDA baseline in the RFQ appendix so disputes cite one equal test matrix.
常见问题
速答： Use nvbandwidth for a written CE/SM bandwidth baseline at delivery; run both engines; expect Gen5 x16 H2D above ~50 GB/s in the worked example; treat low cross-CPU P2P as SYS; prefer 512 MiB default or larger for strict runs; never treat the simulated B300 D2D figure as a PO guarantee.
nvbandwidth measures memcpy bandwidth across host-device, local device HBM, and peer GPU paths using Copy Engine or SM kernels. Use it to lock a written bandwidth baseline at delivery. It does not replace NCCL or full compute burn-in, and it is not a price quote.
Run both where the testcase family exists. Copy Engine (CE) is hardware DMA and does not consume SM compute, so it stresses the link path. Streaming Multiprocessor (SM) kernels consume compute and track application-style memcpy cost more closely.
A worked-example Gen5 x16 host-to-device CE bar is above about 50 GB/s against a theoretical class near 64 GB/s. Below about 40 GB/s is a watch signal—confirm Link Generation, Width, slot wiring, and CPU affinity before blaming the GPU.
Cross-CPU pairs often walk the SYS or PCIe fabric instead of NV18. Expect roughly 45–55 GB/s class on those cells. Fail the box only when an expected NV18 pair collapses to SYS-class numbers, not because SYS itself is low.
GitHub defaults are 512 MiB buffer and three samples. For strict acceptance, prefer a larger buffer and more samples—for example -b 2048 -i 10—plus --format json into a dated folder. Always confirm exact -t names with ./nvbandwidth -l.
No. The ~6848 GB/s D2D CE figure is an illustrative B300 ×8 dual-socket simulation pending confirmation against your OEM datasheet. Treat every pass bar on this page as a worked example, never as a contractual SLA for your purchase order.
References
速答： Anchor every acceptance script to the official NVIDIA nvbandwidth GitHub README, the CUDA compatibility docs, and nvidia-smi pages linked below. Do not cite channel blogs for buffer defaults or flag spellings when the README disagrees with them on your release.
Key Takeaway
核心结论： Sign GPU delivery only after CE and SM nvbandwidth runs map cleanly onto the topology matrix, with NV18 and SYS scored separately. Archive JSON next to nvidia-smi baselines. TAOHAN supplies nvbandwidth acceptance tooling guidance with competitive pricing for B300-class delivery inspection RFQs.