Quick Take
Quick Take: Seven metrics decide an AI cluster GPU shortlist among H100, H200, B300, MI455X, and Ascend 910C in 2026: train versus infer fit, HBM capacity and bandwidth, scale-up fabric, cluster availability, software migration cost, dollars per token, and tokens per watt. Fit is scenario A through E. Unfit is buying peak TFLOPS alone or treating 910C as a drop-in H100.
What this 7-metric map is
Quick Answer: This map is a five-card shortlist, not a single winner. Use it when the load is cluster training or inference and the choice is among H100, H200, B300, MI455X, and Ascend 910C. It does not cover RTX PRO workstation editions. Freeze one primary scenario, then put a second card on a separate RFQ line.
The seven-metric map is a procurement decision frame for one buyer question: which accelerator belongs on a new AI cluster bill of materials in 2026. The cards in scope are NVIDIA H100, NVIDIA H200, NVIDIA B300, AMD Instinct MI455X, and Huawei Ascend 910C. Workstation RTX PRO Editions live on a different page. This page does not publish a live net price, a lead time, or an invented part number.
Fit is a named cluster job. You can say whether the primary load is pretrain or decode, you can name parameter count and context, and you can say whether the model is mixture-of-experts. Unfit paths are explicit:
- Picking the card with the largest TFLOPS cell and ignoring memory, fabric, and cooling.
- Ordering B300 into a hall that has no liquid-cooling capital plan.
- Treating Ascend 910C as a drop-in H100 for frontier CUDA pretrain.
- Treating MI455X paper bandwidth as a proven 10,000-GPU collective.
- Mixing two cards on one SKU row instead of two RFQ lines.
Every peak below is a class figure for decisioning. Label it pending confirmation versus the OEM datasheet and your purchase order before it becomes a pass bar. Soft dollar and yuan corridors are industry estimates pending confirmation. They are not a quotation.
Five cards, five niches
Quick Answer: H100 is the mature CUDA training and inference buy. H200 stays in that Hopper class with larger HBM for long-context decode. B300 is the Blackwell Ultra performance and NVLink 5 class. MI455X is a diversification bet with ROCm risk. Ascend 910C is the compliance track on CANN, not a 910B substitute.
Each card occupies a niche. Substituting the neighbor because the brochure uses the same family name is how clusters miss the job. H200 is not H100 with a new software stack. B300 is not a prior Blackwell step you can size from an older node. MI455X is not MI300X. Ascend 910C is not 910B and not a generic Ascend SKU.
| Card | Buyer niche | Not the same as |
|---|---|---|
| NVIDIA H100 | Mature train and infer buy; CUDA with no migration | H200 (more HBM) or B300 (next generation) |
| NVIDIA H200 | Same Hopper class, larger HBM for 70B-class and long-context decode | H100 80 GB class |
| NVIDIA B300 | Blackwell Ultra compute, memory, and NVLink 5 for frontier train and large-batch infer | Prior Blackwell step; do not invent that BOM here |
| AMD Instinct MI455X | Diversification bet; paper HBM and bandwidth are strong; ROCm and UALink maturity is the risk premium | MI300X |
| Huawei Ascend 910C | Domestic and compliance track on CANN; interconnect is below NVLink class | 910B or other Ascend names |
Use the niche column when you write the RFQ subject line. A subject that only says “GPU servers” forces the seller to guess which of these five jobs you meant.
Metric 1 — Train vs infer fit
Quick Answer: Training is usually compute-bound, while decode is usually memory-bound. A frontier pretrain shortlist starts at B300-class FP8, not at an H100 peak. A 70B inference job can still fit H200, or H100 if context and batch stay modest. Name train versus infer before you compare fabrics.
Metric 1 splits the shortlist before any fabric debate. Pretrain on trillion-parameter class models is compute-bound for long stretches, so FP8 class and scale-up domain dominate. Token decode, especially long context, is memory-bound: the card must hold weights plus KV cache, and bandwidth sets a floor on time per token at batch size 1.
Class compute, pending confirmation versus OEM sheets, looks like this. H100 FP8 sits near 3,958 TFLOPS, with mature NCCL and a large-cluster linearity claim often cited above 90 percent. That linearity figure is an operations claim, pending confirmation, not a guarantee on your job. B300 FP8 sits near 7,000 TFLOPS in the same narrative, about 2.3x the H100 cell, with NVLink 5 near 1.8 TB/s. Train-time commentary often puts a trillion-parameter class run at about 40 to 50 percent of H100 wall time. That ratio is an industry estimate.
Ascend 910C FP16 sits near 800 TFLOPS in the same narrative, about 40 percent of the H100 FP8 story, with HCCS near 400 GB/s. A model-FLOPs-utilization narrative near 30 percent versus about 55 percent on H100 widens the effective train gap toward 4.5x. That gap is an industry estimate, not a benchmark you can paste into a contract. MI455X carries a paper FP8 figure near 20 PFLOPS. Large-cluster ROCm stability on that card is pending confirmation. Do not promote the paper peak into a 10,000-GPU plan.
If the job is mixed, write two rows: a train row and an infer row. One blended “AI GPU” line hides which metric actually failed.
Metric 2 — HBM capacity and bandwidth
Quick Answer: Class HBM pending confirmation is H100 at 80 GB, H200 at 141 GB, B300 at 288 GB, Ascend 910C at 96 GB, and MI455X at 432 GB. A 405B FP16 weight set near 810 GB is about an 11-way tensor split on H100 versus about 4-way on B300. Confirm the OEM sheet before either figure becomes a pass bar.
Metric 2 asks whether weights and KV cache fit, and how fast a decode step can reread those weights. Capacity without bandwidth still stalls long-context serving. Bandwidth without capacity forces a wider tensor split and more collective traffic.
| Card | HBM class | Bandwidth class | Status |
|---|---|---|---|
| H100 | 80 GB | about 3.35 TB/s | Pending confirmation |
| H200 | 141 GB | Hopper-class, above H100 | Pending confirmation |
| B300 | 288 GB | about 8 TB/s | Pending confirmation |
| Ascend 910C | 96 GB | Below NVLink-class peers | Pending confirmation |
| MI455X | 432 GB HBM4 | about 19.6 TB/s paper | Pending confirmation |
An illustrative decode check, not a quote: a 70B FP16 weight set near 140 GB takes about 42 ms to stream at 3.35 TB/s and about 17 ms at 8 TB/s, batch size 1, ignoring overhead. That math only shows why bandwidth moves token latency. It is not a serving SLO.
An illustrative sizing check for Llama 3.1 405B FP16, weights near 810 GB: B300-class 288 GB memory is about a 4-way tensor split, while H100-class 80 GB memory is about an 11-way split. KV cache and batch size push both counts up. Confirm the split on your framework, not on this paragraph. H200 at 141 GB sits between those two splits for 70B-class and long-context inference, which is why scenario B names H200 or B300 rather than H100 alone once the model leaves the small-batch 70B band.
Metric 3 — Scale-up fabric
Quick Answer: NVLink 4 on H100 and H200 is the mature domain of about 8 to 16 GPUs. NVLink 5 on a B300-class rack is the production 72-GPU story. UALink on MI455X is newly announced and still pending confirmation. HCCS on 910C is an 8-GPU domain; close that gap with an optical super-node, not a software flag.
Metric 3 is domain size and maturity, not the fastest number on a slide. A fast link that only spans eight GPUs still forces a scale-out network for a 72-GPU tensor domain. A paper link that spans a rack is not production until your collective library stays up under fault injection.
| Fabric | Per-GPU bandwidth class | Domain size class | Maturity |
|---|---|---|---|
| NVLink 5 (B300 / NVL72 class) | about 1.8 TB/s | about 72 GPU; rack class near 130 TB/s | Production-proven narrative |
| NVLink 4 (H100 / H200) | about 900 GB/s | about 8–16 GPU | Extremely mature |
| UALink (MI455X / Helios) | about 3.6 TB/s; rack near 260 TB/s | about 72 GPU | Newly announced — pending confirmation |
| HCCS (Ascend 910C) | about 400 GB/s | about 8 GPU; CloudMatrix 384 optical super-node narrative | Available on the compliance track |
NVL72-class and DGX-class language on scenario A means a B300-class scale-up domain, not a permission to invent a chassis part number. Helios language on MI455X means the announced UALink rack class. CloudMatrix-class language on 910C means you must budget the optical super-node if the job needs more than an eight-GPU HCCS island. All bandwidth cells remain pending confirmation against the vendor page you attach to the order.
Metric 4 — Cluster availability
Quick Answer: Availability means interrupt rate, RAS, liquid-cool operations, and checkpoint storage bandwidth, not a brochure uptime percent. A published Llama 3 405B run used 16,384 H100 GPUs over 54 days with 466 job interrupts. B300 class TDP near 1,400 W needs liquid cooling. Do not buy that card for an air-only hall.
Metric 4 is whether the cluster finishes the job. A published Llama 3 405B training account used 16,384 H100 GPUs, ran about 54 days, and recorded 466 job interrupts, of which 419 were unexpected. Cite that as a public case study, not as your service level. Your interrupt budget still has to be written into the runbook.
RAS stories differ by generation. B300 and MI455X carry a RAS-engine narrative. H100 and H200 lean on software fault tolerance that operators already know. Ascend 910C carries a domestic RAS claim with limited public proof at the same scale as that 16,384-GPU account. Ask the RFQ what happens on a single-GPU fault: does the job restart from the last checkpoint, or does the rack drain.
Cooling is part of availability. B300 class TDP near 1,400 W means liquid cooling is required. Cold plate is the default. Immersion is optional and raises capital cost. A purchase order that omits the coolant distribution unit is an air-cooled plan wearing a liquid card name.
Checkpoint cadence on large clusters is often 15 to 30 minutes. Larger HBM produces larger checkpoint files. If storage bandwidth cannot drain that file inside the cadence, the GPU domain idles. Put checkpoint target size and storage throughput on the same sheet as the accelerator count.
Metric 5 — Software migration (CUDA, CANN, ROCm)
Quick Answer: Choose CUDA when custom operators are part of the model. Choose CANN for a compliance-only Ascend 910C estate. Choose ROCm for a diversification pilot only if the team already debugs AMD stacks. Operator-library ratios are industry estimates pending confirmation: CUDA above 10,000, ROCm near half, CANN near one fifth.
Metric 5 is the cost of making the model run, not the logo on the box. H100 and H200 stay on CUDA with no migration if the estate is already NVIDIA. B300 stays on that same CUDA path, with a generation step in kernels and numerics you still have to test. Ascend 910C moves the estate onto CANN. MI455X moves it onto ROCm. Those are different projects.
| Stack | Operator library class | PyTorch path | Frontier custom ops |
|---|---|---|---|
| CUDA | 10,000+ | Native first tier | Best covered |
| CANN | about 2,000 (about 20 percent) | Adapter, not the native first path | Risk that you write operators |
| ROCm | about 5,000 (about 50 percent) | Official and improving | Middle |
Those library counts are industry estimates pending confirmation. Do not put “20 percent of CUDA” into a contract exhibit. Use them only to size the migration team.
A practical decision tree:
- Standard GPT, LLaMA, or Qwen shapes: any of the three stacks can be made to run, with very different effort.
- Frontier custom operators: stay on CUDA unless you have already staffed the missing kernels.
- Compliance-only constraint: CANN on Ascend 910C, and budget the operator gaps.
- Supplier diversification: ROCm on MI455X only with in-house AMD skill, and only as a pilot until large-cluster stability is shown.
Metrics 6 and 7 — $/Token and tokens per watt
Quick Answer: Dollars per token and tokens per watt are unit economics, not sticker TDP. Industry commentary puts a B200-class MoE path well below an H100-class path per million tokens; treat that corridor as an industry estimate pending confirmation. B300 class TDP near 1,400 W needs liquid CapEx. A 910C class TDP near 600 W can stay air-cooled.
Metric 6 is inference unit cost. Public industry commentary has described a large mixture-of-experts example where a prior Blackwell-class path lands near $0.01 per million tokens and an H100-class path lands near $0.08 to $0.11. Do not publish those corridors as a quote. They are industry estimates pending confirmation. B300 inherits the same directional story in that narrative: better unit economics than H100-class if memory and batch let the card stay busy. The direction is not a price list.
A separate public narrative compares a 1,024-GPU, three-year training total cost in local currency, with hardware and power in the hundred-million range. Restate that only as direction: newer high-memory cards can cut wall-clock and therefore power on the same token target, while liquid cooling adds capital the H100 air path may not carry. The yuan totals are industry estimates pending confirmation. They are not a TAOHAN price and not a number to paste into a budget slide without a dated RFQ.
Metric 7 refuses to stop at TDP. Class plate power runs from H100 near 700 W, through a prior Blackwell step near 1,000 W, to B300 near 1,400 W. Rack power class moves from about 50 kW toward 120 to 150 kW on an NVL72-class rack. All of those watts are pending confirmation. New AI halls often target a power usage effectiveness at or below 1.25, and some tier-1 city narratives cite 1.15. Local code is pending confirmation. Ascend 910C class TDP near 600 W, with a tokens-per-watt narrative around 65 to 70 percent of H100, is the air-coolable capital advantage versus B300 liquid. Use it when compliance, not frontier train time, is the constraint.
Scenario matrix A–E and how to shortlist
Quick Answer: Pick scenario A through E, then run six RFQ steps: load, HBM, fabric plus stack, availability, dated pricing, and a separate alternate line. Frontier pretrain is B300. Large inference is H200 or B300. Compliance is 910C. Diversification is an MI455X pilot. Mid-size train is H100. Never mix two cards on one SKU row.
| Scenario | First pick | Notes |
|---|---|---|
| A — Frontier pretrain (1T+) | B300 (NVL72 / DGX-class) | Alternate: MI455X Helios only if ROCm is already proven on your ops |
| B — Infer 70B–405B | H200 or B300 | At or below about 70B on a tight budget, H100 can still be acceptable |
| C — Domestic / compliance | Ascend 910C | Pair a CloudMatrix-class optical super-node for the fabric gap |
| D — Supplier diversification | MI455X pilot | Mass-ship narrative is late 2026 or 2027 — pending confirmation |
| E — Budget mid train, 7B–70B | H100 | Mature supply and zero CUDA migration |
How to shortlist an AI cluster GPU with seven metrics:
- Step 1. Lock the primary load: train versus infer, plus model size, context length, and whether the model is mixture-of-experts.
- Step 2. Size HBM capacity and bandwidth for weights plus KV cache at the target batch, then count tensor-parallel and pipeline-parallel nodes.
- Step 3. Lock the scale-up fabric domain, NVLink, UALink, HCCS, or an optical super-node, and the software stack: CUDA, CANN, or ROCm.
- Step 4. Score availability: liquid-cool operations, RAS, checkpoint storage bandwidth, and an interrupt budget.
- Step 5. Price only with dated RFQ lines covering hardware, power, cooling, and migration. Public soft corridors stay industry estimates pending confirmation.
- Step 6. Pick scenario A through E and freeze the alternate card as a separate RFQ line, not a mixed SKU row.
Scenario D is a pilot, not a fleet swap. Do not convert an MI455X paper bandwidth cell into a delivery commitment. Scenario C fails if the optical super-node is missing from the same purchase order as the 910C cards. Scenario A fails if liquid cooling is “phase two.”
Real-World Deployment Scenarios
Quick Answer: Map the three scenarios below to your workload, facility, and commercial constraints before freezing CapEx. Equal-BOM RFQs should cite which scenario drives the shortlist.
Scenario 1: Heterogeneous National Cloud Bid (Policy Bound)
Workload Profile: RFQ requires dual shortlists of NVIDIA and Ascend/AMD silicon.
Architectural Fit: Score all seven metrics (memory, bandwidth, interconnect, MFU, software stack, power, lead time) on one sheet—do not crown a winner on TFLOPS alone.
Scenario 2: CUDA-Locked Model Factory (Ecosystem Bound)
Workload Profile: Production stacks assume CUDA/NCCL toolchains.
Architectural Fit: H100/H200/B300 remain default unless the program funds porting and regression budgets for 910C/MI455X.
Scenario 3: Power-Capped Hall Expansion (Facility Bound)
Workload Profile: Utility feed cannot grow; efficiency per rack decides the PO.
Architectural Fit: Elevate power/cooling metrics beside raw throughput before freezing CapEx.
Frequently Asked Questions
Quick Answer: H100 still fits mature CUDA jobs at or under about 70B. B300 needs liquid cooling. Ascend 910C does not replace H100 on frontier pretrain. MI455X volume timing is pending confirmation. The lowest dollars-per-token story is directional, not a live quote. Lock train versus infer before the other six metrics.
Yes. For a new cluster in 2026, H100 still fits scenario E: mid-size training from 7B to 70B, and inference at or below about 70B when CUDA migration cost is zero. It is the wrong first pick for trillion-parameter pretrain, where B300-class memory and NVLink 5 change the node count. Price H100 on its own RFQ line if you also request B300.
Yes. A class TDP near 1,400 W makes a cold plate the default, with immersion as an optional higher-CapEx path. An air-only hall is unfit. Put the CDU or equivalent facility scope on the same RFQ as the GPU bill of materials. Treat the watt figure as a class number pending confirmation against the OEM datasheet and your purchase order.
No. Ascend 910C is not a drop-in replacement for frontier CUDA training on H100. It is the compliance track on CANN, with an HCCS domain near eight GPUs and a public narrative that puts the effective train gap near 4.5x after MFU. Pair a CloudMatrix-class optical super-node when fabric, not the card, is the bottleneck. Confirm peaks on the Huawei accelerator page before you set a pass bar.
Not on a confirmed calendar. Public narratives point to late 2026 or 2027, and that window is pending confirmation. Treat MI455X as a diversification pilot, not as a proven 10,000-GPU collective substitute. Paper HBM4 near 432 GB and UALink bandwidth are class figures, not a delivered cluster SLA. Ask the RFQ for a dated ship window and a ROCm operator test plan.
None of the soft corridors on this page is a live quote. Industry commentary has described a large mixture-of-experts case where a B200-class path sits near $0.01 per million tokens and an H100-class path sits near $0.08 to $0.11. Those corridors are industry estimates pending confirmation. B300 inherits the same directional story. Price only dated RFQ lines that include power, cooling, and migration.
Lock metric 1 first: train versus infer fit, including model size, context length, and whether the model is mixture-of-experts. HBM, fabric, availability, software stack, dollars per token, and tokens per watt all change once that load is named. Do not start from a TFLOPS leaderboard.
References
Quick Answer: Treat TFLOPS, HBM, and watt peaks as pending confirmation against the OEM page you attach to the purchase order. NVIDIA H100, H200, and NVL72 pages, plus AMD Instinct, Huawei Ascend cards, NCCL, and ROCm docs, are the external anchors. On-site GPU guides belong in Related, not in this list.
- NVIDIA — H100 Tensor Core GPU
- NVIDIA — H200 Tensor Core GPU
- NVIDIA — GB200 NVL72 rack-scale class
- AMD — Instinct accelerators
- Huawei Ascend — accelerator cards
- NVIDIA — NCCL
- AMD — ROCm documentation
- The Llama 3 Herd of Models (arxiv 2407.21783) — published 16,384-GPU training context, not a service level
Key Takeaway
Key Takeaway: Lock train versus infer, then HBM and fabric, before you chase peak TFLOPS. Scenario A is B300 with liquid cooling. Scenario E is still H100. Scenario C is Ascend 910C on CANN. Scenario D is an MI455X pilot only. Shanghai TaoHan Industrial Co., Ltd. supplies H100, H200, B300, MI455X, and Ascend 910C cluster BOMs with competitive pricing.