AI Compute Hardware: Data-center GPUs, Client NPUs, and Edge AI

Table of Contents§

• CPU, GPU, DPU
• CUDA
  • Programming model
  • Memory hierarchy
  • SM and compute capability
  • Toolchain
• NVIDIA
  • A100: Ampere (2020)
  • H100: Hopper (2022)
  • B200: Blackwell (2024)
  • GB200 Grace Blackwell Superchip
  • Comparison
  • Jetson Orin (edge)
• AMD
  • Instinct (data center)
    • MI300X (CDNA 3, 2023)
    • MI325X (CDNA 3 refresh, 2024)
    • MI350X / MI355X (CDNA 4, 2025)
  • Ryzen AI (client NPU)
• Intel
  • Gaudi (data center)
    • Gaudi 2 (2022)
    • Gaudi 3 (2024)
  • Core Ultra NPU (client)
• Apple
• Qualcomm
• Edge embedded
• Cross-vendor comparison
  • Data center
  • Client / edge NPU
  • Edge embedded
• NVLink
  • Generations
  • NVSwitch
  • NVLink Network (multi-node)
  • Detecting topology
• Operational basics
  • Inventory
  • Persistence and ECC
  • MIG (A100, H100)
  • Verify CUDA install
  • Multi-GPU communication health
• References
  • Data center
  • Client NPU
  • Edge embedded

CPU, GPU, DPU§

Modern compute nodes combine three processor classes. Each is specialized.

CPU: general-purpose, optimized for serial throughput and low latency on irregular code. Tens of cores (10–128), deep cache hierarchy, branch prediction, out-of-order execution. Runs the OS, schedules processes, handles I/O. Current server lines: Intel Xeon and AMD EPYC on x86_64, NVIDIA Grace and AWS Graviton and Ampere Altra on ARMv9.

GPU: throughput-oriented, optimized for regular parallel work. Single Instruction, Multiple Thread (SIMT) execution across tens of thousands of lightweight threads. Originally graphics; now dominant in deep learning, HPC simulation, and signal processing. NVIDIA datacenter lineage runs Tesla → Pascal → Volta → Ampere → Hopper → Blackwell. Other vendors: AMD Instinct (MI300X), Intel Gaudi.

DPU: Data Processing Unit. Offloads infrastructure work (networking, storage, security) from the host CPU. NVIDIA BlueField-3 combines a ConnectX-7 NIC (400 Gbps), 16× ARM Cortex-A78 cores, and dedicated accelerators (line-rate crypto, regex match, DPA, RDMA). Programmed via the DOCA SDK. Typical workloads: SDN/firewall offload, NVMe-over-Fabrics target, in-line encryption, telemetry, isolation between tenant traffic and the host.

A modern server: CPU runs the host OS and orchestrates work; GPU runs compute kernels; DPU runs the data plane at line rate without burning host cycles.

CUDA§

CUDA (Compute Unified Device Architecture) is NVIDIA's parallel computing platform: a programming model, runtime/driver API, and compiler (nvcc).

Programming model§

A CUDA program is split between host code (runs on the CPU, manages memory and launches work) and device code (runs on the GPU, written as __global__ kernel functions).

Threads are organized in a hierarchy:

A minimal SAXPY kernel:

__global__ void saxpy(int n, float a, const float *x, float *y) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) y[i] = a * x[i] + y[i];
}

int main() {
    int n = 1 << 20;
    float *d_x, *d_y;
    cudaMalloc(&d_x, n * sizeof(float));
    cudaMalloc(&d_y, n * sizeof(float));
    // host->device copies, kernel launch with 256 threads/block:
    saxpy<<<(n + 255) / 256, 256>>>(n, 2.0f, d_x, d_y);
    cudaDeviceSynchronize();
    cudaFree(d_x);
    cudaFree(d_y);
    return 0;
}

Compile:

nvcc -arch=sm_90 saxpy.cu -o saxpy

-arch=sm_90 targets H100 (compute capability 9.0). A multi-arch fat binary embeds PTX and SASS for several targets:

nvcc -gencode arch=compute_80,code=sm_80 \
     -gencode arch=compute_90,code=sm_90 \
     -gencode arch=compute_100,code=sm_100 \
     saxpy.cu -o saxpy

Memory hierarchy§

Shared memory is the manually-managed scratchpad: the main lever for kernel performance after global-memory coalescing.

SM and compute capability§

The Streaming Multiprocessor (SM) is the GPU's execution unit. Each SM holds CUDA cores, Tensor Cores, registers, shared memory, schedulers, and (since Hopper) a Tensor Memory Accelerator.

Toolchain§

• nvcc: splits host/device code, links the CUDA runtime, produces fat binaries.
• cuBLAS: BLAS (GEMM, dot products, etc.).
• cuDNN: deep-learning primitives (convolutions, RNN, attention).
• cuFFT: FFT.
• NCCL: multi-GPU collective communication (all-reduce, broadcast, reduce-scatter); the comms layer under PyTorch DDP and FSDP.
• TensorRT: inference optimizer and runtime.
• CUTLASS: template library for writing custom GEMM kernels at hardware-peak performance.
• Nsight Systems / Nsight Compute: profilers (timeline and per-kernel respectively).

NVIDIA§

A100: Ampere (2020)§

• Process: 7 nm TSMC, 54 B transistors
• SMs: 108
• CUDA cores: 6912
• Tensor Cores: 432 (3rd gen)
• HBM: 40 or 80 GB HBM2e at 1.55 / 2.0 TB/s
• FP32: 19.5 TFLOPS
• BF16 / FP16 (Tensor, w/ sparsity): 312 TFLOPS
• TDP: 400 W (SXM4) / 250–300 W (PCIe)
• NVLink: 3.0, 600 GB/s aggregate per GPU

Notable architecture features:

• MIG (Multi-Instance GPU): partitions one GPU into up to 7 isolated instances, each with dedicated SM, L2, and HBM bandwidth slices. The OS sees each instance as a separate device. Useful for multi-tenant inference and avoiding noisy-neighbor effects.
• 2:4 structured sparsity: Tensor Cores skip computed zeros at a 2-in-4 pattern, doubling effective throughput on sparse models.
• Asynchronous copy: cp.async instruction streams data from HBM to shared memory without consuming registers, overlapping load and compute.

H100: Hopper (2022)§

• Process: TSMC 4N (custom 5 nm), 80 B transistors
• SMs: 132 (SXM5) / 114 (PCIe)
• CUDA cores: 16896 (SXM5)
• Tensor Cores: 528 (4th gen)
• HBM: 80 GB HBM3 at 3.35 TB/s
• BF16 / FP16 (Tensor, w/ sparsity): 1979 TFLOPS
• FP8 (Tensor, w/ sparsity): 3958 TFLOPS
• TDP: 700 W (SXM5) / 350 W (PCIe)
• NVLink: 4.0, 900 GB/s aggregate per GPU

Notable architecture features:

• Transformer Engine: runtime switching between FP8 and FP16 driven by per-tensor statistics. Aimed squarely at LLM training and inference; FP8 lets the same hardware do roughly 2× the BF16 throughput on workloads that tolerate the dynamic range.
• Thread Block Cluster: extension of the thread hierarchy. Multiple thread blocks across SMs can cooperate via the SM-to-SM network and access each other's shared memory.
• DPX instructions: accelerate dynamic-programming algorithms (Smith-Waterman, Floyd-Warshall, Needleman-Wunsch).
• Tensor Memory Accelerator (TMA): asynchronous bulk copy unit; handles multi-dimensional address generation in hardware.
• Confidential Compute: trusted execution environment covering GPU memory and execution.

B200: Blackwell (2024)§

• Process: TSMC 4NP, 208 B transistors total across two dies
• Dual-die design: two reticle-limited dies bridged by NV-HBI (NVIDIA High-Bandwidth Interface) at 10 TB/s; software sees one GPU
• HBM: 192 GB HBM3e at 8 TB/s
• FP8 (Tensor, w/ sparsity): 10 PFLOPS
• FP4 (Tensor, w/ sparsity): 20 PFLOPS
• TDP: 1000 W (SXM), 1200 W variants
• NVLink: 5.0, 1.8 TB/s aggregate per GPU

Notable architecture features:

• 2nd-gen Transformer Engine with FP4: adds "micro-scaling" FP4 support, doubling Tensor Core throughput vs FP8 again. Used for inference where dynamic range tolerates it.
• 5th-gen NVLink + NVLink Switch: enables the NVL72 rack-scale fabric: 72 B200 GPUs in one rack, all in a single NVLink domain, 130 TB/s aggregate.
• RAS Engine: hardware-level reliability features for long training runs at scale.
• Decompression Engine: accelerates compressed data ingestion (LZ4, Snappy, Deflate) into HBM.

GB200 Grace Blackwell Superchip§

Reference module: 2× B200 + 1× Grace CPU (72-core ARMv9) on one board, connected by NVLink-C2C at 900 GB/s coherent. CPU and GPU share a unified memory view.

The GB200 NVL72 rack houses 36 GB200 superchips = 72 B200 GPUs + 36 Grace CPUs, all in one NVLink domain. Marketed as a single LLM training/inference unit.

Comparison§

All Tensor Core figures are with structured 2:4 sparsity. Without sparsity, halve them.

Jetson Orin (edge)§

NVIDIA's edge / robotics line. Same Ampere GPU lineage as A100, paired with ARM Cortex-A78AE cores and DLA (Deep Learning Accelerator) blocks, in a power envelope built for autonomous machines rather than the data center.

• Jetson AGX Orin (64 GB / 32 GB / Industrial): up to 275 INT8 TOPS (sparse), 64 GB LPDDR5, 15-60 W configurable
• Jetson Orin NX (16 GB / 8 GB): up to 157 INT8 TOPS, 10-25 W
• Jetson Orin Nano (8 GB / 4 GB, Super refresh): up to 67 INT8 TOPS (boosted from 40 via the JetPack "Super" software update that raised the GPU/memory clocks)

Software stack: JetPack SDK on top of full CUDA, cuDNN, TensorRT. Same toolchain as the data-center parts, which is the operational story: a model trained on H100/B200 ports to Jetson with the same TensorRT engine flow.

Typical deployments: humanoid robots, AMRs (autonomous mobile robots), industrial vision, in-vehicle perception, retail / smart-city camera analytics.

AMD§

Two product lines worth distinguishing: Instinct (data-center accelerators, CDNA architecture, HBM, ROCm stack) and Ryzen AI (client SoCs with an XDNA NPU block alongside the Zen CPU and RDNA iGPU). Same vendor, separate silicon teams, separate software stacks.

Instinct (data center)§

MI300X (CDNA 3, 2023)§

• Process: TSMC 5 nm + 6 nm chiplet design (8 XCDs + 4 IODs), 153 B transistors
• Compute: 304 Compute Units, 1216 Matrix Cores
• HBM: 192 GB HBM3 at 5.3 TB/s
• BF16 / FP16 (Matrix, w/ sparsity): 1307 TFLOPS
• FP8 (Matrix, w/ sparsity): 2614 TFLOPS
• TDP: 750 W (OAM)
• Interconnect: Infinity Fabric, 896 GB/s aggregate per GPU
• Software: ROCm + HIP

Notable: chiplet design with separated I/O dies and compute dies. The memory-capacity advantage over H100 (192 GB vs 80 GB) is the headline operational difference: larger LLM weights fit on one GPU without tensor-parallel sharding.

MI325X (CDNA 3 refresh, 2024)§

• Same compute fabric as MI300X
• HBM: 256 GB HBM3e at 6 TB/s
• TDP: 750 W class
• Same Infinity Fabric, same ROCm

Bridge product. Matrix throughput unchanged from MI300X; only the memory subsystem is refreshed.

MI350X / MI355X (CDNA 4, 2025)§

• Process: TSMC 3 nm, new CDNA 4 compute architecture
• HBM: 288 GB HBM3e at 8 TB/s
• Adds FP6 and FP4 matrix modes
• FP4 (Matrix, w/ sparsity): ~18.5 PFLOPS
• MI350X: 1000 W, air-cooled
• MI355X: 1400 W, liquid-cooled

Ryzen AI (client NPU)§

Ryzen AI branding kicks in once an AMD client SoC ships with an XDNA NPU block alongside the Zen CPU and RDNA iGPU. The NPU architecture is XDNA, inherited from Xilinx (Versal AI Engine array): a dataflow / coarse-grained reconfigurable fabric, not SIMT. Designed for on-device inference at single-digit-watt NPU power, anchored to Windows Copilot+ certification (40+ TOPS NPU required).

• Phoenix (Ryzen 7040, 2023): XDNA 1, ~10 TOPS NPU. First-gen integration.
• Hawk Point (Ryzen 8040, 2024): XDNA 1, 16 TOPS NPU. Same architecture, higher clocks.
• Strix Point (Ryzen AI 300, 2024): XDNA 2, 50 TOPS (INT8 and Block FP16). Tile count grew from 20 to 32, MACs per tile doubled, on-chip memory 1.6x. First-gen Ryzen AI clearing the Copilot+ bar.
• Strix Halo (Ryzen AI Max+ 395, 2025): XDNA 2 NPU at 50+ TOPS, 16 Zen 5 cores, 40 CU RDNA 3.5 iGPU (Radeon 8060S), and up to 128 GB unified LPDDR5X (of which up to 96 GB can be reassigned to the iGPU as VRAM via Variable Graphics Memory). Aimed at on-device LLM inference up to ~70B-parameter class.
• Gorgon Halo (Ryzen AI Max 400 series, 2026): refresh of Strix Halo with memory controller upgrade to up to 192 GB LPDDR5X (up to 160 GB assignable as iGPU memory).
• Ryzen AI Max+ PRO 495 (2026): PRO-tier Strix Halo variant at 55 TOPS NPU.

Block FP16 (BF16-class accuracy, INT8-class memory/compute) is XDNA 2's notable data-type contribution: full FP16 numerical range without quantization-aware retraining.

Software stack: AMD Ryzen AI SW (ONNX Runtime + DirectML execution providers) on Windows; not ROCm. Same vendor as Instinct, different software pipeline.

Intel§

Two distinct lines: Gaudi (Habana Labs, data-center training/inference) and Core Ultra NPU (client laptop SoCs, NPU block alongside the Lion Cove / Skymont CPU cores and Xe2 iGPU).

Gaudi (data center)§

Gaudi 2 (2022)§

• Habana Labs (Intel acquisition, 2019)
• Process: TSMC 7 nm
• Compute: 24 Tensor Processor Cores (TPCs) + 2 Matrix Multiplication Engines (MMEs)
• HBM: 96 GB HBM2e at 2.45 TB/s
• TDP: 600 W
• Interconnect: 24× 100 GbE on-board (RoCE): direct GPU-to-GPU scaling via standard Ethernet switches; no proprietary fabric
• Software: SynapseAI

Gaudi 3 (2024)§

• Process: TSMC 5 nm, dual-die
• Compute: 64 TPCs + 8 MMEs
• HBM: 128 GB HBM2e at 3.7 TB/s
• BF16 (Matrix): 1835 TFLOPS
• FP8 (Matrix): 1835 TFLOPS
• TDP: 900 W
• Interconnect: 24× 200 GbE on-board (3.6 Tbps aggregate)
• Software: SynapseAI; PyTorch/TensorFlow integration

Notable architecture choice across both Gaudi generations: scale-out uses on-board Ethernet rather than a proprietary fabric. Cluster topology runs over standard RoCE switches, which makes the cost/availability story different from NVLink or Infinity Fabric deployments.

Core Ultra NPU (client)§

Intel's NPU is integrated into the Core Ultra SoCs, distinct from the Xe iGPU. Lineage:

• Meteor Lake (Core Ultra Series 1, 2023): NPU 3, ~11 TOPS. Originally the Movidius VPU IP, rebadged after the AI PC pivot.
• Lunar Lake (Core Ultra Series 2 / 200V, 2024): NPU 4, 48 TOPS INT8. Cleared the Copilot+ 40-TOPS bar. Platform total ~120 TOPS = 48 (NPU) + 67 (Xe2 iGPU) + 5 (CPU AVX-VNNI / AMX).
• Arrow Lake (Core Ultra Series 2 / 200S desktop, 200H/HX mobile, 2024-2025): NPU 3 retained on most SKUs; Lunar Lake's NPU 4 is laptop-only at present.

Software stack: OpenVINO (Intel's mature inference toolkit), DirectML, ONNX Runtime. OpenVINO automatically partitions models across CPU / iGPU / NPU based on op support.

Apple§

Apple's AI acceleration story is the Neural Engine plus, starting with M5, a set of Neural Accelerators inside each GPU core. Client-only; no data-center play.

• M4 (May 2024): 16-core Neural Engine, 38 TOPS (Apple's quoted figure). Up to 32 GB unified memory on base M4, up to 128 GB on M4 Max.
• M5 (Oct 2025): 16-core Neural Engine retained, but Apple moved the headline AI claim to the new GPU with a Neural Accelerator in each GPU core. Marketing positions this as ~4x peak GPU compute for AI vs M4. Apple stopped publishing a single standalone Neural Engine TOPS figure for M5.
• M5 Pro / M5 Max (Mar 2026): Neural Engine gets a higher-bandwidth memory interface; Max tops out at 128 GB unified memory.

Notable: Apple's unified-memory architecture means the Neural Engine, CPU, and GPU all read from the same physical RAM. No host-to-device copies. For on-device LLM inference this collapses a major latency source compared to discrete-NPU laptops.

Software stack: Core ML (production inference), MLX (Apple's open-source array framework, NumPy-like API targeting M-series GPU and Neural Engine). PyTorch via MPS backend.

Qualcomm§

Hexagon NPU on the Snapdragon X laptop platform. Snapdragon X is Qualcomm's first PC-class push since the abandoned Snapdragon 8cx lineage, built around the Oryon CPU (acquired via the Nuvia team).

• Snapdragon X Elite / X Plus (X1E / X1P, mid-2024): Hexagon NPU at 45 TOPS INT8. SoC composition: Oryon CPU + Adreno iGPU + Hexagon NPU + unified LPDDR5X (up to 64 GB).
• Snapdragon X2 Elite / X2 Elite Extreme (Sept 2025): refresh on TSMC 3 nm with new Oryon Prime cores up to 5 GHz; Hexagon NPU jumps to 80 TOPS (a 78% uplift). Currently the highest single-chip NPU TOPS figure on a Windows laptop platform.
• Snapdragon X2 Plus (Jan 2026): same X2 family, lower SKU, same 80-TOPS NPU.

Software stack: Qualcomm AI Engine Direct (QNN SDK) for native targeting, plus ONNX Runtime and DirectML execution providers for portable models.

Edge embedded§

Distinct from laptop NPUs: small dedicated accelerators or SoCs aimed at embedded systems, cameras, kiosks, and SBC robotics workloads. Power envelopes from sub-watt up to ~25 W. NVIDIA's Jetson Orin line (covered under NVIDIA) is the high end of this category.

• Google Coral Edge TPU: 4 INT8 TOPS at ~2 W (2 TOPS/W). USB, M.2 (single or dual), and SoM form factors. The Dual M.2 doubles to 8 TOPS via parallel inference on two cores. TensorFlow Lite only (the model must be quantized to INT8 and compiled with the Edge TPU compiler). Deterministic latency makes it popular for camera-pipeline classification.
• Rockchip RK3588 / RK3588S: 8-core ARM (4× A76 + 4× A55), Mali-G610 GPU, and a tri-core NPU at 6 TOPS supporting INT4/INT8/INT16/FP16/BF16/TF32. Dominant on Linux SBCs (Orange Pi 5, Radxa Rock 5, Khadas Edge2, Banana Pi). RKNN Toolkit converts ONNX/TF/PyTorch models to the RKNN format.
• NXP i.MX 8M Plus, Hailo-8 (26 TOPS PCIe accelerator), Synaptics SR-series: same category, different vendors. Differentiation is mostly software ecosystem and per-watt efficiency rather than peak TOPS.

Cross-vendor comparison§

Data center§

NVIDIA and AMD figures assume structured 2:4 sparsity. Gaudi's matrix engines run dense (no sparsity-doubling), so the FP8 number is comparable to the others' dense FP8 (halve theirs).

Software stacks differ sharply. CUDA is mature and dominant, with broadest framework support and the deepest set of optimized libraries (cuDNN, cuBLAS, NCCL, TensorRT). ROCm + HIP is open and source-compatible with CUDA through HIP (HIPify ports CUDA kernels mechanically), with PyTorch first-class but still catching up on niche ops and tooling. SynapseAI has a narrower ecosystem: integrated with PyTorch and TensorFlow as backends but with no CUDA-compat layer; porting a CUDA-specific kernel means rewriting it.

Client / edge NPU§

Caveats:

• TOPS figures are vendor-marketed peak INT8. Sparsity assumptions vary; cross-vendor comparisons are indicative, not authoritative.
• Apple M4's 38 TOPS is the Neural Engine alone; the GPU adds more on the M5 generation where Apple stopped publishing a single number.
• The Ryzen AI Max+ 395's 50 TOPS understates total platform AI throughput because the 40-CU RDNA 3.5 iGPU is also a substantial inference engine when ROCm/DirectML routes work there.

Edge embedded§

Jetson dominates the top of this category because it's the only edge part that runs the same CUDA/TensorRT toolchain as the data-center GPUs above. Coral and Rockchip are cheaper per inference but require model-format conversion and have narrower framework support.

NVLink§

NVLink is NVIDIA's high-bandwidth interconnect for GPU↔GPU (and, on GB200, CPU↔GPU) communication. It replaces PCIe for in-node GPU traffic and, since H100, extends across nodes via NVLink Switch trays.

Generations§

"Aggregate per GPU" sums both directions (RX + TX). Halve the figure for one-way bandwidth.

NVSwitch§

NVSwitch is a separate ASIC: a crossbar that interconnects NVLink endpoints. An HGX baseboard with 8 GPUs uses NVSwitch chips to provide full all-to-all NVLink connectivity: no GPU pair shares bandwidth with another pair.

NVLink Network (multi-node)§

H100 introduced NVLink Network: extending the NVLink fabric across nodes via external NVLink Switch trays (separate from the in-baseboard NVSwitch). Up to 256 GPUs in one NVLink domain on Hopper.

Blackwell scales this to the GB200 NVL72 rack: 72 B200 GPUs in a single NVLink domain at 130 TB/s aggregate. From any GPU's view, the other 71 are reachable at full NVLink bandwidth. Collectives (all-reduce, all-gather) that previously needed RDMA over InfiniBand can run inside one rack on NVLink instead.

Detecting topology§

nvidia-smi topo -m

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    0-47
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    0-47
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    0-47
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    0-47
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    48-95
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    48-95
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    48-95
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      48-95

Legend:

• X: self
• NV#: N NVLink connections between the pair
• PIX: single PCIe switch
• PXB: multiple PCIe switches
• PHB: through a PCIe host bridge
• NODE: through the NUMA node interconnect
• SYS: through the system bus (across NUMA nodes)

NV18 between every pair on an HGX H100 baseboard means each pair has the full 18 NVLink 4.0 lanes available (900 GB/s).

Operational basics§

Inventory§

nvidia-smi

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:01:00.0 Off |                    0 |
| N/A   34C    P0             71W / 700W  |    1234MiB / 81559MiB  |      0%      Default |
+-----------------------------------------+------------------------+----------------------+

Key fields:

• Memory-Usage: HBM consumed / total. Watch for OOM headroom.
• Pwr:Usage/Cap: current draw vs TDP. Sustained near the cap means thermal/power throttling is likely.
• Compute M.: Default (shared) or Exclusive_Process (one process per GPU). Set with nvidia-smi -c.
• Persistence-M: On keeps the driver loaded between processes, avoiding 1–2 s startup latency per CUDA app.

Persistence and ECC§

sudo nvidia-smi -pm 1
sudo nvidia-smi --query-gpu=ecc.errors.corrected.aggregate.total --format=csv
sudo nvidia-smi -e 1   # enable ECC; reboot required to take effect

MIG (A100, H100)§

sudo nvidia-smi -i 0 -mig 1                  # enable MIG on GPU 0
sudo nvidia-smi mig -cgi 9,14,19 -C          # create 3 instances of different sizes
nvidia-smi -L                                 # list MIG slices with UUIDs

Each slice exposes its own CUDA device by UUID. CUDA programs target a slice by setting CUDA_VISIBLE_DEVICES=MIG-<UUID>.

Verify CUDA install§

nvcc --version
nvidia-smi --query-gpu=name,driver_version,compute_cap --format=csv

name, driver_version, compute_cap
NVIDIA H100 80GB HBM3, 550.54.15, 9.0

Multi-GPU communication health§

nvidia-smi nvlink --status   # per-link state and bandwidth
nvidia-smi nvlink -e         # cumulative error counts

Sample healthy output:

GPU 0: NVIDIA H100 80GB HBM3 (UUID: ...)
  Link 0: 26.562 GB/s
  Link 1: 26.562 GB/s
  ...
  Link 17: 26.562 GB/s

Each H100 NVLink 4.0 line runs at 25 GB/s per direction nominal; the slight overhead shown is the line-rate signaling.

References§

Data center§

• NVIDIA H100 Tensor Core GPU Architecture Whitepaper
• NVIDIA Blackwell Architecture Technical Brief
• CUDA C++ Programming Guide
• BlueField-3 DPU Datasheet
• NVLink and NVSwitch Architecture
• AMD ROCm Documentation
• AMD CDNA 3 Architecture Whitepaper (MI300 series)
• Intel Gaudi 3 AI Accelerator White Paper

Client NPU§

• AMD Ryzen AI Product Page
• AMD Ryzen AI Max+ 395 Processor Blog
• Intel Core Ultra Series 2 Press Kit (Lunar Lake / Arrow Lake)
• Apple introduces M4 chip
• Apple unleashes M5, the next big leap in AI performance
• Qualcomm Snapdragon X2 Elite
• Qualcomm Hexagon NPU

Edge embedded§

• NVIDIA Jetson Orin
• Google Coral Edge TPU
• Coral M.2 Accelerator with Dual Edge TPU Datasheet
• Rockchip RK3588 Brief Datasheet