AWS adds NVIDIA Multi-Instance GPU partitioning to SageMaker real-time inference
Unite.AI summarizes the November 6 launch: hardware-isolated partitions on P4d/P5/G7/G7e; AWS tests claim higher tokens/s on an H100 MIG slice versus a full A100.
Why it matters
According to Unite.AI’s November 6 coverage of the AWS announcement, SageMaker Inference now supports NVIDIA Multi-Instance GPU so multiple hardware-isolated models can share one accelerator on real-time endpoints that use inference components. Supported families include P4d (A100), P5 (H100), G7 and G7e. AWS-reported benchmarks compare a mig-3g.40gb H100 partition with a full A100-40GB on Llama-3.1-8B-Instruct, claiming higher output tokens per second and lower latency under several loads. Customers still pay for the full instance; layouts cannot be changed in place.
Key facts
- Announced November 6, 2026 per Unite.AI coverage
- MIG for real-time endpoints using inference components
- Instance families: P4d, P5, G7, G7e
- AWS claims higher tok/s on H100 MIG vs full A100 in tests
- Full-instance billing; in-place layout changes not supported