AWS Tackles LLM Loading Bottlenecks with New Infrastructure
According to aws.amazon.com, Amazon Web Services has introduced a solution combining Amazon FSx for Lustre with NVIDIA GPUDirect Storage (GDS) to address large language model loading times on GPU instances. The company states that as models scale to “hundreds of billions of parameters,” model load time negatively impacts total time to first token (TTFT), describing the wait as “painful.” The new approach “reduces minutes of unproductive load time to seconds,” according to AWS.
The post focuses on AWS’s recently launched Amazon EC2 P6e and P6 instance families, powered by NVIDIA’s Blackwell architecture. According to aws.amazon.com, the flagship P6e UltraServer contains 72 NVIDIA Blackwell GPUs with 130 TB/s bisection bandwidth, 13.4 TB of HBM3e memory, and 360 petaflops of FP8 compute.
AWS explains that traditional CPU-based model loading “streams the checkpoint through CPU memory and copies weights to each GPU sequentially over PCIe,” while the new GPUDirect Storage approach pre-splits checkpoints for parallel loading. For multi-node UltraClusters, “each node in the UltraServer loads the model independently from the shared FSx for Lustre filesystem,” according to the post.
Additionally, aws.amazon.com references TurboQuant, described as recently announced technology that provides “a massive increase in context window size” through KV cache optimization.