How much GPU memory does the workload need?

Start with the full working set

Model weights are only the starting point. Training also needs memory for activations, gradients, optimizer state, framework workspaces, and communication buffers. Inference removes some of those costs but introduces its own runtime allocations.

The same model can therefore require very different memory depending on precision, framework, sequence length, batch size, and whether it is trained, fine-tuned, or served.

Inference adds context and concurrency

For language-model serving, the key-value cache grows with active sequences and context length. A configuration that works for one short request may behave very differently under long contexts or concurrent users.

Capacity planning should include the target latency and concurrency—not only whether the weights fit.

• Expected context length
• Concurrent requests or batch size
• Quantization and precision
• Target latency and throughput

More capacity is not automatically more speed

A larger memory pool may enable a workload without making it faster. Memory bandwidth, tensor performance, GPU-to-GPU communication, host memory, storage delivery, and software support can all become the limiting factor.

Treat GPU memory as one part of a system profile. Once capacity is sufficient, the next question is where the workload spends its time.

Information to collect before selecting a GPU

A useful request does not need perfect benchmarks. It needs enough context to rule out the wrong platform.

• Model or application name
• Training, fine-tuning, or inference
• Precision or quantization target
• Dataset and checkpoint size
• Context length, batch size, or user count
• Single-node or scale-out requirement