Supermicro, IBM and Nvidia launch KV cache storage; first-token time up to 56x faster on long-context AI

US server maker Supermicro said on Wednesday (November 5) it has launched a KV cache storage solution with IBM and Nvidia, moving reusable AI inference context out of GPU memory and into shared storage. In joint testing, time to first token on a 130,000-token prompt was up to 56 times faster. (Photo: file image of an AI data centre.)

Moving inference context out of GPU memory

The solution combines Supermicro Petascale all-flash storage servers using AMD EPYC 9005 Series processors, IBM Storage Scale software, Nvidia's Dynamo inference framework and Nvidia Spectrum-X Ethernet networking. IBM Storage Scale Erasure Coding Edition supplies the shared G4 storage tier in Nvidia's memory hierarchy, letting multiple GPU servers share the same inference context instead of recomputing it on every card.

130,000-token prompt: 32 seconds of recompute becomes 0.57 seconds

According to measurements in an IBM Redbooks reference architecture, GPU recomputation on a 130,000-token prompt took 32.14 seconds, versus 0.570 seconds using the storage-backed cache. The joint tests used one Supermicro GPU client node with four Nvidia RTX PRO 6000 GPUs connected to an eight-node Petascale storage cluster, running Dynamo v0.9.0+, vLLM v0.14.1 and the openai/gpt-oss-120b model.

Supermicro said the offering targets long-context inference and agentic workloads, where caching avoids repeated computation as prompts and conversation histories grow, cutting latency and cost. The 56-times figure is a first-token comparison under specific test conditions, and not every prompt length or model will see the same gain.

(Sources: Supermicro / PR Newswire, Unite.AI, November 5, 2026.)