AI benchmarks overlook real-world performance issues

Enterprise AI teams have spent years solving for compute, securing GPU allocations, negotiating cloud capacity, and benchmarking training throughput. The assumption embedded in that work is that the path between storage and compute will keep up. In production, that assumption increasingly does not hold.

Real traffic introduces latency spikes, network jitter, and node degradation that controlled benchmarks fail to capture, resulting in pipelines that perform well in the lab but stall in deployment. A growing response is AI data delivery, deploying an application delivery controller (ADC) or application delivery and security platform (ADSP) in front of storage as a resilient and secure control point. "Provisioning solves for capacity but not for delivery, and that is where the constraint now hides," says Hunter Smit, senior manager of product marketing at F5.

"Enterprises buy enough GPUs and enough storage, then assume the path between them will keep up, but AI traffic is bursty, highly concurrent, and random in its reads in ways ordinary storage networking was never built to absorb." The production gap benchmarks don't show standard benchmark methodology compounds the problem, says Paul Pindell, principal solutions architect for technology alliances at F5. "Benchmark testing is usually built to produce the best possible performance or security result, not the most realistic one," he says. "With S3, latency is a known factor in degrading performance, so meaningful testing has to introduce consistent latency into the path." Most benchmark environments never do that, which means the performance numbers enterprises rely on for infrastructure decisions are drawn from conditions that production systems will never replicate.

To test this assumption, F5 and MinIO conducted throughput testing under degraded network conditions. "What stood out was how quickly S3 throughput falls off once you introduce latency," Pindell says. "Even modest latency takes a real bite out of it, and as latency climbs toward long-haul distances, the degradation gets severe." The testing also showed latency mattered far more than jitter as a driver of throughput loss, which inverted what the team had expected going in.

The upshot for enterprise architects is that S3 object storage deployments cannot be designed around clean-room assumptions; they have to be engineered for the degraded network conditions they will actually face. "In AI infrastructure, people naturally focus on GPUs because they're the most visible and expensive resource," says Tanu Mutreja, senior director of product management at F5. "But in production environments, GPUs generate only as much value as the data path that feeds them." That path runs through storage, networking, databases, security, and orchestration layers, often stitched together from multiple vendors.