We made the GPU cheaper. Now students wait 3× longer.
To cut GPU cost, someone raised max_batch_size from 16 to 64. Throughput went up. So did tail latency.
Before rolling back, the team ran a load test at eight batch sizes. Find the cheapest setting that still meets the SLO.
What is the largest max_batch_size whose p99 latency is at most 800 ms?
Sign in to submit and join the leaderboardAn integer, e.g. 24
Root cause. Bigger batches raise throughput but every request in a batch waits for the batch’s slowest step, so tail latency climbs. p99 crosses the 800 ms SLO between 40 and 48.
- Set max_batch_size to 40: most of the cost saving, inside the SLO.
- Use continuous batching so finished sequences free their slot immediately.
Prevent it next time. Gate serving-config changes on a load test that reports p99 against the SLO, not just throughput.