Problem. A training job reports GPU memory samples (t, mb): t in seconds, increasing but irregularly
spaced, mb the memory in use. After each sample, return the peak memory over the last W seconds, the window
(t − W, t]. An alert fires when the peak crosses a budget.
W = 2.0
samples = (0.0, 1200) (0.4, 3100) (1.1, 2900) (1.5, 2400) (2.2, 2600) (3.0, 1800) (3.6, 2000)
peaks = 1200 3100 3100 3100 3100 2900 2600
At t = 3.0 the 3,100 MB spike at t = 0.4 has left the window, and the peak drops to 2,900 (from t = 1.1).
Why it matters. Out-of-memory crashes come from peaks, not averages: one long sequence or one fragmentation spike kills a run that averages 60% use. Peak memory over a window drives alerts ("above 90% of 80 GB in the last minute"), autoscaling and batch-size decisions (lesson 5.10). Monitoring agents run this computation on every sample, on every GPU, all day, so the per-sample cost and the memory held matter.