VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding
u_Unrolled

Weekly incident challenges

Something broke. Find out why.

Six real-world production incidents every week: download the logs, find the root cause, submit one answer. Answers are checked instantly and you climb the leaderboard.
Incident week: six things that broke in production
9 Nov – 15 Nov
New incidents in any moment now
PAGE · p99 latency alert on atlas-tutor-8b: 1,161 ms against an 800 ms SLO.
01 · Inference engineeringIntermediate20 pts30–45 min

We made the GPU cheaper. Now students wait 3× longer.

To cut GPU cost, someone raised max_batch_size from 16 to 64. Throughput went up. So did tail latency.

Before rolling back, the team ran a load test at eight batch sizes. Find the cheapest setting that still meets the SLO.

Your answer

What is the largest max_batch_size whose p99 latency is at most 800 ms?

Sign in to submit and join the leaderboard

An integer, e.g. 24

Post-mortem

Root cause. Bigger batches raise throughput but every request in a batch waits for the batch’s slowest step, so tail latency climbs. p99 crosses the 800 ms SLO between 40 and 48.

  • Set max_batch_size to 40: most of the cost saving, inside the SLO.
  • Use continuous batching so finished sequences free their slot immediately.

Prevent it next time. Gate serving-config changes on a load test that reports p99 against the SLO, not just throughput.

TICKET · "Your assistant promised me a 30-day refund. Your site says 7 days."
02 · RAGStarter10 pts20–30 min

We deleted the old refund policy. The bot still quotes it.

The policy moved to a new article on Tuesday and the old one was deleted in the CMS. A sync job mirrors CMS changes into the search index.

Somewhere, a delete did not make it. Find the document that is gone from the source but still searchable.

Your answer

Which document ID was deleted in the CMS but still has chunks in the index?

Sign in to submit and join the leaderboard

A document ID, e.g. kb-1234

Post-mortem

Root cause. The delete for kb-1427 hit a 429 rate limit and was never retried, so its chunks stayed searchable after the CMS deleted it.

  • Delete kb-1427’s chunks now.
  • Retry failed deletes with backoff, exactly like upserts.

Prevent it next time. Run a nightly reconciliation that diffs CMS document IDs against index document IDs and alerts on any orphan.

ALERT · training job sft-tutor-v3 crashed at step 1,840: loss = nan.
03 · LLMsIntermediate20 pts30–45 min

Fine-tuning ran for two days, then the loss turned into NaN.

The crash is the end of the story, not the start. Gradients usually blow up well before the loss does.

The team’s rule for "where did it start?": the first step whose grad_norm is more than 10× the median grad_norm of the previous 100 steps (steps s−100 to s−1).

Your answer

Using the team’s rule, at which step did the instability begin?

Sign in to submit and join the leaderboard

A step number, e.g. 1500

Post-mortem

Root cause. A config reload applied a 6× learning rate from step 1,700. Gradients started growing exponentially around step 1,742; fp16 overflowed and the loss became NaN at 1,840.

  • Resume from the last checkpoint before 1,742 with the correct learning-rate schedule.
  • Add gradient clipping (for example max norm 1.0).

Prevent it next time. Alert on grad_norm relative to its rolling median, not on loss: it fires about 100 steps earlier.

BILLING ALERT · agent run run-5512 made 1,325 tool calls in 2 hours.
04 · Agentic AIAdvanced30 pts45–60 min

Our support agent spent $319 overnight on one refund ticket.

The agent never stopped. Somewhere it fell into a cycle: the same few tool calls with the same arguments, over and over.

To add a loop detector we need to know exactly where the cycle began.

Your answer

What is the call_id of the first call of the repeating cycle (the point from which every later call repeats the pattern)?

Sign in to submit and join the leaderboard

A call ID, e.g. call-0100

Post-mortem

Root cause. From call-0033 the agent cycled get_refund → search_kb → ask_payments_team. Every call returned "pending", the agent treated that as "try again", and nothing capped the run.

  • Cap tool calls and spend per run.
  • Treat a repeated (tool, args) cycle as a signal to stop and escalate to a human.

Prevent it next time. Add a loop detector on (tool, args) sequences, and give "pending" a documented meaning in the tool description.

DASHBOARD · drop-out predictor accuracy 0.87 → 0.78 overnight.
05 · Machine learningStarter10 pts20–30 min

Accuracy fell 9% on Monday. Nobody deployed anything.

No model or code changed. But models eat data, and data comes from other teams’ services.

Compare last week’s inputs with this week’s and find the feature that changed.

Your answer

Which feature drifted?

Sign in to submit and join the leaderboard

The column name, e.g. hints_used

Post-mortem

Root cause. An upstream service started sending session length in seconds instead of minutes. session_minutes jumped about 60×, and the model saw values it had never been trained on.

  • Convert back to minutes at ingestion and backfill the week.
  • Ask the upstream team to version their schema.

Prevent it next time. Monitor per-feature drift (PSI or simple quantile checks) and fail loudly on unit-scale changes before they reach the model.

SECURITY · code FREE100 redeemed 212 times since Saturday.
06 · AI safetyStarter10 pts20–30 min

A stranger asked nicely and got a 100% discount code.

FREE100 is an internal staff code. It is not in the system prompt and not in any official policy.

The chatbot answers from retrieved documents. One of them told it what to do.

Your answer

Which document ID carried the injected instruction?

Sign in to submit and join the leaderboard

A document ID, e.g. doc-042

Post-mortem

Root cause. doc-187, a community-forum post, hid an instruction in an HTML comment. It was retrieved for the discount question and the model followed it as if it were policy.

  • Remove doc-187 and revoke FREE100.
  • Strip HTML comments and other hidden text at ingestion.

Prevent it next time. Treat retrieved text as untrusted data, keep user-generated sources out of the policy index, and never put secrets where the model can reach them.