TAGGEDOPERATIONS 7 POSTS

Posts about OPERATIONS.

2026.10.23 OPERATIONS

Your one-pool inference server is paying a prefill-decode interference tax it can't see on the throughput graph.

One batch posts a record tokens/sec while a single prefill chunk stalls every in-flight decode behind it. The number that pays the bill is goodput — SLO-compliant requests/sec — and it can be several-fold lower. The fix is a capacity-planning decision, not a kernel tweak.

13 min →

2026.10.23 OPERATIONS

A bad GPU can corrupt your training run for days without ever raising an error.

The mental model says a faulty accelerator throws an exception, health checks quarantine it, and a restart heals the job. Silent data corruption does none of that: it perturbs gradients with no error code, passes fleet screening, and your restart re-applies the bad update.

14 min →

2026.09.07 OPERATIONS

FIFO is the default scheduler in your inference server and the reason its tail latency is bad.

Continuous batching did not abolish first-come-first-serve — your inference server still admits requests in arrival order, so one long generation blocks every short request behind it and p99 explodes. Shortest-job-first is the tempting fix and it just moves the tail.

15 min →

2026.09.06 OPERATIONS

You are out of inference capacity because you are out of power, not GPUs.

The dashboard counts GPUs, but the breaker and the chiller count watts — and a decode phase that draws a fifth of TDP leaves a row half-empty in power terms. That headroom is oversubscribable capacity most operators never claim.

15 min →

2026.08.03 OPERATIONS

Reactive autoscaling for LLM inference is a guarantee you will miss the burst.

Threshold autoscaling assumes a new replica arrives in seconds. Loading a 70B model's weights onto a fresh GPU takes tens of seconds from SSD and over 18 minutes from a 1 Gbps network — longer than the burst it is meant to absorb. React faster and you still lose.

13 min →

2026.05.20 OPERATIONS

When to call your AI consultancy back. A decision tree.

Most teams either over-call (the consultancy quietly becomes a line item) or under-call (the harness silently rots until a customer escalation forces the conversation). Six triggers, three are routine ops you own, three usually warrant a sprint. Here is the rubric.

12 min →

2026.05.19 OPERATIONS

Three months after handoff is where AI systems quietly die.

An AI system that passed every eval at handoff can be silently below its quality budget a quarter later. The decay is not dramatic — it is three measurable signals drifting slowly under thresholds that were never calibrated for steady state. This post is what month three actually looks like, and what to watch.

11 min →