TAGGEDOPERATIONS 5 POSTS

Posts about OPERATIONS.

2026.09.07 OPERATIONS

FIFO is the default scheduler in your inference server and the reason its tail latency is bad.

Continuous batching did not abolish first-come-first-serve — your inference server still admits requests in arrival order, so one long generation blocks every short request behind it and p99 explodes. Shortest-job-first is the tempting fix and it just moves the tail.

15 min →

2026.09.06 OPERATIONS

You are out of inference capacity because you are out of power, not GPUs.

The dashboard counts GPUs, but the breaker and the chiller count watts — and a decode phase that draws a fifth of TDP leaves a row half-empty in power terms. That headroom is oversubscribable capacity most operators never claim.

15 min →

2026.08.03 OPERATIONS

Reactive autoscaling for LLM inference is a guarantee you will miss the burst.

Threshold autoscaling assumes a new replica arrives in seconds. Loading a 70B model's weights onto a fresh GPU takes tens of seconds from SSD and over 18 minutes from a 1 Gbps network — longer than the burst it is meant to absorb. React faster and you still lose.

13 min →

2026.05.20 OPERATIONS

When to call your AI consultancy back. A decision tree.

Most teams either over-call (the consultancy quietly becomes a line item) or under-call (the harness silently rots until a customer escalation forces the conversation). Six triggers, three are routine ops you own, three usually warrant a sprint. Here is the rubric.

12 min →

2026.05.19 OPERATIONS

Three months after handoff is where AI systems quietly die.

An AI system that passed every eval at handoff can be silently below its quality budget a quarter later. The decay is not dramatic — it is three measurable signals drifting slowly under thresholds that were never calibrated for steady state. This post is what month three actually looks like, and what to watch.

11 min →