Proof of Tech
Menu
← FIELD NOTES
TAGGEDEVAL 13 POSTS

Posts about EVAL.

2026.06.30 EVAL

Constrained decoding doesn't cost you accuracy — your prompt does.

The widely-cited result that structured output hurts a model's reasoning measured something real — but not what it is quoted as measuring. Most of the drop is a non-equivalent prompt and regex-based scoring, plus a decoder that renormalizes probabilities naively. With an equivalent prompt, a distribution-aware decoder, and a schema roomy enough to hold the model's reasoning, the causal effect of the format constraint on accuracy is near zero.

→
2026.06.10 EVAL

Your agent's benchmark score is not its reliability.

A benchmark score is a point estimate on a curated, static, in-distribution task set. Production reliability is a distribution under drift and stress — and the measured gap between the two widens exactly as conditions get harder.

→
2026.06.10 EVAL

Your LLM router is probably losing to the baseline.

An LLM router is supposed to send each query to the cheapest model that can handle it. Under unified evaluation, many routers — including a commercial one — fail to beat the trivial baseline of always using the single best model. The dominant failure is routing collapse to the expensive model, and the cause is structural: routers are trained to predict scalar quality scores for what is really a discrete ranking decision.

→
2026.06.08 EVAL

An agent benchmark a do-nothing agent can win.

On a popular agentic benchmark, an agent that returns nothing scores as successful, and passing the bundled unit tests need not mean the bug was fixed. Auditing the grader, not the agent, moves headline scores by tens of points — so a leaderboard rank is an artifact of its checker until proven otherwise.

→
2026.05.27 EVAL

GraphRAG or vector RAG: a decision guide.

A knowledge graph earns its build-and-maintenance cost over plain vector retrieval only for specific query classes — multi-hop reasoning and global summarization. For fact lookup it loses, and it always costs more to index. A guide by query type.

→
2026.05.25 EVAL

Hybrid search: when BM25 still beats your embeddings.

Dense embeddings win in-domain and on paraphrase. Lexical BM25 wins on rare exact terms and on corpora the embedding model never trained on. Production retrieval usually needs both — and a clear reason for the split.

→
2026.05.22 EVAL

Reranking is the cheapest RAG win you are not using.

A cross-encoder reranker often beats retrieving more documents — it scores the query and passage together, where vector search only compared two embeddings made apart. But it scales linearly with candidates and does not always earn its latency. When to add one.

→
2026.05.20 EVAL

Eval-driven development: write the eval before the feature.

An eval written after a feature ships can only ratify what the feature already does. Written first, the graded set becomes the specification — it forces the ambiguity out of the requirement before any code exists. Why the eval is the real spec.

→
2026.05.16 EVAL

Stop OCR-ing your PDFs: retrieve on the page, not the transcript.

Most document RAG OCRs a PDF, rebuilds the layout, and chunks the text — losing every table, chart, and column it touches. ColPali-class visual retrieval embeds the page image directly. When that wins, when it doesn't.

→
2026.05.16 EVAL

Faithfulness is not groundedness. And "accuracy" is not a RAG metric.

Teams say their RAG is 'accurate' and mean different things. Faithfulness is whether the answer is true; groundedness is whether every claim traces to a source. They fail differently — and a deploy depends on measuring both.

→
2026.04.29 EVAL

Your golden set is rotting.

A golden evaluation set is not a fixed asset — it decays. The world changes, the product shifts, the team overfits, and the pass rate quietly stops meaning anything. Eval data needs a maintenance protocol.

→
2026.04.22 EVAL

You don't have a RAG problem. You have a chunking problem.

Most teams blame the retriever. The retriever is fine. Your chunks don't carry their context — and no amount of reranking saves them.

→
2026.03.30 EVAL

LLM-as-judge is a model you also have to evaluate.

Teams wire an LLM into the eval harness as the judge and treat its scores as ground truth. But the judge is a model — with measurable biases, shaky calibration, and silent drift. Evaluate it before you trust it to gate a deploy.

→
NEW ENGAGEMENT · INTAKE

Tell us about it.

The more specific you are, the more useful our first reply.

SERVICE AREA
↩ ENCRYPTED IN TRANSIT
ASK THE FIELD NOTES BETA