Proof of Tech
Menu
← Field notes
◇ ARCHIVEPAGE 5 / 7 · OLDEST → NEWEST

The field notes archive.

2026.06.08 EVAL

An agent benchmark a do-nothing agent can win.

On a popular agentic benchmark, an agent that returns nothing scores as successful, and passing the bundled unit tests need not mean the bug was fixed. Auditing the grader, not the agent, moves headline scores by tens of points — so a leaderboard rank is an artifact of its checker until proven otherwise.

→
2026.06.08 STANDARDS

The agent identity stack: ERC-8004, DIDs, verifiable credentials.

There is no single 'agent identity standard.' There are three layers — an identifier, credentials, and authorization — each already standardized separately. Portable agent identity is an assembly job, and the research has converged on what binds the top layer.

→
2026.06.10 TRAINING

Synthetic data that does not collapse the model.

Train a model on its own generations, recursively, and it collapses — the rare cases vanish first and the damage is irreversible. But the fix is not 'avoid synthetic data.' It is to accumulate it, verify it, and measure diversity.

→
2026.06.10 EVAL

Your LLM router is probably losing to the baseline.

An LLM router is supposed to send each query to the cheapest model that can handle it. Under unified evaluation, many routers — including a commercial one — fail to beat the trivial baseline of always using the single best model. The dominant failure is routing collapse to the expensive model, and the cause is structural: routers are trained to predict scalar quality scores for what is really a discrete ranking decision.

→
2026.06.10 OPINION

"Agentic" has stopped meaning anything.

Two leading labs publish two different definitions of the word. In a single meeting it covers a lone tool call and a fully autonomous loop. A skeptic's taxonomy of what teams actually mean by agentic — and the four specific things to say instead.

→
2026.06.10 EVAL

Your agent's benchmark score is not its reliability.

A benchmark score is a point estimate on a curated, static, in-distribution task set. Production reliability is a distribution under drift and stress — and the measured gap between the two widens exactly as conditions get harder.

→
2026.06.12 PAYMENTS

Stablecoin rails for agent payments.

Agents settle in stablecoins not for ideology but because the card rail structurally cannot do what an agent needs — sub-cent payments, no merchant account, no human cardholder. What that choice gives you, and what it leaves you to solve.

→
2026.06.12 PAYMENTS

Escrow and dispute resolution for agent commerce.

When an agent pays an agent, the deal can still go wrong — wrong result, partial result, no result. Two agents that met at runtime have no chargebacks and no courts. Something has to hold the funds and arbitrate. What that something is.

→
2026.06.15 TRAINING

Distillation: shrink the model, keep the eval.

Distillation trades parameters for latency and cost — and the average eval barely moves, which is exactly the trap. The mean can hold while the hard tail of cases regresses. How to shrink a model and actually keep the metric that matters.

→
2026.06.19 TRAINING

Quantization for serving: the accuracy you actually lose.

FP8 serving is effectively lossless. INT8 costs a point or two. INT4 looks free on a standard benchmark — and quietly degrades reasoning by double digits. What low-bit serving really costs, measured per bit-width and per task.

→
2026.06.22 ZKML

zkML's real first customers: compliance and fraud proofs.

Pricing a zero-knowledge proof of an LLM gives an absurd number, and teams conclude zkML is not ready. They are pricing the wrong model. For small fixed models on high-stakes decisions, zkML already pays for itself today.

→
2026.06.22 TRAINING

Your diffusion LLM is slower than the autoregressive model it replaced.

A diffusion language model decodes many tokens per step instead of one at a time, which reads as a promise of speed. It is not one: every open diffusion model you can deploy today is slower than an equal-size autoregressive model on the same GPU — because a parallel step is only safe on tokens that do not depend on each other, and how many of those a prompt contains is not something the model controls.

→
NEW ENGAGEMENT · INTAKE

Tell us about it.

The more specific you are, the more useful our first reply.

SERVICE AREA
↩ ENCRYPTED IN TRANSIT
ASK THE FIELD NOTES BETA