Proof of Tech
Menu
← Field notes
◇ ARCHIVEPAGE 6 / 7 · OLDEST → NEWEST

The field notes archive.

2026.06.23 AGENTS

Tool-calling reliability plateaus at your tool descriptions, not your model.

An agent that calls the wrong tool too often looks like a model problem, so teams reach for a bigger model or a fine-tune. The wrong calls cluster where two tools look near-identical to the model — and rewriting or de-duplicating the ambiguous descriptions recovers double-digit accuracy a bigger model does not. The tool interface is a primary ceiling on tool-calling reliability, and the model's own choice margin is a usable pre-execution alarm.

→
2026.06.26 PRIVACY

Differential privacy for fine-tuning: when it earns its cost.

Fine-tune on data with PII in it and the model can be made to recite it back verbatim. DP-SGD bounds that — at a measured cost of a point or two of accuracy for a pretrained model, not the catastrophe the folklore claims. When the threat model justifies paying it.

→
2026.06.28 TRAINING

Test-time compute moved the compute-optimal pre-training point.

Chinchilla tells you the model size that minimizes loss for a training-compute budget. But a model is not trained to sit in a checkpoint — it is served, increasingly with repeated sampling and long reasoning traces. Once inference is priced into the budget, the compute-optimal point moves hard into the overtraining regime: a smaller model trained far longer beats a larger Chinchilla-optimal one at equal total cost.

→
2026.06.29 VOICE

Voice agent evals: scoring a conversation, not a transcript.

A voice agent can pass every transcript-level metric — low word error, correct answers — and still be unbearable to talk to. The caller experiences timing, turn-taking, and recovery, none of which a transcript records. How to score the conversation.

→
2026.06.30 EVAL

Constrained decoding doesn't cost you accuracy — your prompt does.

The widely-cited result that structured output hurts a model's reasoning measured something real — but not what it is quoted as measuring. Most of the drop is a non-equivalent prompt and regex-based scoring, plus a decoder that renormalizes probabilities naively. With an equivalent prompt, a distribution-aware decoder, and a schema roomy enough to hold the model's reasoning, the causal effect of the format constraint on accuracy is near zero.

→
2026.07.01 SECURITY

Weights provenance: supply-chain security for the model itself.

A tampered model passes your benchmarks — that is what the tamper is built to do — and you cannot reliably scrub a backdoor out after the fact. The only defense is knowing, and proving, that the weights were never altered. Provenance for the model as an artifact.

→
2026.07.01 SECURITY

You scan the model you receive and deploy a different one.

A clean backdoor scan on a downloaded model certifies one artifact. Quantization, distillation, merging, and fine-tuning each produce a different artifact — and a quantization-activated backdoor is engineered to be invisible in the model you scan and live in the model you serve. For the other transforms the scan misleads you either way. The only sound scan is of the post-transform bytes you actually deploy.

→
2026.07.04 TRAINING

Process reward models grade fluency, not reasoning.

A process reward model is a learned grader of reasoning steps — and what it learned to detect is confident, fluent-sounding presentation, not logical validity. Point reinforcement learning at that gap and the gap becomes the objective.

→
2026.07.05 TRAINING

Your MoE model does not route evenly.

The load-balancing loss in a mixture-of-experts model is a training-time regularizer that stops the router from collapsing — not a promise of even routing at inference. A competently trained MoE ships with deliberately skewed routing, because even routing means the experts never specialized. Expert-parallel serving sized for uniform load under-provisions the hot GPU and mis-budgets tail latency.

→
2026.07.07 VOICE BUILD NOTE

Turn-based voice RAG: the whole loop on one stack.

A mic button on a grounded-answer dialog: Whisper for transcription, the unchanged retrieval path, per-sentence TTS for playback. What broke, what it costs, and why the label says turn-based instead of realtime.

→
2026.07.08 SECURITY

Approving an agent's action is not authorizing it.

An agent workflow that pauses for human approval and then resumes looks safe — a person clicked approve. But the approval decision and the authority the resumed step runs under are two different objects, and most systems conflate them by carrying the approval as an in-band signal the resumed call trusts. That is a confused-deputy bug: a forged resume, a replayed request, or an approval granted at one gate authorizes an action it was never meant to. The fix is to stop transporting approval and start deriving authority from the trusted approval record at resume time, scoped to the exact suspended step — capabilities enforced at the tool boundary, fail-closed by default.

→
2026.07.09 TRAINING

Your learning-rate schedule silently overrides your data-curation decisions.

A quality-ascending curriculum beats random shuffling — until a decaying schedule delivers your best data exactly when the learning rate is too small to absorb it. The same coupling flips proxy-model ablations: which dataset wins depends on the schedule, not the data alone.

→
NEW ENGAGEMENT · INTAKE

Tell us about it.

The more specific you are, the more useful our first reply.

SERVICE AREA
↩ ENCRYPTED IN TRANSIT
ASK THE FIELD NOTES BETA