Engineering notes from the field. Production AI and verifiable on-chain inference, written by the engineers who ship them.https://proofoftech.org/en-ushttps://proofoftech.org/blog/prompt-canaries-rollback/https://proofoftech.org/blog/prompt-canaries-rollback/A one-word change to a system prompt can move accuracy by dozens of points, and a provider's model update can regress your app overnight. A prompt or model swap is a deploy. Give it a staged rollout and a one-action rollback path.Wed, 03 Jun 2026 00:00:00 GMTAGENTSProof of Techhttps://proofoftech.org/blog/agent-fleet-cost-observability/https://proofoftech.org/blog/agent-fleet-cost-observability/The monthly inference bill arrives as one number, and nobody can say which agent, which customer, or which tool spent it. Agent cost is too variable to estimate and has to be attributed after the fact — per run, per tool, per tenant. The layer most stacks skip.Mon, 01 Jun 2026 00:00:00 GMTAGENTSProof of Techhttps://proofoftech.org/blog/human-in-the-loop-checkpoints/https://proofoftech.org/blog/human-in-the-loop-checkpoints/An agent that asks permission for everything trains its reviewers to rubber-stamp, and the one dangerous action slips through in the noise. Approval gates belong on consequence and on uncertainty — not on every step. Where to put them.Fri, 29 May 2026 00:00:00 GMTAGENTSProof of Techhttps://proofoftech.org/blog/graphrag-vs-vector-rag/https://proofoftech.org/blog/graphrag-vs-vector-rag/A knowledge graph earns its build-and-maintenance cost over plain vector retrieval only for specific query classes — multi-hop reasoning and global summarization. For fact lookup it loses, and it always costs more to index. A guide by query type.Wed, 27 May 2026 00:00:00 GMTEVALProof of Techhttps://proofoftech.org/blog/hybrid-search-bm25/https://proofoftech.org/blog/hybrid-search-bm25/Dense embeddings win in-domain and on paraphrase. Lexical BM25 wins on rare exact terms and on corpora the embedding model never trained on. Production retrieval usually needs both — and a clear reason for the split.Mon, 25 May 2026 00:00:00 GMTEVALProof of Techhttps://proofoftech.org/blog/reranking-rag/https://proofoftech.org/blog/reranking-rag/A cross-encoder reranker often beats retrieving more documents — it scores the query and passage together, where vector search only compared two embeddings made apart. But it scales linearly with candidates and does not always earn its latency. When to add one.Fri, 22 May 2026 00:00:00 GMTEVALProof of Techhttps://proofoftech.org/blog/eval-driven-development/https://proofoftech.org/blog/eval-driven-development/An eval written after a feature ships can only ratify what the feature already does. Written first, the graded set becomes the specification — it forces the ambiguity out of the requirement before any code exists. Why the eval is the real spec.Wed, 20 May 2026 00:00:00 GMTEVALProof of Techhttps://proofoftech.org/blog/when-to-call-the-consultancy-back/https://proofoftech.org/blog/when-to-call-the-consultancy-back/Most teams either over-call (the consultancy quietly becomes a line item) or under-call (the harness silently rots until a customer escalation forces the conversation). Six triggers, three are routine ops you own, three usually warrant a sprint. Here is the rubric.Wed, 20 May 2026 00:00:00 GMTOPERATIONSProof of Techhttps://proofoftech.org/blog/three-months-after-handoff/https://proofoftech.org/blog/three-months-after-handoff/An AI system that passed every eval at handoff can be silently below its quality budget a quarter later. The decay is not dramatic — it is three measurable signals drifting slowly under thresholds that were never calibrated for steady state. This post is what month three actually looks like, and what to watch.Tue, 19 May 2026 00:00:00 GMTOPERATIONSProof of Techhttps://proofoftech.org/blog/tool-design-for-agents/https://proofoftech.org/blog/tool-design-for-agents/A tool's name, description, and JSON schema are the only thing the model sees — it picks and fills tools from those words alone. Treating that surface as API plumbing instead of as prompt is why agents call the wrong tool. How to design it.Mon, 18 May 2026 00:00:00 GMTAGENTSProof of Techhttps://proofoftech.org/blog/agent-memory-microservice/https://proofoftech.org/blog/agent-memory-microservice/"Add memory" usually means "dump everything into a vector DB." Real agent memory is tiered, has a write/forget/consolidate policy, and runs as a service the agent calls — not a library bolted into the loop.Sat, 16 May 2026 00:00:00 GMTAGENTSProof of Techhttps://proofoftech.org/blog/agent-observability/https://proofoftech.org/blog/agent-observability/Request traces and dashboards were built for request/response services. The ways an agent fails — a tool returning 200 with garbage, a truncated context, a looping planner — trip none of them. What agent-native observability has to capture.Sat, 16 May 2026 00:00:00 GMTAGENTSProof of Techhttps://proofoftech.org/blog/agent-spend-rails/https://proofoftech.org/blog/agent-spend-rails/x402, ERC-8004, and AP2 gave agents the rails to hold and spend money. The controls that stop a prompt-injected agent from draining a wallet — spend ceilings, treasury isolation, circuit breakers — did not ship with them.Sat, 16 May 2026 00:00:00 GMTPAYMENTSProof of Techhttps://proofoftech.org/blog/auditing-agent-wallets/https://proofoftech.org/blog/auditing-agent-wallets/Agents now sign transactions. The attack surface — a prompt injection that ends in a signed transfer — is new, and almost no security auditor covers it. What an agent security audit actually checks.Sat, 16 May 2026 00:00:00 GMTSECURITYProof of Techhttps://proofoftech.org/blog/colpali-document-retrieval/https://proofoftech.org/blog/colpali-document-retrieval/Most document RAG OCRs a PDF, rebuilds the layout, and chunks the text — losing every table, chart, and column it touches. ColPali-class visual retrieval embeds the page image directly. When that wins, when it doesn't.Sat, 16 May 2026 00:00:00 GMTEVALProof of Techhttps://proofoftech.org/blog/decentralized-training-verification/https://proofoftech.org/blog/decentralized-training-verification/Decentralized pretraining now reaches into the tens of billions of parameters — but you still cannot cryptographically prove the GPUs did the work they claim. How production networks check untrusted workers, and why ZK-proven training is years out.Sat, 16 May 2026 00:00:00 GMTTRAININGProof of Techhttps://proofoftech.org/blog/deepprove-first-proven-llm/https://proofoftech.org/blog/deepprove-first-proven-llm/DeepProve, from Lagrange, produced the first zero-knowledge proof of a full LLM inference — GPT-2. It moves "prove a transformer" from impossible to merely expensive. What that unlocks, and what is still years away.Sat, 16 May 2026 00:00:00 GMTZKMLProof of Techhttps://proofoftech.org/blog/faithfulness-vs-groundedness/https://proofoftech.org/blog/faithfulness-vs-groundedness/Teams say their RAG is 'accurate' and mean different things. Faithfulness is whether the answer is true; groundedness is whether every claim traces to a source. They fail differently — and a deploy depends on measuring both.Sat, 16 May 2026 00:00:00 GMTEVALProof of Techhttps://proofoftech.org/blog/mcp-production-gaps/https://proofoftech.org/blog/mcp-production-gaps/MCP won the tool-integration standard. But "works in a demo" and "works in production" are different claims — and four gaps bite at scale: sticky sessions, server fan-out, governance, and what happens when a session drops mid-task.Sat, 16 May 2026 00:00:00 GMTSTANDARDSProof of Techhttps://proofoftech.org/blog/opml-or-zkml/https://proofoftech.org/blog/opml-or-zkml/Two ways to make an off-chain model output trustworthy on-chain. zkML is cryptographic, expensive, and small-model-only. opML is optimistic, cheap, and runs Llama-2-scale models today. Choosing by stakes, model size, and latency.Sat, 16 May 2026 00:00:00 GMTZKMLProof of Techhttps://proofoftech.org/blog/cascaded-vs-end-to-end-speech/https://proofoftech.org/blog/cascaded-vs-end-to-end-speech/A 2026 voice agent forks at the first design decision — STT→LLM→TTS cascade, or a single speech-to-speech model. End-to-end wins on latency and naturalness; the cascade wins on everything you debug, audit, and control. Here is the trade study.Fri, 15 May 2026 00:00:00 GMTVOICEProof of Techhttps://proofoftech.org/blog/deterministic-replay-agents/https://proofoftech.org/blog/deterministic-replay-agents/An agent run is non-deterministic — sampling, tool responses, and timing all vary — so a bug seen once may never recur. Deterministic replay records every non-deterministic input so the run can be replayed exactly.Wed, 13 May 2026 00:00:00 GMTAGENTSProof of Techhttps://proofoftech.org/blog/pricing-agent-consumable-api/https://proofoftech.org/blog/pricing-agent-consumable-api/When the buyer is an autonomous agent paying per call, human pricing breaks. Seats, signup-gated free tiers, and annual commitments stop making sense for a machine that reads the price and comparison-shops every call.Mon, 11 May 2026 00:00:00 GMTPAYMENTSProof of Techhttps://proofoftech.org/blog/rl-environments-new-dataset/https://proofoftech.org/blog/rl-environments-new-dataset/Post-training has shifted from supervised fine-tuning on static labeled data toward reinforcement learning, and that moves the unit of data work from a labeled file to an executable environment. Building good environments is the new data engineering — and the scarce input.Sat, 09 May 2026 00:00:00 GMTTRAININGProof of Techhttps://proofoftech.org/blog/context-engineering-agents/https://proofoftech.org/blog/context-engineering-agents/For a long-running agent, the system prompt is a small part of the problem. The real discipline is managing the context window across the whole run as a budget — keep, drop, compact, retrieve.Thu, 07 May 2026 00:00:00 GMTAGENTSProof of Techhttps://proofoftech.org/blog/erc-8004-agent-identity/https://proofoftech.org/blog/erc-8004-agent-identity/Agents need to prove who they are to each other without going through a central directory. ERC-8004 is the first standard that ships the three registries needed for that. Here is what it does and what it does not.Tue, 05 May 2026 00:00:00 GMTSTANDARDSProof of Techhttps://proofoftech.org/blog/voice-graceful-failure/https://proofoftech.org/blog/voice-graceful-failure/A voice call is real-time and unforgiving — there is no spinner to show, and dead air reads as a broken product. When STT, the LLM, or TTS fails mid-call, the system has to degrade, not drop.Sat, 02 May 2026 00:00:00 GMTVOICEProof of Techhttps://proofoftech.org/blog/golden-set-maintenance/https://proofoftech.org/blog/golden-set-maintenance/A golden evaluation set is not a fixed asset — it decays. The world changes, the product shifts, the team overfits, and the pass rate quietly stops meaning anything. Eval data needs a maintenance protocol.Wed, 29 Apr 2026 00:00:00 GMTEVALProof of Techhttps://proofoftech.org/blog/signing-key-custody-agents/https://proofoftech.org/blog/signing-key-custody-agents/Assume the model gets injected — then ask where the signing key lives. MPC, HSMs, multisig, and session keys, judged on one question: can a fully compromised agent reach the key?Sat, 25 Apr 2026 00:00:00 GMTSECURITYProof of Techhttps://proofoftech.org/blog/chunking-not-rag/https://proofoftech.org/blog/chunking-not-rag/Most teams blame the retriever. The retriever is fine. Your chunks don't carry their context — and no amount of reranking saves them.Wed, 22 Apr 2026 00:00:00 GMTEVALProof of Techhttps://proofoftech.org/blog/a2a-vs-mcp/https://proofoftech.org/blog/a2a-vs-mcp/A2A and MCP get framed as rivals. They are not. MCP connects an agent to its tools; A2A connects agents to each other — different jobs at different layers, and a serious multi-agent system needs both.Tue, 21 Apr 2026 00:00:00 GMTSTANDARDSProof of Techhttps://proofoftech.org/blog/pii-redaction-retrieval/https://proofoftech.org/blog/pii-redaction-retrieval/Stripping PII before documents reach the embedding model is often necessary. But naive redaction destroys the semantic signal retrieval depends on. How to redact without wrecking retrieval.Fri, 17 Apr 2026 00:00:00 GMTPRIVACYProof of Techhttps://proofoftech.org/blog/multi-agent-one-too-many/https://proofoftech.org/blog/multi-agent-one-too-many/Splitting a task across coordinating agents adds context-handoff loss, compounding latency and cost, and a wider failure surface — overhead that usually exceeds the benefit. Start with one agent.Tue, 14 Apr 2026 00:00:00 GMTAGENTSProof of Techhttps://proofoftech.org/blog/decentralized-training-2026/https://proofoftech.org/blog/decentralized-training-2026/A grounded look at distributed pretraining across untrusted GPUs. DiLoCo, DisTrO, INTELLECT-2, Bittensor's Templar, 0G's DiLoCoX — what each actually shipped, and what hasn't.Fri, 10 Apr 2026 00:00:00 GMTTRAININGProof of Techhttps://proofoftech.org/blog/voice-turn-taking/https://proofoftech.org/blog/voice-turn-taking/Transcription is largely solved. Knowing when the caller has finished, when to stop for an interruption, and when an 'mm-hm' is not a turn — that is not. Endpointing, barge-in, and backchannels, measured.Tue, 07 Apr 2026 00:00:00 GMTVOICEProof of Techhttps://proofoftech.org/blog/red-teaming-mcp-servers/https://proofoftech.org/blog/red-teaming-mcp-servers/Everyone audits the agent. Almost nobody audits the servers it calls — and an MCP server writes straight into your model's context. This is the supply side of agent security.Fri, 03 Apr 2026 00:00:00 GMTSECURITYProof of Techhttps://proofoftech.org/blog/llm-as-judge-evaluation/https://proofoftech.org/blog/llm-as-judge-evaluation/Teams wire an LLM into the eval harness as the judge and treat its scores as ground truth. But the judge is a model — with measurable biases, shaky calibration, and silent drift. Evaluate it before you trust it to gate a deploy.Mon, 30 Mar 2026 00:00:00 GMTEVALProof of Techhttps://proofoftech.org/blog/confidential-rag/https://proofoftech.org/blog/confidential-rag/Most private RAG protects the user's query in transit and leaves the corpus exposed. But the corpus is the sensitive asset — the embeddings, the vector store, and the chunks the model sees all need protecting.Wed, 25 Mar 2026 00:00:00 GMTPRIVACYProof of Techhttps://proofoftech.org/blog/which-zkml-ships/https://proofoftech.org/blog/which-zkml-ships/EZKL, Modulus, Giza, Ora, RISC Zero. Same model, same input, same target chain. Proof times, gas costs, gotchas — and the one we'd put in front of a customer.Sun, 22 Mar 2026 00:00:00 GMTZKMLProof of Techhttps://proofoftech.org/blog/folding-schemes-zkml/https://proofoftech.org/blog/folding-schemes-zkml/zkML cannot scale to large models because proving a whole computation in one shot is ruinously expensive. Folding schemes — Nova and its lineage — prove a long, repetitive computation step by step instead. Explained without the cryptography.Tue, 17 Mar 2026 00:00:00 GMTZKMLProof of Techhttps://proofoftech.org/blog/prompt-injection-vulnerability-class/https://proofoftech.org/blog/prompt-injection-vulnerability-class/You do not patch prompt injection any more than you patched SQL injection. It is a vulnerability class with four members, and each one needs a different architectural defense.Fri, 13 Mar 2026 00:00:00 GMTSECURITYProof of Techhttps://proofoftech.org/blog/agent-budgets/https://proofoftech.org/blog/agent-budgets/An agent that hits a wall and asks for more compute is not reasoning. It is panicking. The budget is part of the spec, not a fallback.Sun, 08 Mar 2026 00:00:00 GMTAGENTSProof of Techhttps://proofoftech.org/blog/x402-economy/https://proofoftech.org/blog/x402-economy/HTTP 402 is no longer reserved. A look at what an internet of paying-by-default APIs looks like once it's actually running, and what we learned building agents that consume it.Sat, 28 Feb 2026 00:00:00 GMTPAYMENTSProof of Techhttps://proofoftech.org/blog/280ms-budget/https://proofoftech.org/blog/280ms-budget/Sub-300ms voice agents are a specific engineering problem. Here is every millisecond a packet spends between the user's mouth and the agent's reply — and where you actually claw the time back.Sun, 15 Feb 2026 00:00:00 GMTVOICEProof of Techhttps://proofoftech.org/blog/agent-latency-math/https://proofoftech.org/blog/agent-latency-math/A 4-second agent looks great on stage and falls over in production. The demo has a few tricks. Once you see them, the latency claims of every other framework get a lot less impressive.Fri, 30 Jan 2026 00:00:00 GMTOPINIONProof of Techhttps://proofoftech.org/blog/fhe-vs-tee-for-ml/https://proofoftech.org/blog/fhe-vs-tee-for-ml/Two ways to compute on data you can't see. One is cryptographically pure and 100,000x slower; the other is fast and depends on a chip vendor not being broken. A decision tree.Sun, 18 Jan 2026 00:00:00 GMTPRIVACYProof of Techhttps://proofoftech.org/blog/onchain-agents-vs-agents-that-touch-chains/https://proofoftech.org/blog/onchain-agents-vs-agents-that-touch-chains/Four levels of agent-chain integration, three of which are conflated in every pitch deck. A short field guide to what people actually mean when they say "on-chain agent."Sat, 20 Dec 2025 00:00:00 GMTOPINIONProof of Tech