Proof of Tech
Menu
← FIELD NOTES
TAGGEDSECURITY 14 POSTS

Posts about SECURITY.

2026.10.01 SECURITY

Anyone with your open weights deletes the safety in one rank-one edit, no fine-tuning required.

Refusal in an aligned model lives in a handful of directions in the residual stream. Anyone holding the weights projects them out with a difference-of-means edit — no fine-tuning, no jailbreak, nothing a filter sees. Weight-level safety on an open release is unenforceable.

→
2026.10.01 SECURITY

Your fine-tuning API's moderation filter passes the poison.

Hosted fine-tuning APIs are assumed safe because the provider moderates the training data you upload. But moderation is a classifier, and poison can be optimized to be classified benign while still teaching the harmful behavior — at up to a 100% pass rate. The poison budget is small and fixed, not a detectable fraction of the dataset, so it hides in any clean corpus. Provider-side data moderation is not the control.

→
2026.10.01 SECURITY

A backdoor planted before alignment survives every alignment step you run afterward.

A behavioral backdoor planted before alignment survives supervised fine-tuning, RLHF, and adversarial training — and adversarial training can make it stealthier, not weaker. 'We ran the safety pipeline' is not a backdoor defense.

→
2026.09.26 SECURITY

Your agent is multilingual. Your safety layer is not.

Safety alignment is shaped by English data and English red-teaming. Send the same unsafe request in a low-resource language and it gets through far more often — and the defenses trained in English do not transfer to other languages either.

→
2026.08.28 SECURITY

A handful of documents owns your RAG answers.

RAG corpora ingest documents from sources no one fully trusts, on the assumption that a wrong document is just noise the retriever averages out. It is not noise. A few crafted passages — sometimes one — reliably flip the answer to a targeted question.

→
2026.08.24 SECURITY

Your chain-of-thought trace is not an audit log.

An agent's chain-of-thought reads like a record of how it reached an answer. The research says otherwise: CoT is frequently a post-hoc story, swayed by cues the model never mentions — useful to monitor, unsafe to trust as the audit log of what actually happened.

→
2026.07.16 SECURITY

Indirect prompt injection, by the numbers.

Our taxonomy argued prompt injection is a vulnerability class you contain, not a bug you fix. Here is the quantitative half — against the strongest published defenses, indirect-injection attack success stays high, and for agents that can act it stays alarming.

→
2026.07.08 SECURITY

Approving an agent's action is not authorizing it.

An agent workflow that pauses for human approval and then resumes looks safe — a person clicked approve. But the approval decision and the authority the resumed step runs under are two different objects, and most systems conflate them by carrying the approval as an in-band signal the resumed call trusts. That is a confused-deputy bug: a forged resume, a replayed request, or an approval granted at one gate authorizes an action it was never meant to. The fix is to stop transporting approval and start deriving authority from the trusted approval record at resume time, scoped to the exact suspended step — capabilities enforced at the tool boundary, fail-closed by default.

→
2026.07.01 SECURITY

You scan the model you receive and deploy a different one.

A clean backdoor scan on a downloaded model certifies one artifact. Quantization, distillation, merging, and fine-tuning each produce a different artifact — and a quantization-activated backdoor is engineered to be invisible in the model you scan and live in the model you serve. For the other transforms the scan misleads you either way. The only sound scan is of the post-transform bytes you actually deploy.

→
2026.07.01 SECURITY

Weights provenance: supply-chain security for the model itself.

A tampered model passes your benchmarks — that is what the tamper is built to do — and you cannot reliably scrub a backdoor out after the fact. The only defense is knowing, and proving, that the weights were never altered. Provenance for the model as an artifact.

→
2026.05.16 SECURITY

Auditing an agent that holds a wallet.

Agents now sign transactions. The attack surface — a prompt injection that ends in a signed transfer — is new, and almost no security auditor covers it. What an agent security audit actually checks.

→
2026.04.25 SECURITY

Signing-key custody for autonomous agents.

Assume the model gets injected — then ask where the signing key lives. MPC, HSMs, multisig, and session keys, judged on one question: can a fully compromised agent reach the key?

→
2026.04.03 SECURITY

Red-teaming an MCP server.

Everyone audits the agent. Almost nobody audits the servers it calls — and an MCP server writes straight into your model's context. This is the supply side of agent security.

→
2026.03.13 SECURITY

Prompt injection is a vulnerability class, not a bug.

You do not patch prompt injection any more than you patched SQL injection. It is a vulnerability class with four members, and each one needs a different architectural defense.

→
NEW ENGAGEMENT · INTAKE

Tell us about it.

The more specific you are, the more useful our first reply.

SERVICE AREA
↩ ENCRYPTED IN TRANSIT
ASK THE FIELD NOTES BETA