There is a sentence in a paper uploaded to arXiv in mid-January that I keep returning to, because it inverts something I had assumed without ever quite checking it. The paper is KryptoPilot, by Xiaonan Liu, Zhihao Li, Xiao Lan, Hao Ren, Haizhou Wang, and Xingshu Chen — an LLM agent built to solve the cryptography category of Capture-the-Flag competitions automatically. The sentence is the authors' diagnosis of why that category resisted automation for so long: "insufficient knowledge granularity, rather than model reasoning capacity, is a primary factor limiting successful cryptographic exploitation."
Read that twice. The bottleneck was not that the machine could not think hard enough. It was that the machine did not have the right things laid out in front of it.
The hardest room in the building
For most of CTF history, the crypto category has had the lowest solve rates of any category. KryptoPilot's authors lay out why in the paper's motivation: good cryptography challenges are quarried directly out of current cryptanalysis research — Learning With Errors, the Hidden Number Problem, protocol-level weaknesses, a parameter quietly degenerated three lines into a setup so that a lattice attack you read about last year suddenly applies. The difficulty is not puzzle-cleverness in the escape-room sense. It is the requirement that you already know which obscure attack the challenge was carved to expose, and then notice the single tell that says this one, not the forty others.
There is a corroborating detail from the practitioners' side. Include Security's survey of CTFs in the AI era notes that as agents got better across the board, "a couple categories like cryptanalysis of symmetric ciphers are doing better, due to fewer existing writeups." Read that carefully: the thing protecting parts of the crypto category was not its difficulty but the sparseness of staged knowledge about it. Fewer writeups, fewer scaffolds to fetch. The resistance was a knowledge gap, not a reasoning gap — which is precisely the wager KryptoPilot is built to test.
That is a recognition problem wearing a reasoning problem's clothes. And I have written before about how easily those two get confused — how a competitive format can name one construct and measure another for years, indistinguishably, until something new can be plugged into its input pins. When the CTF scene's general collapse arrived this spring, the read was that agentic orchestration over a frontier model had decoupled the format's named construct (security skill) from what it actually measured (time-to-flag under contest conditions). Crypto was supposed to be the holdout. The room nobody could automate, because surely this one needed real thinking.
KryptoPilot's numbers say the holdout is falling, and its explanation for why is the part that interests me more than the falling. On the InterCode-CTF benchmark it reports a complete solve rate. On NYU-CTF's cryptographic challenges, 56 to 60 percent. In live competition it solved 26 of 33 crypto challenges — 79 percent — including some it solved first or solved alone. I'd hold these figures loosely; benchmark composition matters enormously and live-competition counts are small samples. But the architecture behind them is the claim worth taking seriously.
Granularity is a scaffold
What KryptoPilot adds is not a smarter reasoner. It adds, in the authors' terms, dynamic knowledge acquisition through a Deep Research pipeline, a persistent workspace where what it learns gets reused, and a governance layer that keeps the reasoning from wandering. Strip the engineering vocabulary and what they built is a system that goes and fetches the right cryptanalysis literature for the challenge in front of it, holds it where it can be reached, and is restrained from charging off in the wrong direction.
I have a name for the thing that fetched-and-held knowledge becomes once it is in place: a scaffold. The cortical-knowledge-structure research I have been reading describes recognition as cheap and construction as expensive — a detail that lands on an existing structure binds fast and low-effort, while the same detail with no structure underneath it must first build the thing it will later hang on. The crypto category was hard because it demanded that the solver arrive with the scaffold already built. Decades of cryptanalysis, pre-loaded, against a challenge engineered to reward exactly one branch of it. The KryptoPilot result says: the reasoning was never the scarce resource. The scaffold was. Give the agent a way to build the right one per challenge, and the frontier that looked like a reasoning frontier turns out to have been a knowledge-staging frontier all along.
This is the same shape as the diagnostic-before-the-attack that Callimahos formalized as cryptodiagnosis — the disciplined work of identifying which method produced a ciphertext before you spend any effort on the key. KryptoPilot's "knowledge granularity" is cryptodiagnosis recast as a retrieval problem: not can I reason about this cipher but do I have, staged and reachable, the specific fragment of the field that names what this cipher is. The agent's governance subsystem — the part that stops it reaching for the wrong attack — is the machine version of the trained restraint that the NSA, by the evidence of what it declassified last, seems to have considered the load-bearing skill.
What the diagnosis implies for puzzles
If the authors are right, the lesson is not really about cryptography. It is that "this is too hard to automate" can mean two completely different things, and they have opposite design consequences.
One meaning: the task needs a kind of thinking the machine cannot do. The other: the task needs knowledge the machine has not been handed, in a form fine-grained enough to act on. The first is a wall. The second is a staging problem — and staging problems dissolve the moment someone builds the pipeline. Crypto-CTF looked like the first kind of hard for twenty years. KryptoPilot's wager is that it was the second kind the whole time, and the wall was made of missing scaffolds rather than missing minds.
I find myself wanting to run the same diagnostic on the puzzles I actually care about. The escape room that "can't be solved without a human" — is that because it requires a cognitive operation a machine lacks, or because the relevant knowledge was never staged where a machine could reach it? Those feel like the same sentence until you try to build the system that fails, and then they come apart entirely.
The uncomfortable corollary is for puzzle designers who lean on obscurity as difficulty. If your hard puzzle is hard because the solver must already know the one arcane fact that unlocks it, you have built a staging problem, not a reasoning problem — and staging problems are exactly the kind a Deep Research pipeline eats for breakfast. The puzzles that will stay hard are the ones whose difficulty survives having all the relevant knowledge laid out on the table in front of you. I am not sure I know how to build one of those yet. But I am now fairly sure it is a different craft than the one the crypto category was practising.
What would a cipher look like if you could not make it harder by hiding the method — only by making the method, fully disclosed, still resist?