Part Three of Three · Article 12 of 12 · The AI Capability Series

The Professions That Never Had to Build an Immune System

A litigator, a radiologist, an auditor and a historian have something important in common. It is not a technical education, and it is not intelligence.

It is that each of them has an external standard against which a claim can be run, and a professional habit of running it. The litigator has precedent and an adversary. The radiologist has differential diagnosis. The auditor has evidence requirements and a file somebody else will inspect. The historian has source criticism. In each case there is a procedure for establishing whether an assertion holds, the procedure is explicit enough to teach, and using it is most of what the job consists of.

Now the uncomfortable part. Why do those disciplines have that apparatus when others do not? Not because they are more rigorous by temperament. Because their failures were expensive and visible. Engineering failures fall down. Medical failures kill people. Audit failures get prosecuted. Those fields built checklists, peer review, sign-off hierarchies and post-mortems because the alternative was catastrophe they could not conceal.

Fields where failure is diffuse, slow to surface or hard to attribute never had to build any of it. Much of management, strategy, marketing and internal advisory work runs on plausibility, because nobody could easily prove the advice wrong and the loop between a recommendation and its consequence is long enough that attribution rarely happens. That is not laziness. It is what a missing forcing function looks like after a century.

Then generative AI made plausibility worthless as a signal of competence, everywhere, at once.

So the exposed professions are not the ones you would expect, and they are not exposed for being less sophisticated. They are exposed because they never needed the immune system that engineering and medicine were forced to develop, and the deficiency is being tested for the first time.

Here is the bias in my own observation. I consistently see better AI output from people trained as engineers and scientists, and a machine-learning practitioner saying so is marking his own homework. Two objections are real. I may be recognising work that is legible to me rather than work that is better. And technically trained people may simply be pointing these tools at more verifiable tasks, in which case I am observing a property of the task rather than the person.

Having sat with both, I think the observation holds and the category was wrong. The variable is not engineering. It is training in a discipline that makes its verification procedure explicit, which is why litigators and historians belong in the same group as structural engineers, and why the historical anchors I keep reaching for come from aviation, cryptanalysis and medicine. Those are the fields that had to solve this.

Bletchley Park is usually told as a story about a machine that broke a code, which has it backwards. The bombe did not read German and did not produce intelligence. It narrowed an impossible search space to a set of candidates people could test. The cribs, the traffic analysis, the translation, the judgment about what a decrypt meant and what could safely be done with it all happened around the machine, inside a strict sequence, on the explicit assumption that it would never hand anybody a finished product.

Something to try this week. Take one significant category of work your team does and write a single sentence: what would have to be true for a claim in this domain to be demonstrably wrong?

If you can write it, you have most of what you need. The method designs itself from there, because you now know what the checking consists of. If you cannot write it, no tool purchase will help, and that is the more useful outcome, because you have found what to build before you deploy anything else.

That is scaffolding work of the oldest kind, which is worth remembering, because scaffolding comes down. Masons worked for years at heights that would have killed them, on timber that appears in no painting of the finished cathedral. What remains afterwards is not the absence of the scaffold. It is a building, and a workforce that knows how to raise the next one.

The long version

The Scaffolding of Intellect

Read the full paper →
← All twelve articles