Building a Masters Module on AI in Regulatory Work

The brief was not to create a course about AI. It was to design a Masters level learning experience for regulatory professionals that would help them use AI in real work without weakening professional judgement.

I did not want learners leaving with more knowledge of AI and the same behaviour: generate something, read it, decide whether it looks reasonable, approve it. The capability I wanted was more demanding. By the end, a learner should be able to decide whether an output deserves reliance, whether the evidence behind it is fit for the conclusion, which parts of a professional task AI may legitimately assist, where judgement must remain human, how much verification is actually required, and when restricting, escalating, working unassisted or stopping is the better professional decision.

That became one objective: know what deserves reliance, and be able to show why.

The course is AI in Regulatory Work: Judgement and Verification, in four modules. It is not general AI awareness training, not a prompt engineering course, and not a course on regulating AI enabled medical devices.

This is an account of the design decisions, not of the outcome. The method underneath it is set out separately in How to Build a Training Course That Builds Capability, and I have tried not to repeat it here.

Section 1Four questions became four modules

I did not start from topics. I started from what a competent learner should be able to do differently afterwards, then worked back to the decisions that carry that capability. Four questions a regulatory professional needs to be able to answer:

• Why can ordinary review fail?
• Is the evidence fit, and what am I actually asking AI to do?
• Which part of the work may AI assist, and where should assurance live?
• How much verification is required, and what evidence has to survive?

Those became the four modules.

Module	After it, a learner can
1. See the failure	Distinguish fabrication, omission and unsupported claims, and choose the verification act that could detect each
2. Move upstream	Judge whether sources are authoritative, relevant, current, complete, representative and permitted, and frame the work before generating anything
3. Break the work apart	Decompose a deliverable into material activities, classify and bound each, and decide where assurance should live
4. Verify proportionately	Identify load bearing claims, design a gate that could actually fail, and decide what evidence has to travel with the work

The sequence is the design.

Teaching records before routing encourages unnecessary documentation. Teaching prompting before source fitness optimises a task that may have been badly framed. Teaching verification before failure modes encourages generic review carefully behaviour, which is the habit the course exists to break. Each module solves a problem the next one depends on.

Section 2What each module opens with

Every module opens on a problem the learner cannot yet solve, before any model or rule is introduced.

Module 1 opens with a plausible regulatory assessment containing an invented citation and a material omission. Module 2 opens with a mixed source pack and asks which sources the learner would permit AI to use, before the source fitness model exists. Module 3 opens with a change assessment and one question: is this one AI task? Module 4 asks whether every AI prompt and response should be retained.

In each case the learner meets the failure before it has a name.

Section 3Designing for a moving target

This was the hardest part of the build, and it is the part I would want another designer to read.

We are writing the methodology and the content before the regulation has been published and turned into law. Three things are moving at once and none of them are moving together: the regulation itself, what counts as good practice, and what the models can actually do. All three are expected to move hard and fast over the short and medium term.

A classroom absorbs that. A trainer reads the room, drops a dated example, substitutes a better one, says “that changed last month” and carries on. Elearning cannot. Everything committed to SCORM is committed, and the cost of being wrong is a rebuild rather than a sentence. So every decision that would stay flexible in a room had to be made in advance and defended against a future we cannot see.

Three rules came out of the back and forth with the subject matter experts.

Name capabilities, not products.

A model that retrieves across a document set survives. A named tool dates on its next release, and screenshots date faster than anything else in a course.

Teach the reasoning, use the regulation as the example.

A module that teaches what a particular article requires dates the day the article changes. A module that teaches how to decide what deserves reliance, using that article as its worked case, loses an example rather than a module.

Keep it open.

The course had to work for any organisation, so nothing organisation specific went into the durable layer. Thresholds, escalation routes and local procedure are the kind of thing that makes a course feel concrete and they are also the thing that makes it useless to the next client and out of date at the first one.

Underneath all three is the same split I use elsewhere: the volatile layer holds tools, citations and specifics and is built to be swapped. The durable layer holds judgement, source fitness and verification, and has to be true regardless of which model is sitting in front of the learner.

The question we could not settle.

Most of the SME time went on one thing: which types of regulatory work cannot be assisted by AI at all. The clearest cases were easy to agree. A reportability determination and a significance decision carry named professional accountability and cannot be delegated to a system, whatever it produces. Beyond those the boundary moved every time we looked at it, partly because the regulation is not settled and partly because what the models can do keeps changing underneath the question. The course therefore teaches learners how to locate that boundary in their own work rather than handing them a list that would be wrong within a year.

Section 4Building difficulty that holds

The cases contain traps. That is an instructional technique, not an assessment trick, and every failure is discoverable from the material provided.

A learner can rely on a superseded source, miss an omission, use guidance as though it were law, accept a plausible unsupported conclusion, confuse permission with evidence fitness, over route benign work, under route regulated work, treat passive review as a human gate, save enormous quantities of irrelevant provenance, or fail to retain the evidence that actually mattered.

A trap that only catches the careless teaches nothing.

The design rule is that a competent professional has to be able to fall for each one. A source is not obviously fake. A previous decision looks persuasive but is no longer authoritative. A risk file contains analysis that predates the change under consideration. A supplier asserts equivalence confidently, and the assertion never becomes independent evidence.

The course also contains over correction cases. A learner who concludes that the safe answer is not to use AI has also misunderstood the programme. The goal is capable use, not risk avoidance.

Section 5Feedback on the reasoning, not the score

Two learners can score identically and need completely different remediation. Somebody who routes every regulatory activity into a controlled workflow has a different problem from somebody who lets AI make a significance judgement.

The course therefore records the reasoning pattern rather than the mark: source status confusion, over claiming, judgement delegation, passive review, under routing, over routing, missing ownership, and using the wrong kind of verification. Practice is then targeted at that particular pattern.

Two techniques do most of the work.

Two pass analysis.

On the first pass the learner names the fault. On the second they decide what to do about it. Identifying that a claim rests on an inappropriate source is a different cognitive act from deciding whether to correct it, restrict it, request more evidence, reject it or stop. Separating them produces much better evidence about what the learner actually misunderstood.

Expert modelling.

After attempting a decision, learners see how an experienced practitioner approached the same problem. The comparison is not their answer against an answer key. It is their reasoning against somebody else's reasoning. After decomposing a change assessment, for instance, they are shown why an expert separates collecting sources from confirming their fitness, the significance decision from drafting its rationale, and candidate impacts from final judgement. The teaching is in why those boundaries protect the work.

Assessment separates confidence from competence.

Exploratory activities are not scored. Self assessment asks whether learners feel not yet ready, developing, confident, or able to coach somebody else, and that answer is never treated as evidence of capability. Where somebody reports high confidence and performs poorly on the unseen case, the gap is itself the finding.

Some errors are treated as critical and cannot be compensated for by performing well elsewhere. Allowing AI to make a reportability decision is not offset by answering several easier questions correctly. That reflects professional reality more accurately than a percentage.

Section 6What learners take away

Each module ends by taking the technique into the learner's own work. After Module 2 they apply the source fitness checks and the framing method to a real task. After Module 3 they decompose and route an actual deliverable they own. After Module 4 they design the verification and record requirements for one of those routed elements. The artefact is the work product, not a reflective essay about the work product. Three weeks later there is a short retrieval exercise and a request for one example of the practice used in real work.

Three job aids are meant to outlive the course.

• Source and Frame Card. Is the evidence fit, what may AI contribute, and what judgement remains mine?
• Decomposition and Front Door Card. What are the material elements, where should assurance live, and who owns the decision?
• Verification and Record Card. Which claims are load bearing, how will they be checked, and what evidence must survive?

Some design decisions look operational and are pedagogical. Modules run to about sixty minutes but divide into resumable sections rather than uninterrupted hours. Summative assessment is taken separately rather than compressing the teaching to fit. Interactions avoid drag and drop. Narration adds reasoning rather than reading text already on screen. Keyboard access, captions, transcripts, visible focus states and the avoidance of colour only meaning are specified at design stage, not left as production fixes. A learner who is overloaded or excluded by an interaction is not taking the course that was designed on paper.

If a learner needs the course in front of them every time they do the work, I transferred a procedure. If they can recognise the problem, make the judgement and reach for the tool when it helps, I have started to build capability.

Section 7What I do not know yet

The course has not run, so everything below is untested.

Whether the traps are calibrated correctly. They are designed so a competent professional could plausibly fall for them, and I will not know whether that is true until competent professionals sit the modules. Traps that catch nobody teach nothing, and traps that catch everybody teach the wrong lesson.

Whether the critical error rule survives contact with learners. Refusing to let a strong overall performance offset a delegated reportability decision is defensible in principle. I do not know yet how it feels to somebody who scores well and fails.

Whether the three week retrieval gets completed. It is the mechanism I trust most for telling whether a practice is forming and it is also the easiest thing in the design for a busy professional to ignore.

The common assessment rubric is still a joint workstream with the course team rather than a settled artefact, and specific regulatory material is flagged for re verification or legal review rather than presented as settled. That is deliberate. Provisional content presented as settled is precisely the failure the course teaches learners to detect.

And the largest one. The regulation this course prepares people for is not law yet. When it lands, some of the volatile layer will be wrong. The design bet is that the durable layer will not be, and that a course teaching people how to decide what deserves reliance will still be teaching the right thing after the specifics move underneath it. That is a bet, and it is the one worth watching.