Compliance Assessment Fails When It Tests Recall and Claims Behaviour

The dashboard is green.

It says complete in a font large enough to reassure a board committee and small enough to conceal what, exactly, has been completed.

There is a large assigned population, most of it yours.

There is a module called Code_of_Conduct_Annual_FINAL_SCORM.zip.

There is a short quiz.

There is an acknowledgement box.

Somewhere, an export is being prepared for audit.

Then somebody meets the condition the module was supposed to prepare them for.

The question is no longer whether they can recognise the correct answer among plausible distractors.

The question is whether they recognise the condition.

Choose the right action.

Use the reporting route.

Stop the work if required.

Escalate before the problem becomes an incident.

Those are not the same event.

A completion result is evidence of an assessment event. Correct behaviour is evidence of performance at the operational decision point.

Both matter.

The trouble starts when the first is allowed to impersonate the second, because it arrives as a clean field in your LMS.

The quiz answers a smaller question than you asked

Assessment has an irritating habit of requiring a claim before it will behave.

The starting sentence is this.

Using this result, you will decide whether the learner can do something, under stated conditions.

Name the performance.

Name the condition.

Name your population, your decision maker, and the consequence.

A fifteen-minute diagnostic can identify a missing prerequisite.

It should not quietly become permission for independent equipment operation.

That distinction is not academic paperwork.

A claim that somebody can recognise a reporting rule can use selected responses.

A claim that somebody can perform a procedure, make a safety decision, or explain a failure needs observed performance, an artefact, a scenario with a rationale, or some combination of those.

So the weighting is a decision, not a residue.

What share of the assessment sits on the routine procedure, on diagnosis, on the safety decision, on the documentation?

That distribution is what makes the claim inspectable.

Your completion quiz is not useless.

It can efficiently establish that a worker saw the specified content and answered the included questions.

It can reveal a misconception worth correcting.

It can be exactly right for awareness and acknowledgement.

What it cannot do is acquire a behavioural claim through enthusiasm, a pass mark, or a slide titled “Knowledge Transfer.”

The Standards for Educational and Psychological Testing put it more politely. Validity concerns the evidence and theory supporting interpretations of scores for proposed uses.

A 92 percent mean score shows that a cohort answered most of the included questions correctly.

By itself it does not show that the content represented the required work.

Or that the score is stable.

Or that a work-authorisation decision built on it is fair.

That is the uncomfortable part.

The score can be perfectly accurate about recall and still be the wrong evidence for the decision.

Recall is a useful signal. It is not the scene of the work.

In an office course, a learner can identify that a report should be escalated.

In the actual moment, that learner may need to distinguish a concern from a routine disagreement.

Preserve information.

Use a local reporting channel.

Stop a process while an accountable person decides.

The action has conditions.

The quiz has options.

The difference gets sharper around physical work.

Regulators say plainly that online training alone may not satisfy a standard where hands-on experience and questions are needed.

Guidance for hazardous-waste work describes documented proficiency through a written assessment and a skill demonstration. Where the written test accompanies the demonstration, it specifies at least 25 questions.

That is not a universal rule that every compliance course needs 25 questions.

It is a concrete example of a regulator distinguishing written recall from demonstrated skill.

Copy the distinction.

Not the number.

Build the blueprint before the items

Put the content or task areas down one side.

Put the actual performance demand across the top: recall, interpretation, procedure, explanation, judgment, production.

Then make your critical errors non-compensable where the decision requires it. Every designated critical criterion has to be met, with no trading permitted against the rest.

That rule is operational, not academic.

A person can accumulate a comfortable total while missing the one criterion the control existed for.

Then pilot the thing before you scale it.

In a thirty-minute pilot, three people stopping at the same instruction is evidence about the instruction.

It is not an outbreak of laziness.

For a twenty-item written assessment, a reviewer can map every item to its target, its demand, its key, its source and its decision.

And if an outcome says “analyse” while the task asks for a definition, the assessment measured recall.

Report it as recall.

Your system will survive the honesty.

The record has to say what happened

An assessment result is not a moral verdict.

If a worker fails a practical demonstration, the useful status is not yet authorised.

Targeted remediation follows.

The manager keeps the worker outside the task.

The safety owner asks whether the instruction, the equipment, the local procedure or the role assignment caused the failure.

Replace that history with a later completion flag and you have made the audit prettier and the control weaker.

So keep the original assessment, the version, the observer sign-off and your correction history.

The certificate can travel with the person.

Your system of record has to preserve what it actually means.

None of this argues against the quiz

Completion tracking, acknowledgement, knowledge checks and audit files all solve real problems.

You have to know which rule applies to which worker.

Whether the right version was assigned.

Whether the course was delivered.

Whether the record can be retrieved.

Whether an overdue requirement needs action.

Completion data also supports triage.

A low recall result can trigger feedback.

A missing acknowledgement can block assignment to a policy-controlled workflow. A version migration can identify people who need the changed content.

Payroll, access control, scheduling, HR and learning all need a common evidence event, even though nobody has yet invented a cheerful way to reconcile their identifiers.

The point is not to abolish the wrapper.

The point is to stop calling it the durable object.

Completion is evidence that a programme event occurred. Assessment is evidence for a bounded claim. Behaviour is what the workplace control was trying to produce.

Keep those three apart and you can defend the programme without making claims your records cannot carry.

AI makes the available evidence look like the important evidence

For years this was managed by people who knew that a green completion flag meant “ask another question.”

Then systems began ingesting learning histories, due dates, scores, acknowledgements, incident logs and job records, to rank risk, recommend remedial training and report workforce compliance.

The model sees a completion flag, because it is structured.

It sees a quiz score, because it is numeric.

It sees an overdue date, because it fits in a column.

It may not see a supervisor’s field observation.

Or the local work condition.

Or the unavailable equipment.

Or the moment somebody correctly stopped work and escalated.

Those records live in notes, in incident systems, and in the durable memory of a manager who has never met a data contract they liked.

So the system gives the heaviest weight to the proxy that is easiest to ingest.

That is not an AI failure.

That is a data contract failure wearing a risk heatmap.

An AI system can use completion data well.

Identify missing assignments.

Retrieve evidence.

Detect stale versions.

Route a case to review.

It just has to keep the course version, the assessment format, the claimed decision, the work scope, the practical evidence, the correction log, the incident context and the known limitation attached to every recommendation.

A completion score may support a recall claim.

It should not be promoted into a behavioural finding because your dashboard prefers a single number.

Put the operational decision back into the design

The repair is less glamorous than a machine-learning compliance score, and more useful on Tuesday morning.

Start each of your requirements with its authority, its scope and its decision claim.

Decide whether the programme needs awareness, recall, practical procedure, judgment, escalation or authorisation evidence.

Then map the evidence source to the claim, rather than asking one quiz to carry every conclusion.

For work that can create immediate consequences, use scenarios that force the critical decision, and practical observation where execution matters.

Lock the rubric.

Identify the non-compensable errors.

Calibrate scorers with anchor responses.

Log the administration exceptions.

A ninety-minute examination with a seven-minute outage needs an incident record, because the conditions are part of what the score means.

Connect passing results to a real work gate of yours.

Review incidents and near misses by task, location, cohort and prior instruction.

When equipment, process, work organisation, material or job assignment changes, reopen the requirement, instead of replaying the annual module as a small corporate solstice.

Then report the result in language it can support.

“Completed the recorded course version on this date” is useful.

“Selected 19 of 24 answers correctly under standard conditions” is useful.

“Passed a documented practical check for this task, at this site, under these conditions” is useful.

“Will behave correctly whenever the relevant condition occurs” is a much larger claim.

It needs the work to answer it.