Epistemic Guardrails and AI Poka-Yoke

Idea one

The illusion of algorithmic fluency

The dangerous errors arrive in complete sentences, with a confident tone, a plausible sequence of reasoning and sometimes a list of sources. They resemble the work of a capable colleague who has weighed the evidence and reached a measured conclusion.

A language model generates probable continuations. It does not hold the commitment to truth an accountable professional is expected to hold. It can produce statements that are accurate, incomplete, outdated, fabricated, improperly generalised, or valid only under conditions absent from this case — and fluency makes those distinctions hard to see, because human beings are naturally inclined to infer competence from coherence.

A broken answer invites scrutiny. A hesitant one prompts questions. An answer that states its uncertainty creates an opening for escalation. A polished response lowers the psychological defences of the reader. It appears to have done the intellectual work already, and it offers relief from ambiguity — especially to someone managing a queue under time pressure.

Now the distinction that matters most. Take an agent that retrieves an expired regional presentation describing a temporary service exception, blends it with current policy, and produces a polished answer saying the customer is eligible for an adjustment. The account details may be correct. It may quote a real policy provision. The conclusion is still wrong — and the agent may not have invented anything at all. It has assembled real fragments into a conclusion that does not follow from the authoritative context.

Organisations use "hallucination" as though obvious fabrication were the main danger. An agent can be wrong because it chose the wrong source, omitted a qualifying condition, treated an exception as a rule, confused two similar entities, relied on stale data, or made an inference exceeding its evidence. Each of those can be delivered with perfect linguistic confidence.

Idea two

A second fluent answer is not a second opinion

A single chat response gets read by one person. A multi-agent workflow turns one weakly grounded output into the input for several downstream actions. A research agent summarises. A planning agent builds a recommendation on that summary. An execution agent invokes tools on the recommendation. A communication agent explains the result to a customer. At each stage, fluency makes the prior output look more authoritative than it was.

Downstream agents may never see the original source, or may be instructed to treat the upstream output as a trusted summary to save cost and latency. The original uncertainty disappears through compression. What began as one possible interpretation of an outdated document becomes the applicable rule, then the approved action, then the completed customer resolution.

By the time a person meets the final artifact, the chain looks coherent. Internal consistency, however, is not the same as external correctness.

That is aligned failure. A coding agent generates a flawed implementation from an incorrect requirement; a testing agent produces tests confirming the implementation behaves as specified; a documentation agent explains the design clearly; a review agent finds no inconsistency between requirement, code, tests and documentation. Every component agrees, and all of them are aligned around the same initial defect.

Traditional quality practice struggles here because it assumes independent-looking checks are genuinely independent. In the agentic enterprise, independence must be designed. If the generator and the validator share the retrieved context, the prompt library, the model family or the unstated assumptions, the validator may simply restate the generator's mistake in a more critical tone. A second fluent answer is not necessarily a second opinion.

Idea three

Move the guardrail earlier than the review

In manufacturing, Poka-Yoke makes errors difficult to commit, easy to detect, and impossible to pass unnoticed into the next stage. A fixture accepts a component only in the correct orientation; a sensor stops the machine when a part is missing. These controls do not assume workers are careless. They recognise that capable people work under time pressure, distraction and fatigue — and that a well-designed process does not ask attention alone to carry the whole burden of quality.

So relying on post-generation review as the primary quality mechanism is a design error. Asking people to inspect every polished output is neither reliable nor humane: exhaustive verification creates extraneous load, turning knowledge workers into monitors of machine-generated plausibility, reconstructing evidence under conditions that reward speed over challenge. Fatigue then makes the fluent answer more persuasive. The reviewer sees an apparently complete recommendation, senses the queue growing, and accepts it because deeper verification feels too expensive.

The organisation then mistakes exhausted acceptance for reliable automation.

The epistemic equivalent of mistake-proofing is to make unsupported claims, weak lineage, source confusion and unjustified action difficult for the workflow to produce or propagate at all. It asks whether an agent has access only to approved sources for this decision; whether each consequential claim links to an authoritative source, effective date, scope and qualifying condition; whether the workflow can tell evidence from commentary, current policy from historical context, verified fact from model inference. And whether the system knows when it does not know.

The objective is not to remove uncertainty — knowledge work is full of genuine ambiguity. The objective is to prevent uncertainty from being disguised as certainty. A transparent pause at the point of uncertainty costs less than a confident error that creates downstream rework, customer harm and compliance exposure. And the goal is not timid agents but calibrated autonomy: low-consequence work proceeds with lighter controls, high-consequence work carries stronger evidence requirements and tighter authority boundaries.

Idea four

Retrieval is not permission

The first design principle is source-bound generation. For consequential tasks the agent should not answer from general model knowledge and loosely retrieved fragments blended without distinction. It should construct claims from approved evidence inside a defined information boundary — which means the system has to know which sources are permitted for the decision at hand, and which are merely supplementary, historical or prohibited.

The expired regional presentation is genuinely valuable as history. It explains why a temporary exception existed and may help a specialist understand an old customer communication. It must not be eligible to determine present operational policy.

So a source-bound system does not retrieve documents merely because they are semantically similar to the question. It retrieves through an authority-aware path, asking: is this source current, is it approved for this purpose, does it apply to this jurisdiction, customer, product or risk category, has it been superseded, does another source override it here? That is what the semantic structure of lesson five is for — representing not only that a policy exists but that it is authoritative for a particular decision type, effective during a defined period, owned by an accountable function, and constrained by named exceptions.

Which produces the distinction to carry out of this idea. An agent may be permitted to see a historical document without being permitted to use it as the basis of a recommendation. It may retrieve a non-authoritative source in order to notice that a conflict exists, and then be required to find the controlling authority or escalate. Access to information must not be confused with authorisation to rely upon it.

Idea five

Claim-level traceability, and separating infer from do

A list of citations at the end of an answer is not enough. Citations can create a performance of rigour while obscuring whether the cited material actually supports the claims. A response may cite an authoritative policy while making an inference the policy does not justify, quote a clause accurately while omitting the qualifying condition in the next paragraph, or cite several conflicting sources and invite the reader to assume the conclusion emerges cleanly from all of them.

So for high-consequence output the system should identify the material claims and connect each one to the source that supports it, preserving the passage, version, effective date and applicability. Where the agent has made an inference rather than restating a verified fact, that distinction should be labelled. Where a conclusion rests on an assumption, the assumption should be visible rather than buried inside fluent prose.

And when a required fact cannot be verified, the agent must not convert the gap into a confident conclusion. It should say that eligibility cannot be determined because the classification is unavailable, and that human review is required before any commitment. This is not a failure of automation. It is successful mistake-proofing.

The third principle follows: separate what the agent can infer, what it can recommend, and what it can do. Being capable of synthesising evidence and drafting a recommendation does not mean being permitted to execute the action. A model may propose an action only after required facts are verified, invoke a tool only if policy conditions are satisfied, communicate externally only if its authority level permits it and the evidence accompanies the action.

Which is the central lesson, and it is worth stating flatly. An agent's ability to produce a persuasive answer is not evidence that it has produced a trustworthy one. Where outputs travel at machine speed and trigger further machine action, fluency must be treated as a risk signal as well as a capability.