Where AI should not be allowed to decide
Automation is safe when it is reversible and observable. It is dangerous when a wrong output silently becomes a fact that other systems trust.
The four categories
Money movement and credit decisions. Legal and regulatory filings. Anything that affects a person’s access to a service. Safety-critical instructions.
In all four, a model can draft and recommend, but a named human or a deterministic rule must decide.
Assist, do not approve
Design the system so the model proposes and a person approves, with the proposal and the approval both logged. That preserves speed while keeping accountability attached to a role.
Silent failure modes
The dangerous failure is not a dramatic wrong answer; it is a plausible one that passes review because it looks normal. Sampling and periodic audit catch what review misses.
Common questions
- Does human review remove all risk?
- No, but it bounds it and keeps responsibility with a role. Without review, responsibility diffuses until an incident assigns it for you.
- How much should we log?
- Input, output, model version, prompt version and the reviewer decision. Without those five, incidents cannot be reconstructed.
Next step
If this question is live in your business right now, describe the situation and you will get a written view on the options, including whether you should build at all.
Discuss your projectRelated notes
All insightsPractical AI automation, and where it should stop
Where language models genuinely reduce work in a business workflow, where deterministic code is cheaper and safer, and how to tell the difference.
ReadAI and automationAutomating invoice and document workflows
A practical architecture for extracting, validating and posting document data, including the exception queue that makes it reliable.
ReadAI and automationAI customer support that does not invent answers
Retrieval, refusal behaviour and escalation rules that keep an AI support assistant accurate and honest.
Read