Why governance belongs in the system, not the slide deck
Most AI governance is performance, not protection. It looks like oversight, reads well in a board paper, and does not survive a regulator's question, because the controls it describes exist only on paper. A policy that says "human oversight" means nothing if the system ships with an approve button nobody audits. A policy that says "no decisions about individuals" means nothing if the model quietly ranks them anyway.
The failure usually happens at the handoff. One firm writes the governance strategy, a different team writes the code, and the controls the board needs never make it into the system. Our position, proven across the systems we have shipped, is that the governance and the code have to be written by the same team in the same pass. The test of every control below is the same: it must hold even when the model misbehaves. If a control depends on the model behaving, it is not a control.
Six controls we build into AI systems
1. Read-only by construction. The database itself rejects any write the AI attempts, enforced as a read-only transaction at the database layer rather than as an instruction in a prompt. The analytics AI cannot change what it reads, whatever it is asked, so the control does not depend on the model behaving.
2. An append-only decision ledger. Every AI suggestion is recorded with the model used, the input, the output, and the named human's decision: accept, modify or reject, with a timestamp. The table has no update and no delete, so the record cannot be quietly tidied after the fact. This is the audit trail a regulator means when they ask for one.
3. Confidence floors and reason codes. A configured threshold below which deterministic code overrules the model and routes the case to a named person, with an enumerated reason code recording why. The decision that takes effect is the human's, and code, not the model, decides when that handover happens.
4. Citation and substring checks. Every quotation the AI attributes to a source must be a literal substring of that source, or the extraction fails and is never published. A denied-vocabulary check blocks the speculative language an extraction must not contain. A claim the system makes is checkable, by construction.
5. Human approval gates and advisory-only design. The AI assists; a named person decides; the system cannot do otherwise. In our public-sector work this is a hard constraint: no scoring, no automated outcomes, no influence on decisions about a person or a procurement. Advisory-only design also keeps a tool clear of the automated decision-making rules entirely, which is the cleanest compliance position there is.
6. Schema-constrained outputs. The model's output must conform to a defined schema before anything downstream consumes it. Free-form text cannot leak into a field that drives an action, which bounds what the AI can cause and keeps the record legible.
What those controls look like in production
These are not framework diagrams. Each comes from a real deployment for a UK organisation, anonymised.
In an insolvency intelligence engine, every AI-extracted quote must be a literal substring of the source document. Extractions that fail the check are not published. The result is a system whose claims a sceptical practitioner can verify in one click.
In an AI operations system for a property manager, the analytics layer is read-only by construction and a confidence floor keeps the AI out of the ERP: below the threshold, deterministic code routes the invoice to a person with a reason code attached.
In a bespoke operating system for a surveying practice, an append-only ledger records every accept, modify or reject decision by a named surveyor. The AI disclosure at the foot of each report is generated from that ledger rather than typed from memory, which is what the RICS standard on responsible AI use expects done properly.
In a public-sector evidence workspace, the AI is advisory-only by hard constraint and every output is traceable to the exact source passage through chunk-level citations, which is what lets a combined authority publish an honest transparency record for it.
The pattern across all four is the same. The governance is a property of the code, and the system produces its own evidence as it runs. That is what we mean by responsible AI in practice: not a value statement, but an architecture.