AI Governance for Enterprise Software Deployments

AI governance is usually introduced as a policy exercise: principles, a committee, a register. Those have their place, but a policy that is not implemented in the codebase and the release process governs nothing. This is what governance looks like as engineering practice.

Version everything that changes behavior

In an AI-enabled system, behavior is determined by more than code. It is determined by the model version, the instructions or prompts, the retrieval corpus, the configuration thresholds and the tools the system may call.

All of these belong under version control with the same discipline as application code — reviewed, tagged, and deployed through the same pipeline. The common anti-pattern is prompts edited in a console by whoever has access, which makes it impossible to answer the most basic incident question: what was the system running when this happened?

Log what a future investigator will need

For any consequential action, record the input, the retrieved context, the tools called, the output, the version identifiers of everything above, and the identity of any human who approved it.

The test is reconstruction: given a case number and a complaint six months later, can someone reproduce what the system saw and why it acted as it did? If not, the system cannot be defended, corrected with confidence, or improved on evidence.

Retention and access to those logs need their own decision, since they frequently contain personal data.

Define the permission surface explicitly

Write down what the system may do, as a list. Separate reversible actions from irreversible ones. Treat anything that moves money, contacts a customer, changes a record of account or deletes data as requiring either a deterministic path or explicit approval.

Permission surfaces expand quietly during development — a tool added for convenience, an integration granted broader scope than it needs. Reviewing the list as a distinct artefact at release is how that gets caught.

Build the stop mechanism before you need it

There must be a way to pause the system that does not require a code deployment, a named owner permitted to use it, and a documented fallback for what happens to in-flight work.

Organizations discover the absence of this during an incident, which is the worst possible time to design it. A feature flag and a written runbook cost an afternoon.

Evaluate before release, monitor after

A held-out evaluation set with agreed correct outcomes is the only way to tell whether a change improved the system. Run it on every change to any of the versioned components above — including a model upgrade the provider describes as an improvement, because improvement in general does not guarantee improvement on your distribution.

After release, monitor for drift: the input distribution shifting, the override rate moving, the escalation rate changing. These are early signals that the conditions the system was validated under no longer hold.

Map to the frameworks rather than starting from them

The published frameworks are useful as checklists once the engineering is in place. The NIST AI Risk Management Framework organises the work into govern, map, measure and manage functions (NIST, 2023). ISO/IEC 42001 specifies requirements for an AI management system, which suits organizations that need certifiable structure. Singapore's Model AI Governance Framework addresses internal governance, human involvement, operations management and stakeholder communication (IMDA and PDPC, 2020). The EU AI Act imposes obligations that vary by risk classification and may apply to systems placed on the EU market regardless of where the provider is established (Regulation (EU) 2024/1689).

Which of these apply to a given organization depends on sector, market and data, and that determination belongs with qualified legal advice rather than with a technology supplier. What is true generally is that the engineering practices above satisfy most of what the frameworks ask for, and that the reverse route — writing the policy first and retrofitting the engineering — is considerably more expensive.

Accountability, concretely

Name the person answerable for the system's outcomes. Give them the authority to stop it, the information to judge it, and a review cadence. Governance without a named owner is a document; governance with one is a practice. The companion piece on human oversight covers how to keep their review meaningful.

References

1. National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1.
2. ISO/IEC 42001:2023. Information technology — Artificial intelligence — Management system. International Organization for Standardization.
3. Regulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence.
4. Infocomm Media Development Authority & Personal Data Protection Commission, Singapore (2020). Model Artificial Intelligence Governance Framework, second edition.