The Architecture of Modern AI-Enabled Business Applications

The layer of an AI application that changes fastest is the model and everything immediately around it. In most implementations, that is also the layer coupled most tightly to the business logic. This is the central architectural mistake of the current period, and it is avoidable.

Four layers worth separating

Interface. What the user sees and does. Should know nothing about which model is in use.

Process logic. The rules of the business: what constitutes a valid case, which approvals apply, what happens on rejection. Deterministic, testable, and the part most specific to the organization.

Inference. Model calls, prompts, retrieval, output parsing and validation. The volatile layer.

Integration. Connections to systems of record, each with its own authentication, data model, rate limits and downtime.

The valuable boundary is between process logic and inference. Cross it with a defined interface — a typed request and a validated response — and the model becomes replaceable. Let business rules leak into prompts, or model-specific output formats leak into process logic, and replacing the model means rewriting the application.

Validate output before acting on it

Model output is untrusted input. It should be parsed into a defined structure, checked against constraints the business logic can state, and rejected or escalated when it fails.

Concretely: if the system extracts an invoice amount, validate the type, the range and the currency against the record before any downstream action. If it selects a category, verify it is a member of the permitted set rather than a plausible-looking near-match. These checks cost little and catch the specific failure mode that matters — confidently wrong output that would otherwise flow straight through.

Design for the failure modes of a metered dependency

An inference provider is a dependency with characteristics most application dependencies do not have: variable latency, per-call cost, rate limits, occasional unavailability, and behavior that can change when the provider updates a model.

The implications are practical. Timeouts and retries need budgets, because retries multiply cost as well as latency. A degraded path should exist for when inference is unavailable — usually queueing work or falling back to a deterministic route, rather than failing the request. And cost per operation belongs in monitoring next to latency, because a change that triples token use is a production incident even when nothing breaks.

Keep state in your own systems

Conversation history, case state and decision records belong in your database, not in a vendor's session abstraction. This is partly about exit and partly about basic operational capability: querying, reporting and auditing all require the data to be somewhere you control.

Make the system explainable by construction

Explainability in practice is usually not model interpretability. It is a record: what the system was given, what it retrieved, what it produced, under which versions, and who approved the action.

That record is an architectural feature. It has to be designed into the request path, because the information is gone once the request completes. Systems that add logging after an incident capture everything except the incident.

Evaluate as a build step

An evaluation set of representative cases with agreed correct outcomes should run in the pipeline, the way tests do. It gates changes to prompts, retrieval and model versions — including provider upgrades, which are changes to your system whether or not you initiated them.

What this buys

An application built this way survives the field moving. When a better or cheaper model appears, adopting it is a change to one layer, verified against the evaluation set, with the business logic untouched. That is the difference between an AI application with a five-year life and one that is rewritten annually.

The related discussion of build versus buy covers where this boundary sits when part of the stack is purchased, and the software development page sets out the company's wider engineering approach.

References

1. Sculley, D., Holt, G., Golovin, D., et al. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems, 28.
2. National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1.
3. ISO/IEC 42001:2023. Information technology — Artificial intelligence — Management system. International Organization for Standardization.