AI Architecture

Choosing the Best AI Architecture for Your Product

A practical guide to choosing between deterministic software, machine learning, model calls, RAG, workflows, and agents, with production trade-offs and evaluation criteria.

2026 · 10 min read · Curated and presented by Arthur Sedek

When a team says, "We need an AI agent," I like to ask two questions first: what does the system need to decide, and what is it allowed to do? The answers tend to matter more than the model name.

There is no single best AI architecture. There is a best fit for a particular job, dataset, risk level, and operating environment. A useful design may be a conventional prediction model, one language model call, a search-backed assistant, a fixed workflow, or an agent that chooses its next step. Sometimes it is ordinary software with no AI in the critical path.

The practical goal is to start with the simplest design that meets the product requirements, then earn each additional layer with evidence.

Define the job before choosing the components

Write down what the user is trying to accomplish. Be specific about the information the system can use, the decision or action it must support, and what happens when it gets the answer wrong.

I use questions like these to make the problem concrete:

Is the input text, images, audio, structured data, or a mix?
Does the answer depend on current or private information?
Is there a stable rule, a repeatable pattern, or genuine uncertainty?
Does the user need an answer immediately, or can the work run in the background?
Can the system take an action, and would that action be reversible?
What would a useful result look like to the person doing the work?
What is the acceptable latency, error rate, and cost per completed task?
When must the system ask a person, or admit that it does not know?

These are architecture questions, not details to fill in after a model has been selected. They help avoid building a polished demo that answers the wrong question or cannot be trusted in the real workflow.

Match the problem to the smallest useful pattern

This table is a starting point, not a procurement decision. The same product may use several patterns, each for a different responsibility.

The problem looks like this	A reasonable starting point	Why
The rule is explicit and changes under your control	Deterministic application code	Easier to test, secure, and explain
Historical examples contain a repeatable signal	Predictive machine learning	Designed for classification, ranking, forecasting, and similar tasks
The task is a focused language transformation	One model call with a clear output contract	Often enough for drafting, extraction, summarisation, or classification
The answer depends on changing documents	Retrieval-augmented generation (RAG)	Supplies relevant evidence at request time
The steps are known in advance	A workflow with model calls where needed	Keeps ordering, validation, and retries under application control
The next step depends on what happens along the way	A bounded agent	Lets the model choose tools and adapt to intermediate results
The capability is common, such as speech recognition or document extraction	A managed AI service	Can avoid operating a specialised model yourself

Keep stable rules in code

Calculations, access checks, account balances, eligibility rules, and database writes should normally be enforced by deterministic services. A model can interpret a request such as "show me the overdue invoices," but it should not be the authority that decides which records the user is allowed to see or calculates the total.

The boundary is simple: let the model help understand language, and let application code enforce facts and rules that must always hold.

Use predictive ML for predictions

If the task is to estimate a future outcome from historical examples, a conventional machine learning model may be a better fit than a generative model. Examples include predicting equipment failure, classifying an image, ranking search results, or forecasting demand.

Before training anything, check whether the labels are reliable, the examples represent the cases the model will see, and the team can monitor drift. Accuracy alone can hide expensive mistakes. A model that misses a rare but dangerous failure may score well on an imbalanced test set and still be the wrong choice.

For common capabilities such as speech, translation, or document extraction, compare a managed service with a custom model. A custom model makes sense when you have a clear performance, privacy, deployment, or cost reason to own more of the lifecycle.

Decide what the language model needs to know

A direct model call is a good baseline for work that can be completed from the request and a well-defined set of instructions. It is often the right first version for drafting, rewriting, extracting fields, or explaining information the user has already supplied.

If the answer depends on private or frequently changing knowledge, the model needs access to evidence. That is where RAG can help. The application retrieves relevant passages, includes them with the question, and asks the model to answer from that material.

Diagram of a RAG application flow and the document ingestion pipeline

The useful part of RAG is not the vector database by itself. It is the full path from source documents to relevant evidence and a response the user can verify. Diagram: Microsoft Azure Architecture Center.

RAG is a system, not a checkbox. It includes preparing source data, choosing how to split and enrich it, generating embeddings when appropriate, retrieving and ranking results, assembling context, and evaluating the final answer. Keyword search can be important for exact names and identifiers. Vector search can help with paraphrases. Hybrid retrieval combines the two, and filters can enforce attributes such as document type or tenant.

The security boundary belongs in the retrieval and application layers. Do not retrieve documents a user is not authorised to see and hope a prompt will hide them. Preserve permissions when indexing and querying, and test that the right user gets the right evidence.

RAG is not automatically better. It adds a data pipeline, index maintenance, latency, operational work, and another place where results can go wrong. If the task needs an exact live value, a validated database query or API is usually a better source than a paragraph in a document. If the model only needs facts already present in the user's request, retrieval may be unnecessary.

Fine-tuning solves a different problem

Fine-tuning can help when you have many examples of a stable task and need the model to follow a particular style, format, or behaviour more consistently. It is usually not the right way to keep facts current. When facts change, retrieval or a live tool is easier to update and inspect.

Keep the distinction clear: retrieval supplies information for this request; fine-tuning changes how a model tends to respond. They can be combined, but neither removes the need to evaluate the actual product workflow.

Choose a workflow before an agent

People use the word "agent" to describe quite different systems. A useful architectural distinction is between a workflow, where application code controls the sequence, and an agent, where the model decides which steps and tools to use next.

A fixed workflow is a strong choice when the steps are known. For example, a support assistant might classify a request, retrieve an account record, draft a response, and wait for an employee to approve it. The order is predictable, each step can be tested, and the application can stop the process at a clear boundary.

An agent is more useful when the work is open-ended and the next step depends on what the system discovers. Investigating a production incident across several logs and services is one example. The system may need to choose a query, inspect the result, and decide what evidence to gather next.

That flexibility has a price. Tool calls can fail, the model can choose an unhelpful path, and a small mistake can compound over many steps. Keep tools narrow and clearly described. Set limits on time, calls, and spend. Make permissions explicit. Require confirmation before consequential actions, and make actions safe to retry where possible. Give the agent a way to stop, ask for help, or report that it could not complete the task.

Anthropic's engineering guidance and OpenAI's developer documentation both describe different levels of control, from direct model calls to managed or application-run agent runtimes. Those options are not interchangeable. Choose based on who should own orchestration, state, tools, approvals, and the execution environment, not on which one has the most features.

A worked example: answering a maintenance question

Imagine a field technician asks, "What should I do if pump 14 stays above its normal pressure after startup?"

A plain model call might produce a plausible answer, but it cannot know which pump is installed, which procedure is current, or whether the technician is authorised to see a particular record. A more dependable design separates those responsibilities:

The application authenticates the technician and checks access to the asset.
A structured service retrieves the pump's current readings and equipment details.
Search retrieves the relevant, authorised maintenance procedure and returns source passages.
The model explains the evidence in plain language and links to the source.
If the readings or sources do not support a safe answer, the system says so and routes the question to a person.

That is a model call plus tools and retrieval, but it does not need an autonomous agent if the steps are fixed. If a later product requirement asks the system to investigate several changing signals, choose among diagnostic tools, and adapt its plan, an agent may be justified. Any action that changes equipment state should have a stronger control boundary than a read-only search.

Treat production constraints as part of the design

The model endpoint is only one part of a production architecture. Identity, networking, data storage, monitoring, deployment, and failure handling shape whether the product is dependable.

Baseline production AI architecture showing identity, networking, application services, model access, and monitoring

A production AI application needs the same engineering attention as the rest of the product. This Azure reference architecture shows some of the surrounding services and controls. Diagram: Microsoft Azure Architecture Center.

Before choosing a hosting or model setup, write down the requirements that could rule options out:

Data and identity: Which data can leave the environment? How are users authenticated? Are permissions enforced on every retrieval and tool call?
Location and availability: Where may inference happen? What latency do users need? What happens if a region, provider, or model is unavailable?
Traffic shape: Is demand steady, spiky, or mostly offline? Forecast peak requests and tokens, not only daily averages. Separate work a user is waiting for from work that can run asynchronously.
Failure behaviour: What should the product do after a timeout, rate limit, malformed result, missing source, or provider outage? Use bounded retries with backoff. Do not let a retry loop multiply load.
Operations: Can you trace a result to the model version, prompt, retrieved sources, and tool calls that produced it? Can you see latency, error rates, token use, and cost by feature?
Governance: If several applications or teams share model access, do you need a gateway for identity, quotas, routing, and central logging? A gateway adds another service to operate, so it should solve a real coordination problem.

These choices can change over time. Start with the simplest deployment that satisfies your current data, latency, and reliability requirements. Revisit capacity, routing, and regional redundancy when measured usage or a compliance requirement calls for it. Provider offerings and model availability change, so verify current limits and regional support before committing to a design.

Evaluate the system users will actually experience

Do not select an architecture from a demo prompt. Assemble a small test set from representative tasks, including difficult inputs, ambiguous requests, missing evidence, and cases where the system should refuse or ask a question. Compare your candidate with a simpler baseline, including the non-AI way of doing the job.

Measure the whole task, not just whether the answer sounds good. Depending on the product, useful measures include:

task completion and time saved;
factual correctness and severity of errors;
retrieval relevance, source coverage, and citation support;
correct abstention when evidence is missing;
permission and tool-use correctness;
p50 and p95 latency, failure rate, and recovery time;
cost per successful task, including retrieval and orchestration.

For RAG, evaluate retrieval separately from generation. Did search find the evidence? Was the selected context current and relevant? Did the answer faithfully represent it? The Azure RAG design and evaluation guide recommends a structured process across preparation, chunking, enrichment, embeddings, retrieval, and end-to-end evaluation. That makes it easier to find whether a disappointing answer came from the data, search, or generation.

Keep the test set and run it when you change prompts, models, indexes, chunking, or tool definitions. Evaluation is not a launch-day ceremony. It is how you tell whether a new architecture change made the product better or merely different.

A decision sequence you can use in a design review

Work through these questions in order:

Can a stable rule or existing service solve the task? If yes, keep that part deterministic.
Is the task a prediction from examples? Compare a conventional ML model with the current manual baseline.
Does the user need language generation or interpretation? Start with one model call and a clear output contract.
Does the answer need private or changing knowledge? Add retrieval, and evaluate the evidence path.
Does the system need an exact value or to perform an action? Use a typed, permission-checked tool backed by the source of truth.
Are the steps predictable? Orchestrate them in code. Give the model control only when the path genuinely needs to adapt.
Do the quality, safety, latency, and cost results justify the additional components? If not, simplify.

The best architecture is the one you can defend with evidence

Good AI architecture is not a contest to fit the most techniques into one diagram. It is a set of deliberate boundaries: what the model interprets, what the application verifies, which evidence is available, what actions are allowed, and how the team will know when the system is failing.

Start with a useful baseline. Put rules in code, connect the model only to the information and tools it needs, and measure the outcome with real work. Add RAG when current evidence is necessary. Add an agent when fixed orchestration is no longer enough. Keep the system understandable enough that someone can trace a result, challenge it, and improve it.