Choosing the Right AI Architecture for a Real Product

A successful AI product rarely begins with a model name. It begins with a clear understanding of the decision, workflow, and risk involved.

Teams now face a wide range of architectural choices. A solution might use deterministic rules, conventional machine learning, semantic search, retrieval augmented generation, an agent, or a combination of these. Choosing the most complex option by default increases cost and uncertainty without guaranteeing better outcomes.

Arthur Sedek uses a capability-first approach: select the smallest architecture that can reliably solve the real problem, then add complexity only when evidence supports it.

Start with the job to be done

Define the user, the decision, and the desired outcome before discussing technology. A useful problem statement explains:

• what information is available;
• what action the user needs to take;
• how quickly a result is required;
• what an incorrect result would cost;
• whether the task can be reviewed by a person;
• how success will be measured.

This definition prevents a common failure mode where a technically impressive system solves the wrong problem.

Prefer deterministic software when the rules are stable

Traditional software remains the best choice when the required behaviour can be expressed with clear and stable rules. Calculations, validations, permission checks, workflow transitions, and database operations should generally remain deterministic.

A language model can help users discover or configure these operations, but it should not replace reliable logic. Keeping deterministic responsibilities outside the model improves testability, security, and cost control.

Use predictive machine learning for repeatable patterns

Conventional machine learning is appropriate when historical data contains patterns that support a defined prediction. Classification, regression, anomaly detection, ranking, and computer vision often fit this category.

The decision depends on more than model accuracy. Teams should examine label quality, representativeness, drift, latency, explainability, and the cost of errors. A simple model with stable behaviour can be more valuable than a larger model that is difficult to operate.

Use retrieval when knowledge changes

Retrieval augmented generation is useful when answers depend on a body of documents that changes over time. It allows a language model to work with current policies, product documentation, research, or organisational knowledge without encoding every fact in model parameters.

Retrieval is not a substitute for structured queries. If the answer requires an exact calculation or a current database value, the system should call a validated tool. Retrieval should supply relevant knowledge and evidence.

Use agents for multi-step work

An agent becomes useful when a task requires planning, selecting among tools, observing intermediate results, and adapting the next action. Examples include investigating an incident, preparing a structured report from several sources, or coordinating a multi-stage analytical workflow.

Agents introduce additional failure modes. A production design needs bounded tools, explicit permissions, execution limits, traceability, and approval points for consequential actions. If a fixed workflow can solve the task, it is usually easier to test and operate than an autonomous agent.

Combine patterns deliberately

Many valuable products use a layered architecture:

1. deterministic services enforce business rules and permissions;

2. retrieval supplies relevant knowledge;

3. predictive models generate specialised signals;

4. a language model interprets intent and explains results;

5. an agent coordinates steps only where adaptation is necessary;

6. the interface keeps the user informed and in control.

Each component should have one clear responsibility. This makes failures easier to isolate and allows individual components to evolve without destabilising the whole product.

Evaluate the complete system

Architecture decisions should be tested against product outcomes. Measure task completion, error severity, user effort, latency, cost, and operational reliability. Model-level metrics remain important, but they are not enough.

Run evaluations on representative workflows, including ambiguous inputs and expected failure cases. Compare the AI solution with a simpler baseline. If the additional complexity does not produce a meaningful improvement, simplify the design.

A practical decision sequence

When considering an AI architecture, ask these questions in order:

1. Can explicit rules solve the problem reliably?

2. Is there a repeatable pattern that predictive machine learning can learn?

3. Does the task require knowledge from changing documents?

4. Does it need controlled access to external tools or live data?

5. Does it genuinely require adaptive multi-step planning?

This sequence keeps the architecture proportional to the problem.

Build the minimum dependable system

Good AI architecture is not measured by the number of advanced components. It is measured by how reliably the product helps someone achieve a valuable outcome.

The best starting point is the minimum dependable system. It creates a clear baseline, reaches users sooner, and provides evidence for every later investment in complexity.