How to Design an AI System
Designing an AI system is a sequence: write the job and SLOs, pick chatbot vs workflow vs agent, decide where facts and permissions live, freeze an eval, then security, traces, and serving. The model is a component, not the architecture.
Explain like I'm new to AI
Every earlier category was a brick. This one is the house. Teams fail by starting at "which 70B" or "we need agents." 2026 practice is the opposite: narrow the job until a simpler shape works, and prove it with a gold set.
Who is the user, what is 'done,' and the four knobs: quality floor, p95 latency, $ per successful task, reliability. If you cannot write this, you cannot pick a model.
| Wrong order | This order | |
|---|---|---|
| Starts with | A model name and a framework | The user job + four SLOs |
| Agency | A loop because it's fashionable | Workflow until eval demands a decision |
| Facts | Fine-tune the wiki | RAG / APIs; fine-tune habits later |
| Safety | A stern system prompt | Tiny tools, ACL at retrieve, HITL |
| Done | A demo transcript | CI gold set + traces + $ / success |
Mental model
Building a kitchen: you don't buy a fighter-jet stove first. You write the menu (job), the service time (SLO), then the stations (retrieval, tools), the health inspection (eval/security), the ticket printer (traces), then the stove size (inference).
How it works
- Job & SLO — success verb ("resolve the ticket," "cite the policy"). Quality floor, p95, $, reliability (evaluation + inference).
- Shape — chatbot / workflow / agent (Agent vs Chatbot vs Workflow). Start left.
- Knowledge & tools — changing facts → RAG or live APIs; stubborn format → LoRA later; tools = permissions (security).
- Context & memory — pack the window on purpose; session vs long-term (context engineering, memory).
- Eval — 50–200 real tasks, never train on them, CI.
- Security — untrusted retrieve, least privilege, output encoding, secrets out of the window.
- Observability — same trace shape in eval and prod; prompt versions.
- Serve — prefill/decode, routing, cache, KV memory. Iterate. Add agency only where the gold set says the workflow failed.
MCP is how tools connect, not a reason to grant them.
Real-world example
"AI support." Bad: 70B + 40 MCP servers. Good: lookup + policy RAG + structured refund preview + HITL. Eval on held-out tickets. Traces on tool spans. Route FAQs to a small model.
Technical explanation
Data-flow diagram before class diagram: user → auth → retrieve/tools → model → encode → UI. Every arrow is a trust boundary.
Non-goals are design: if you won't call refund, don't put the tool in the catalog (injection blast radius).
Provider vs self-host is an inference/cost decision after the job is specified.
The next six lessons are worked examples of this loop. Use them as templates, not as vendors.
Common mistakes
Picking the model in week one and the gold set in month four. You will fine-tune a demo.
- Multi-agent because the slide said "team of agents."
- Skipping tenant ACL because the prototype used one PDF.
When to use it
- Any system that will see a second user. The loop is the checklist even for a spike — write the SLO on a sticky note.
When NOT to use it
- Don't run the full production loop for a throwaway that never leaves your laptop. Do run it the moment someone else depends on the answers.
Alternatives
- Buy a narrow product for a narrow job. You still owe eval and data handling — you just didn't build the engine.
Quick quiz
Related concepts
- What is RAG? — RAG retrieves relevant external information and gives it to a language model as context before it answers.
- What is an AI Agent? — An AI agent is an LLM that can decide and take actions in a loop, using tools to pursue a goal rather than just replying once.
- Agent vs Chatbot vs Workflow — A chatbot replies, a workflow runs fixed steps, and an agent decides its own next step — knowing the difference tells you which to build.
- Quality, Latency, Cost, and Reliability — Production AI is a tradeoff of quality, latency, cost, and reliability. Optimize dollars per successful task under an SLO — not always the smartest model.
- What is Prompt Injection? — Prompt injection mixes untrusted text with instructions in one window — the model cannot tell them apart. Constrain what it can do in code.
Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI