How to Build AI Tools: A Platform Engineer's Stack, From Prompt to Production

"How do I build an AI tool?" has two honest answers. The demo answer is twenty lines: take input, call a model API, show output. The production answer is the rest of this post, because the model call is the least interesting part of the system. What separates a tool people rely on from a weekend demo is everything wrapped around that call: structure, evaluation, isolation, and cost control.

I build and operate this kind of infrastructure for a living, so this is the stack I'd actually use in 2026, layer by layer.

Layer 1: Model Access — Buy the API First

Start with a hosted API (Anthropic, OpenAI, or a managed provider). Self-hosting open-weight models is a real option — I've written about vLLM vs Ollama for exactly that decision — but it's a second step you take for cost, latency, or data-residency reasons once the tool works. Self-hosting before product-market fit is buying GPUs to avoid a $40 API bill.

Two non-negotiables from day one, regardless of provider:

• An abstraction seam. Route every model call through one module you own. Not necessarily a framework — a 50-line wrapper is fine — but model choice, retries, timeouts, and token accounting live in one place. You will switch models.
• Token + cost logging per request. Cost in an AI tool is a runtime variable, not a line item. If you can't answer "what did this feature cost yesterday," you shipped a liability.

Layer 2: Structured Output — JSON or It Didn't Happen

The single biggest reliability upgrade for any AI tool: stop parsing prose. Every modern API supports constrained/structured output (JSON schema mode, tool definitions). Define the schema for what your tool produces, validate with the same schema library you use everywhere else (Zod, Pydantic), and treat validation failure as a retry trigger — once, with the error message fed back — then a hard failure.

This converts the model from "creative text generator" into "unreliable but useful function," which is the correct mental model for everything that follows.

Layer 3: Context — RAG Before Fine-Tuning, Always

Nine times out of ten, "the model doesn't know our stuff" is a retrieval problem, not a training problem. The boring pipeline works: chunk your docs, embed them, store vectors (pgvector if you already run Postgres — don't add a vector database before you need one), retrieve top-k, stuff the prompt. Fine-tuning earns its complexity only when you need style or format the base model can't produce, on stable data, at volume. Knowledge that changes weekly belongs in retrieval, never in weights.

Layer 4: Tool Calling — Where Tools Become Agents

Letting the model call functions — search the catalog, query the database, run code — is where real capability lives, and where the risk concentrates. The rules I hold the line on:

• Tools are typed, narrow, and idempotent where possible. "Run SQL" is a bad tool; "look up order by ID" is a good one.
• Execution is sandboxed. Anything that touches a shell, filesystem, or interpreter runs in a disposable, locked-down environment — gVisor, Firecracker, or at minimum a no-credentials container. I covered the patterns in sandboxing AI agents.
• Credentials are per-tool and short-lived, minted at call time. The model never holds a secret; the tool gateway does.
• Budgets are hard limits: max steps, max tokens, max wall-clock per task. A loop that can retry is a loop that can run forever.

If you're deciding how much autonomy the system should have at all, I wrote a whole piece on that boundary: AI agents vs agentic AI.

Layer 5: Evals — Your New Test Suite

You cannot unit-test a model, but you can eval the system. Maintain a versioned dataset of real inputs with expected properties, and score every prompt/model change against it before shipping. Three tiers, cheapest first: assertion checks on structured output (valid schema, required fields, no hallucinated IDs); programmatic scoring (exact match, retrieval hit rate); LLM-as-judge for the genuinely fuzzy cases — calibrated against a sample you've graded by hand.

The discipline matters more than the tooling: no prompt change merges without the eval suite running. Prompts are code. Treat regression in eval scores like a failing CI gate, because that's what it is.

Layer 6: Shipping It — The Part You Already Know

The deployment story is deliberately boring, and that's the point — an AI tool is a service like any other once the layers above are in place. Async-first design (model calls are slow; queue anything over a few seconds), streaming to the UI for perceived latency, aggressive caching of identical requests, rate limits per user and per budget. On Kubernetes, the one genuinely new wrinkle is GPU scheduling if you self-host — see running GPU workloads with the NVIDIA GPU Operator — and capacity planning that accounts for one user action fanning out into dozens of model calls.

The Checklist

Before you call an AI tool production-ready: structured outputs validated against a schema; per-request cost logging with alerts; an eval suite gating prompt changes; sandboxed execution for any tool calls; per-task budget limits; a kill switch; and a fallback behavior for when the model is down or over budget — because it will be.

See also

• Agentic AI vs Generative AI: One Is a Model, the Other Is an Architecture

Frequently Asked Questions

What's the best language and framework for building AI tools?

The one your team already ships. Python has the deepest AI library ecosystem and TypeScript has first-class SDKs from every major provider — both are fully viable. Frameworks (LangChain, LlamaIndex, agent SDKs) are optional conveniences; the load-bearing pieces are structured outputs, evals, and sandboxing, which you can build with the standard library of any language.

Do I need to fine-tune a model to build a useful AI tool?

Usually not. Retrieval (RAG) handles "the model doesn't know my data" and is cheaper, faster to iterate, and instantly updatable. Fine-tuning is for stable style/format requirements at volume — and it adds a training pipeline, eval burden, and deployment complexity that most tools never need.

How do I keep AI tool costs under control?

Log tokens and cost per request from day one, cache aggressively (identical inputs are common in real usage), route easy requests to cheaper models and hard ones to frontier models, and enforce hard per-task budgets in code rather than hoping. Cost regressions should page someone the same way latency regressions do.

How is building an AI agent different from building an AI tool?

A tool makes one model call (or a fixed pipeline of them) per user action; an agent loops — deciding which actions to take, observing results, and continuing. Agents multiply every concern in this post: more calls per task, more credentials in play, more failure modes. Build the tool version first; add agency only where the loop demonstrably beats the pipeline. The distinction is the subject of AI agents vs agentic AI.

Building AI capability into your platform and want the infrastructure side done right? Let's talk.

For fetching and rendering live pages as a data source, see Cloudflare Browser Rendering and the Crawl API.

Official References

• vLLM documentation — serving, batching and parallelism options
• Scheduling GPUs — device plugins and GPU resource requests