AI & LLM Infrastructure
Running models yourself, and knowing when not to.
Serving and scaling large language models on your own infrastructure — Ollama and vLLM, GPU scheduling and the operator that makes it work, quantisation and the VRAM maths behind it, and the cost and observability problems that only appear once inference is in production.
21 articles in this series
- 019 min
EKS Auto Mode for GPU Workloads
Auto Mode ships NVIDIA drivers and the device plugin, so the GPU Operator stops being your problem. In exchange you get a 14-day node expiry, weekly AMI replacement, and no SSH — which matters a lot more for training than for inference.
Read - 0212 min
Scale-to-Zero LLM Inference on Kubernetes
An idle GPU pod bills exactly the same as a saturated one. Here is how to scale LLM inference to zero replicas with KEDA, and how to cut the cold start from eight minutes to under two.
Read - 0310 min
Track LLM Inference Cost in Kubernetes With OpenCost
Your GPU node bill is easy to find. Your cost per million tokens is not — and it's the only number that compares to API pricing. Here's how to derive it by joining OpenCost's dollars to vLLM's token counters.
Read - 049 min
Give an AI Agent Read-Only Access to Kubernetes
Read-only is the wrong mental model for agent credentials. The agent can be steered by the data it reads, and everything it reads leaves your cluster. Here's what changes about scoping access.
Read - 0514 min
Build an AI Kubernetes Troubleshooting Agent
A read-only agent that reads pod status, events, and logs, then tells you why a workload is broken. The interesting part isn't the prompt — it's the RBAC ceiling and the token budget.
Read - 0613 min
LangGraph vs LangChain: When a Graph Beats a Chain (and When It Doesn't)
LangGraph and LangChain are not competitors — LangGraph is built by the LangChain team to solve the one problem chains can't: loops. Here's the real architectural difference, the same task written in both, and an honest decision framework for when a stateful graph earns its complexity.
Read - 0712 min
Agentic AI vs Generative AI: One Is a Model, the Other Is an Architecture
Generative AI produces content from a prompt — one inference, stateless, done. Agentic AI wraps that same model in a loop with goals, tools, memory, and feedback from the environment. The difference isn't the model; it's the architecture around it — and it changes your latency, cost model, failure modes, and security surface.
Read - 088 min
What Is Ollama? Running LLMs on Your Own Machine Explained
Ollama is an open-source tool that downloads and runs large language models locally on your own computer. It bundles model weights, quantisation, and a server into a single command, exposing an HTTP API on localhost so applications can use a private LLM with no API key and no data leaving the machine.
Read - 099 min
When to Move Off Ollama: The Point Local Inference Stops Scaling
Ollama is excellent until roughly the third concurrent user, and then it falls off a cliff — not gradually, but structurally, because it lacks continuous batching. Here's the specific signal that tells you you've outgrown it, why the failure is architectural rather than a tuning problem, and what the migration actually costs.
Read - 109 min
Local LLM VRAM Requirements: The Math Behind Which Models You Can Run
Every 'best local model' list goes stale in months. The arithmetic underneath it doesn't. Here's how to compute the memory a model needs from its parameter count and quantisation level, why the KV cache is the number people forget, and how mixture-of-experts broke the old rule that parameters predict speed.
Read - 1111 min
Why AI Workloads Blow Up Your Observability Bill — and How to Cap It
GPU spend gets scrutinized line by line. The observability bill that grows right alongside it doesn't — until it lands as a surprise. AI workloads inflate exactly the three axes observability vendors bill on: active series, log volume, and spans. Here's the mechanics of why, and a playbook to cap it.
Read - 125 min
The Best AI Workflow Orchestration Tools in 2026 (and When Each One Is Wrong)
Airflow, Dagster, Prefect, Temporal, Flyte, Kubeflow, LangGraph, n8n — everyone claims to orchestrate AI workflows now. They solve different problems. Here's how I'd actually choose between them, with the failure modes the landing pages don't mention.
Read - 137 min
How to Build AI Tools: A Platform Engineer's Stack, From Prompt to Production
Most 'build an AI tool' guides stop at the API call. The actual work is everything around it: structured outputs, tool calling, evals, sandboxing, cost controls, and shipping the thing behind real infrastructure. Here's the stack I'd use in 2026, layer by layer.
Read - 149 min
AI Agents vs Agentic AI: What the Terms Actually Mean (and Why It Matters for Your Platform)
The terms get used interchangeably in every vendor pitch, but they describe different things: an AI agent is a software artifact; agentic AI is a system property. Here's the distinction that actually matters when you're building or operating these systems.
Read - 158 min
vLLM vs Ollama: Choosing the Right LLM Inference Tool
vLLM is built for production GPU serving with high throughput. Ollama is built for running models locally with minimal setup. They solve different problems — using either one in the wrong context wastes either significant GPU capacity or fails under any real load.
Read - 1614 min
Running Llama 3 70B on Kubernetes: AWQ Quantization and Tensor Parallelism
Llama 3 70B at full BF16 precision needs 140 GB of VRAM — more than a single A100 80GB can hold. AWQ 4-bit quantization cuts that to ~40 GB. Tensor parallelism splits the model across GPUs. Together they make 70B inference practical on g5.12xlarge. Here's the VRAM math, the quantization tradeoffs, and the exact Kubernetes configuration.
Read - 1713 min
NVIDIA GPU Operator: Running GPU Workloads on Kubernetes
Running GPU workloads on Kubernetes without the GPU Operator means manually installing NVIDIA drivers, the container runtime, device plugin, and monitoring components on every GPU node. The GPU Operator automates all of this. But it also adds complexity — this post covers what the operator manages, how to configure it for different GPU sharing models, and the production failure modes.
Read - 1813 min
Kubernetes GPU Workloads: Scheduling Machine Learning Jobs on EKS
Running GPU workloads in Kubernetes requires the right node configuration (NVIDIA device plugin, appropriate instance types), the right scheduling primitives (resource requests, node selectors, tolerations), and the right job patterns for training vs inference. Getting any of these wrong means GPU memory errors, CUDA version mismatches, or expensive GPU nodes sitting idle.
Read - 1914 min
How to Deploy an LLM on Kubernetes: GPU Nodes, Model Serving, and Autoscaling
Running LLMs in production on Kubernetes means GPU node management, model serving (vLLM or Triton), resource limits that actually work, and KEDA-based autoscaling. Here's the full picture.
Read - 209 min
NVIDIA OpenShell: The Missing Security Layer for Autonomous AI Agents
AI coding agents now write and execute code autonomously. NVIDIA's OpenShell is a Rust-built, policy-enforced sandbox runtime that finally puts a security boundary between your agent and your production infrastructure.
Read - 216 min
Slash LLM Costs by 60%: The Ultimate Guide to JSON Compression with TOON
Discover how TOON format reduces JSON token usage by nearly 60%. Learn how to cut API costs, optimize data transmission, and improve LLM performance today.
Read
Other series
Troubleshooting Guides
Paste the error, get the fix.
Tool Comparisons
X vs Y, decided on what matters in production.
What Is… Explainers
Plain definitions of terms that get used loosely.
AWS Service Guides
One service at a time, from someone who runs them.
CI/CD Pipelines
Building, testing and shipping, without the pipeline owning you.
Kubernetes
The cluster, end to end.