When to Move Off Ollama: The Point Local Inference Stops Scaling

Most advice about local LLMs stops at "it works on your laptop". The interesting question is the one after that: you've built something useful on Ollama, other people want to use it, and now you need to know whether the thing you built will survive contact with a second user.

The honest answer is that it will survive two or three, and then degrade in a way that tuning narrows but doesn't close. That's not a criticism of Ollama — it's a consequence of what it was designed for. Knowing exactly where the wall is saves you from discovering it in production.

The wall is concurrency, not model size

The instinct when things get slow is to reach for a smaller model or a bigger GPU. If your problem is concurrency, neither helps, because the bottleneck isn't compute — it's scheduling.

Out of the box, Ollama processes requests against a loaded model one at a time. The OLLAMA_NUM_PARALLEL setting defaults to 1, so request two waits for request one to finish generating every token. With single-user interactive use this is invisible and completely fine. With five users it means the fifth person waits for four full generations before their first token appears.

You can raise that setting, and you should before concluding you need to migrate. But it changes the ceiling, not the architecture — the llama.cpp backend underneath still schedules work in discrete batches rather than continuously, so raising the limit narrows the gap without closing it.

The counterintuitive part: your GPU is mostly idle while this happens. Generating tokens one at a time for one request is memory-bandwidth-bound, not compute-bound. The card spends most of its time waiting on weights moving from VRAM, with compute units doing very little. You are queueing users behind hardware that is largely unused.

Continuous batching — the core feature of a production inference server like vLLM — fixes exactly this. It runs many requests through the model together, and crucially it lets new requests join a batch already in flight rather than waiting for it to drain. The same GPU that served one request at a time now serves dozens, because those dozens share the same weight-loading cost.

The gap this opens is not 20% or 50%. Published benchmarks on an A100 at 128 concurrent requests put vLLM near 793 output tokens/sec against Ollama's ~41 — and the shape matters more than the exact figures: vLLM's throughput climbs as concurrency rises, while Ollama's plateaus almost immediately. That divergence is the whole argument for migrating, and it's why this isn't a tuning problem you can finish solving where you are.

The signals you've outgrown it

Move when you see these, not before:

More than two or three genuinely concurrent users, after raising OLLAMA_NUM_PARALLEL. Not "three people have accounts" — three people whose requests overlap in time. Try the cheap fix first; if latency is still bad with parallelism raised, that's the real trigger.

GPU utilisation is low while users are waiting. Watch nvidia-smi during a slow period. If utilisation sits well below capacity while a queue exists, you are hitting the batching limit, not a hardware limit. Buying a bigger card will not help.

Cold starts hurt. Ollama unloads models after about five minutes idle. OLLAMA_KEEP_ALIVE=-1 pins them, but that trades away the ability to host several models on one box, because nothing ever frees memory.

You need per-user access control or quotas. Ollama has no authentication at all. Once you need to know who is calling and cap what they use, you're building an authenticating layer anyway — and that layer is a natural place to put a real inference backend behind.

You need predictable tail latency. Ollama offers no meaningful control over queueing. If you have an SLO on p99 latency, you need a server that exposes scheduling and admission control.

Equally, here's when to stay: single user or a small team with bursty, non-overlapping use; strict privacy requirements met by the machine under your desk; and any situation where operational simplicity is worth more than throughput. A great many internal tools never outgrow Ollama, and moving them would be pure cost.

What the migration actually costs

The optimistic framing is that both speak an OpenAI-compatible API, so you change a base URL. That's true of the application layer and misleading about everything else.

What genuinely changes:

Quantisation format. Ollama serves GGUF weights, quantised for CPU-and-GPU-split execution. vLLM can load GGUF, but doing so throws away most of the reason you migrated — benchmarks put GGUF on vLLM around 93 tokens/sec against roughly 741 for the same model in Marlin-kernel AWQ. In practice you pull a different build of the same model rather than carrying your files over. Not hard, but not a config change either.

Memory headroom stops being optional. Ollama's automatic CPU spillover is a comfort blanket — if a model doesn't quite fit in VRAM, it still runs, just slowly. Production inference servers expect the model and its KV cache to fit in VRAM. A model that "worked" on Ollama with a partial CPU split will simply refuse to load. See the VRAM math before sizing hardware.

You now operate a service. Ollama is one systemd unit that mostly looks after itself. A production inference stack brings model loading time, health checks, GPU scheduling, and metrics. If it's running on Kubernetes, add node pools and device plugins — deploying an LLM on Kubernetes covers what that involves, and quantisation and tensor parallelism covers fitting large models across multiple GPUs.

The observability bill arrives. Per-request tracing on an inference service generates far more telemetry than most teams expect, and it lands as a surprise. I've written separately on why AI workloads inflate observability costs.

The realistic estimate for a straightforward migration is days, not hours — mostly spent on model format and capacity planning rather than code.

The intermediate step people skip

There's a move between "Ollama on a laptop" and "vLLM on Kubernetes" that solves a surprising number of cases: keep Ollama, put it on a dedicated always-on server, and pin the model in memory.

That single change eliminates cold starts, removes the laptop-sleeping problem, and gives you a stable private endpoint — while keeping the operational footprint at one systemd unit. It handles a small team's non-concurrent use comfortably. Self-hosting Ollama with private remote access is a 45-minute job and buys most of the benefit people think they need a migration for.

Take the step to a real inference server when concurrency is the problem. Take this step when availability is the problem. They get conflated constantly, and the cheaper fix usually turns out to be the right one.

The decision, compressed

The last row deserves emphasis. Some of the time the honest answer isn't a better local server, it's that the task needs a model you can't run yourself. Reaching for vLLM won't close a capability gap that quantisation and parameter count opened.

Frequently Asked Questions

How many concurrent users can Ollama actually handle?

By default, one at a time — OLLAMA_NUM_PARALLEL ships set to 1. Raise it and you'll comfortably handle a handful of overlapping requests, with the practical ceiling depending on model size, generation length, and available memory. What you can't tune away is the lack of continuous batching, so throughput plateaus early and queueing starts to dominate somewhere in the low single digits of sustained concurrency.

Will a bigger GPU fix my concurrency problem?

Usually not. If GPU utilisation is already low while requests queue, you are limited by request scheduling rather than compute, and a faster card leaves that untouched. Check utilisation during a slow period first — that measurement tells you whether you have a hardware problem or an architecture problem, and it's the cheapest diagnostic available.

Can I use the same model files when I migrate?

Technically yes, practically no. Ollama distributes GGUF weights and vLLM can load them, but GGUF on vLLM runs several times slower than a GPU-oriented format like AWQ or GPTQ with Marlin kernels — so reusing your files forfeits most of the throughput you migrated for. Pull a different build of the same model instead, and budget time to confirm the quantised variant you want exists and that its quality holds up for your task.

Is vLLM the only option after Ollama?

No. vLLM is the most common choice, but TensorRT-LLM, SGLang, and Hugging Face TGI all serve the same role with different tradeoffs around hardware support, ease of setup, and peak throughput. If you're already on Kubernetes, the deciding factor is often which one has the better operator and autoscaling story for your setup rather than raw benchmark numbers.

Should I just use a hosted API instead of self-hosting at all?

Often, yes — and it's worth pricing honestly rather than assuming. Self-hosting wins on privacy, on offline capability, and on cost at steady high utilisation of hardware you own. It loses on bursty workloads, on frontier-model capability, and on the engineering time you spend operating it. A rented GPU idling between bursts is more expensive than per-token pricing.

See also

• What Is Ollama? — the definitional starting point.
• vLLM vs Ollama — a direct feature-by-feature comparison.
• How to Deploy an LLM on Kubernetes — GPU nodes, model serving, and autoscaling.
• Running Llama 3 70B on Kubernetes — quantisation and tensor parallelism for large models.
• Why AI Workloads Blow Up Your Observability Bill — the cost that arrives with the platform.

Trying to work out whether you actually need a production inference stack, or just a better-placed Ollama? Talk to us at Coding Protocols — we size inference infrastructure against real concurrency numbers rather than benchmarks.

Official References

• vLLM documentation — serving, batching and parallelism options
• Scheduling GPUs — device plugins and GPU resource requests