What Is Ollama? Running LLMs on Your Own Machine Explained

Ollama is an open-source tool that downloads and runs large language models directly on your own computer. It packages three things that were previously separate — the model weights, the inference engine that executes them, and a local HTTP server — behind a single command, so ollama run gemma4 goes from nothing to a working chat prompt. Because the model runs on your hardware, there is no API key, no per-token bill, and no prompt leaving the machine.

It is best understood by analogy: Ollama is to language models roughly what Docker is to applications. There's a registry you pull from, a tag-based naming scheme (qwen3.5:9b), a local store of what you've pulled, and a small declarative file — the Modelfile — for building your own variants. That analogy is deliberate, and it's why the tool felt immediately familiar to infrastructure people.

How Ollama works

Three components do the work.

The model registry. ollama pull qwen3.5:9b fetches a model from a public library, the same way docker pull fetches an image. Models are versioned by tag, where the tag usually encodes the parameter count. They're stored locally — on Linux, under /usr/share/ollama/.ollama/models.

The inference engine. Ollama wraps llama.cpp, a C++ implementation that runs transformer models efficiently on consumer hardware. It handles the hard parts automatically: detecting whether you have a GPU, deciding how many layers fit in VRAM, and spilling the remainder to CPU when they don't. That automatic layer-splitting is the reason Ollama works at all on a laptop with a modest graphics card.

The server. Ollama runs as a background service listening on 127.0.0.1:11434. Everything else — the CLI, your scripts, your editor plugin — is a client of that server. This is the part people underrate. It means Ollama isn't a chat application; it's a local inference API that happens to ship with a chat client.

curl http://localhost:11434/api/generate -d '{
  "model": "qwen3.5:9b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

It also exposes an OpenAI-compatible endpoint at /v1/chat/completions. Any code already written against the OpenAI SDK can be redirected to a local model by changing the base URL and passing a dummy API key. That single compatibility decision is most of why Ollama spread so fast — adopting it costs two lines, not a rewrite.

Quantisation: why a 27B model fits on your laptop

A model's parameters are numbers. Trained at 16-bit precision, a 27-billion-parameter model needs about 54 GB just to hold its weights — more than any consumer card has.

Quantisation stores those numbers at lower precision. At roughly 4 bits per parameter, the same model needs about 16 GB. Ollama serves 4-bit quantised weights by default, which is what makes local inference practical at all.

The working rule of thumb: about 0.6 GB of memory per billion parameters, plus a gigabyte or two for context. A 9B model wants ~6 GB; a 20B model wants ~13 GB. A model's download size on disk is a good proxy for the memory it will need.

Quantisation is not free — it degrades output quality slightly. In practice the 4-bit versions of good models remain clearly useful, and the tradeoff is overwhelmingly worth it, because the alternative on consumer hardware is not running the model at all. The full VRAM and quantisation math matters enough to be worth understanding before you choose a model.

One modern wrinkle: several current models use a mixture-of-experts architecture, activating only three to four billion parameters per token despite holding far more in memory. They need the RAM of a large model but run at the speed of a small one, which breaks the old intuition that parameter count predicts speed.

What Ollama is good at

• Privacy-bound work. Client code, internal documents, health or financial data — anything you can't paste into a third-party API. This is the strongest single argument for local inference and it's not really about cost.
• Development and prototyping. Iterating against a free, rate-limit-free endpoint before committing to a hosted provider.
• Offline and edge use. Once pulled, a model needs no network at all.
• Local tooling and automation. Scripts that summarise, classify, or extract structured data, running on a schedule without accruing charges.
• Predictable cost at steady volume. Hardware you already own has no marginal cost per token.

When Ollama is the wrong tool

Ollama optimises for one person getting one model running easily. Push past that and it stops fitting:

• Serving multiple concurrent users. OLLAMA_NUM_PARALLEL defaults to 1, so requests to a model queue by default, and raising it narrows the gap without closing it — there's no continuous batching underneath. Production serving needs that, which is what vLLM exists to do, and the gap between the two is large.
• Anything needing authentication. Ollama has none. Exposing port 11434 publicly hands anyone a free GPU. Access control has to come from the network or a reverse proxy.
• Frontier-model reasoning. A quantised 27B model is not competitive with the largest hosted models on hard reasoning, long-context work, or code generation at the top end. Pretending otherwise leads to disappointment.
• Kubernetes and autoscaling. If you need GPU scheduling, horizontal scaling, and multi-tenancy, you want a purpose-built stack — see deploying an LLM on Kubernetes.

Common misconceptions

"Ollama is a model." It isn't. It's a runtime and package manager for models. The models — Gemma, Qwen, Llama, GPT-OSS, DeepSeek — are trained by other organisations and Ollama distributes and executes them.

"Local means completely offline." Inference is offline; distribution isn't. You pull models over the internet and Ollama checks for updates. Your prompts stay local, which is the part that matters, but the model itself arrived from somewhere.

"You need a powerful GPU." You need enough memory, and a GPU mostly buys speed. Small models run acceptably on CPU. Apple Silicon is a genuine special case: unified memory lets the GPU address all system RAM, so a 32 GB MacBook runs models that would need an expensive discrete card on a PC.

"Ollama and llama.cpp are competitors." Ollama is built on llama.cpp. It adds the registry, the server, model management, and the packaging. Using Ollama means using llama.cpp with the sharp edges filed off.

"Local inference is always cheaper." Only at steady, high utilisation on hardware you already own. A rented cloud GPU sitting idle between bursts of personal use costs more than a hosted API would have.

Frequently Asked Questions

Is Ollama free?

Yes. Ollama is open source under the MIT licence and there is no paid tier for running it. The models it distributes carry their own licences, which vary — most permit commercial use, but some restrict it, so check the licence of any model you plan to ship a product on.

What hardware do I need to run Ollama?

Roughly 0.6 GB of memory per billion parameters. 8 GB of RAM runs small models (2B–4B) usably on CPU; 16 GB comfortably runs a 9B model; 24 GB of VRAM handles 20B–27B models well. No GPU is required — it only affects speed, often dramatically. Apple Silicon Macs punch above their weight because system RAM doubles as VRAM.

What is the difference between Ollama and vLLM?

They target opposite ends of the same problem. Ollama optimises for one user on one machine, with easy setup and automatic CPU/GPU splitting; it serves one request per model at a time by default. vLLM optimises for throughput — continuous batching and paged attention to serve many concurrent requests on server GPUs, with throughput that keeps climbing as concurrency rises. Ollama is far easier to start with; vLLM is what you move to when concurrency matters.

Can I use Ollama with existing OpenAI code?

Yes, in most cases. Ollama implements an OpenAI-compatible API at http://localhost:11434/v1/. Set that as your base_url and pass any non-empty string as the API key — it's required by the SDK and ignored by Ollama. Coverage of the OpenAI surface is good but not total, so unusual parameters may not be honoured.

Is it safe to expose Ollama on my network?

Not without protection. Ollama ships with no authentication whatsoever, so anything that can reach port 11434 can use your hardware freely. Bind it to a private network interface (a VPN or mesh network) rather than 0.0.0.0, or put an authenticating reverse proxy in front of it. Publicly exposed instances get found by scanners quickly.

See also

• Run an LLM Locally with Ollama — a hands-on install-to-API walkthrough.
• Self-Host Ollama with Private Remote Access — moving it to a server safely.
• vLLM vs Ollama — the two tools compared directly.
• Local LLM VRAM Requirements — the memory math behind model selection.

Weighing local inference against a hosted API for something real? Talk to us at Coding Protocols — we help teams work out where the privacy, cost, and throughput lines actually fall for their workload.

Official References

• vLLM documentation — serving, batching and parallelism options
• Scheduling GPUs — device plugins and GPU resource requests