Run an LLM Locally with Ollama: From Install to Your Own API

Every tutorial about "using an LLM" starts the same way: sign up, get an API key, paste in a credit card. This one doesn't. Ollama downloads a model onto your own machine and runs it there. No key, no bill, no request leaving your laptop — which matters if you're working with client code, internal documents, or anything you'd rather not post to a third party.

The catch is honest and worth stating up front: a model small enough to run on your laptop is not as capable as the largest hosted models. What it is is private, free, offline, and fast enough for a huge range of real work — summarising, drafting, classifying, extracting structured data, and powering local tooling. When local stops being enough, I've written about where that line actually falls.

By the end you'll have a model answering in your terminal and an OpenAI-compatible HTTP API on localhost that existing code can talk to with a two-line change.

What You'll Build

• Ollama installed and running as a background service
• A model chosen to fit your actual RAM, pulled and chatting in the terminal
• The REST API on http://localhost:11434 answering curl requests
• A Python script talking to your local model through the OpenAI SDK
• A custom model variant with its own system prompt, built from a Modelfile

Step 1: Install Ollama

On Linux, one command does everything — it installs the binary, creates an ollama system user, and registers a systemd service:

curl -fsSL https://ollama.com/install.sh | sh

On macOS, download the app from ollama.com/download and drag it to Applications. Launching it starts the same background server.

Piping a remote script into a shell deserves a moment's thought. If you'd rather read it first, curl -fsSL https://ollama.com/install.sh | less and then run it. Or install manually by extracting the release tarball into /usr:

curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr

Verify the install:

ollama -v

On Linux, confirm the service is up:

systemctl status ollama

You should see active (running). Ollama is now listening on 127.0.0.1:11434 — localhost only, which is the right default. Nothing on your network can reach it yet.

Step 2: Pick a Model That Actually Fits

This is the step people get wrong, and it's the difference between "this is great" and "my laptop froze".

A model's memory requirement is roughly its file size on disk, plus a gigabyte or two for context. Ollama serves models quantised to about 4 bits per parameter by default, which works out near 0.6 GB per billion parameters. So a 9B model needs ~6 GB, a 27B model needs ~17 GB.

If it doesn't fit in VRAM, Ollama will spill into system RAM and run — just far slower. If it doesn't fit in RAM either, things get ugly.

Start here based on what you have:

Pull one:

ollama pull qwen3.5:9b

A note that will save you confusion later: several current models — gpt-oss:20b, gemma4:26b, qwen3-coder:30b — use a mixture-of-experts architecture. They hold a large number of parameters in memory but only activate three to four billion of them per token. That means they need the memory of a big model but run at the speed of a small one. Don't let the parameter count scare you off if you have the RAM.

See what you've got locally:

ollama ls

Step 3: Talk to It

ollama run qwen3.5:9b

The first run loads the model into memory, which takes a few seconds. Then you get a prompt. Type a question:

>>> Explain what a reverse proxy does, in three sentences.

Inside that session, /bye or Ctrl+D exits. Type /? to list the other slash commands available in your version — they cover things like clearing the conversation context and inspecting model settings, and the set changes between releases.

You can also do one-shot calls without entering the chat, which is what makes Ollama genuinely useful in scripts:

ollama run qwen3.5:9b "Summarise this in one line: $(cat README.md)"

While a model is loaded, check what's resident and how much memory it's using:

ollama ps

Models unload automatically after five minutes idle — that's the documented default, adjustable with OLLAMA_KEEP_ALIVE. To free the memory immediately:

ollama stop qwen3.5:9b

Step 4: Use the REST API

The reason Ollama is more than a chat toy: that background service exposes an HTTP API the whole time, no extra setup.

curl http://localhost:11434/api/generate -d '{
  "model": "qwen3.5:9b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

Without "stream": false you get a stream of newline-delimited JSON objects, one per token — which is what you want in a real application, and noisy in a terminal.

The response is JSON with the text in .response. Piping through jq makes it readable:

curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen3.5:9b",
  "prompt": "Name three uses for a Raspberry Pi.",
  "stream": false
}' | jq -r '.response'

That's a private LLM available to any script on your machine, over plain HTTP, with no authentication because it's bound to localhost.

Step 5: Point Real Code at It

Ollama also speaks the OpenAI API format. Any library or tool that talks to OpenAI can be redirected to your machine by changing the base URL.

curl -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5:9b",
    "messages": [{ "role": "user", "content": "Say this is a test" }]
  }'

From Python with the official OpenAI SDK (pip install openai):

1from openai import OpenAI
2
3client = OpenAI(
4    base_url="http://localhost:11434/v1/",
5    api_key="ollama",  # required by the SDK, ignored by Ollama
6)
7
8completion = client.chat.completions.create(
9    model="qwen3.5:9b",
10    messages=[{"role": "user", "content": "Say this is a test"}],
11)
12print(completion.choices[0].message.content)

Two lines changed — base_url and a dummy api_key — and an application that was sending your data to a vendor is now running entirely on your hardware. That is the whole pitch for local models in one code block.

Step 6: Make It Yours with a Modelfile

A Modelfile bakes a system prompt and parameters into a reusable named model, so you're not re-pasting instructions every time.

Create a file called Modelfile:

1FROM qwen3.5:9b
2
3# Lower temperature = more deterministic, better for technical work
4PARAMETER temperature 0.3
5
6SYSTEM """
7You are a terse senior platform engineer. Answer in at most five sentences.
8Prefer concrete commands over explanation. If you are unsure, say so plainly
9instead of guessing.
10"""

Build and run it:

ollama create sre -f Modelfile
ollama run sre "my pod is in CrashLoopBackOff, what do I check first?"

sre now behaves that way in the CLI and through the API — just pass "model": "sre". This is how you turn a general model into a purpose-built tool, and it costs nothing but a text file.

Common Issues

Error: could not connect to ollama app — the background service isn't running. On Linux: sudo systemctl start ollama. On macOS, launch the Ollama app. If you installed manually with no systemd unit, run ollama serve in its own terminal.

Generation is painfully slow — the model didn't fit in VRAM and is running on CPU. Check with ollama ps: the PROCESSOR column tells you whether it's on GPU, CPU, or split between them. Drop to a smaller model.

The machine freezes or the process is killed — the model exceeded available RAM and the OOM killer stepped in. Check journalctl -e -u ollama on Linux. Use a smaller model; the table in Step 2 is deliberately conservative.

Answers get cut off or the model forgets earlier context — the default context window is 4096 tokens. Raise it for the session with OLLAMA_CONTEXT_LENGTH=8192 ollama serve, or add PARAMETER num_ctx 8192 to a Modelfile. Longer context costs more memory.

No space left on device — models are large and accumulate fast. ollama ls to see them, ollama rm <model> to delete. On Linux they live in /usr/share/ollama/.ollama/models.

Frequently Asked Questions

Do I need a GPU to run Ollama?

No. Ollama runs on CPU and will work on a machine with no discrete graphics at all — it's just slower, often several tokens per second instead of dozens. A small model like qwen3.5:2b is genuinely usable on CPU. Apple Silicon Macs are a special case: unified memory means the GPU can address all system RAM, so a MacBook with 32 GB runs models that would need an expensive discrete card on a PC.

Is Ollama actually private?

Yes, for inference. The model runs on your machine and prompts never leave it — you can verify by pulling a model, disconnecting from the network, and continuing to chat. Ollama does contact its servers to download models and check for updates, so the model comes from the internet even though your data doesn't.

How much disk space will this use?

Budget roughly the same as the memory figures: ~2 GB for a 2B model, ~6 GB for a 9B, ~16 GB for a 26B. They add up quickly if you experiment, so prune with ollama rm. Set OLLAMA_MODELS to point at a bigger disk if your root partition is tight.

Can I run more than one model at once?

Yes, memory permitting. OLLAMA_MAX_LOADED_MODELS defaults to three (per GPU, or three for CPU inference), and models stay resident for five minutes after last use. Two 9B models loaded simultaneously need roughly 12 GB. ollama ps shows what's currently loaded, and OLLAMA_KEEP_ALIVE controls how long they linger — a negative value keeps them loaded indefinitely.

Is Ollama good enough to replace a paid API?

For many tasks, yes: summarising, classifying, extracting structured data, drafting, and local tooling all work well on a 9B–27B model. For tasks needing the strongest reasoning, very long context, or high concurrency, hosted frontier models still win clearly. The realistic pattern is using local models for the bulk of routine work and reaching for a hosted API for the hard cases.

Tear Down

Remove models you no longer want:

ollama rm qwen3.5:9b
ollama rm sre

Remove Ollama entirely from Linux:

1sudo systemctl stop ollama
2sudo systemctl disable ollama
3sudo rm /etc/systemd/system/ollama.service
4sudo rm $(which ollama)
5sudo rm -r /usr/share/ollama
6sudo userdel ollama
7sudo groupdel ollama

On macOS, quit the app and drag it to the Trash, then rm -rf ~/.ollama to reclaim the model storage.

Official References

• Ollama quickstart — the official getting-started flow
• Linux install — manual install, systemd unit, and logs
• CLI reference — every command and its flags
• OpenAI compatibility — which OpenAI endpoints and parameters are supported
• Model library — every model and tag available to pull
• FAQ — environment variables, storage paths, and networking

Next steps: move this off your laptop and onto a server you can reach from anywhere with Self-Host Ollama with Private Remote Access, or work out exactly which models your hardware can handle in Local LLM VRAM Requirements.