Self-Host Ollama: A Private LLM Server You Can Reach From Anywhere

Running Ollama on your laptop is the right place to start, and the wrong place to stay. Your laptop sleeps, its fans spin up during every generation, and the model is only available when you're sitting in front of it.

Moving Ollama to a server fixes all three. It also introduces the single most dangerous mistake in this entire topic, so let me put it before the first command rather than bury it in a warning box halfway down:

Ollama has no authentication. None. Anyone who can reach port 11434 can run any model, load arbitrary models onto your disk, and burn your GPU indefinitely. Setting OLLAMA_HOST=0.0.0.0 on a cloud VM with an open security group means you have published a free GPU to the internet, and scanners find these in hours.

The entire design of this tutorial is to get remote access without ever doing that. We'll bind Ollama to a private network interface that only your own devices can see.

What You'll Build

• Ollama running as a tuned systemd service on a dedicated Linux server
• GPU acceleration enabled and verified (if your server has one)
• A private network joining your server, laptop, and phone — with no ports open to the internet
• Your local model reachable from any of your devices by hostname
• Models preloaded and pinned in memory so the first request isn't slow
• A firewall that fails closed if the private network ever drops

Step 1: Install Ollama on the Server

SSH into the server and run the standard installer, which sets up a systemd service and a dedicated ollama user:

curl -fsSL https://ollama.com/install.sh | sh

Confirm it's running and note the default bind address:

systemctl status ollama
ss -tlnp | grep 11434

You should see it listening on 127.0.0.1:11434. That's localhost-only, and for the moment that's exactly right — we're not changing it until there's a private network to bind to.

Step 2: Enable the GPU (If You Have One)

Skip this if your server is CPU-only; everything still works, just slower.

For an NVIDIA card, a working driver must be present before Ollama can use it. Don't pin a version number from a tutorial — driver branches move, and the one that was current when this was written may already be superseded. Let Ubuntu pick:

sudo apt update
ubuntu-drivers devices

That lists the available branches for your card and marks one recommended. Install it:

sudo ubuntu-drivers install
sudo reboot

After the reboot, verify the driver sees the card:

nvidia-smi

Ollama detects a working GPU automatically — there's no configuration flag to set. Restart it so it re-probes, then confirm:

sudo systemctl restart ollama
ollama pull qwen3.5:9b
ollama run qwen3.5:9b "hello" >/dev/null
ollama ps

The PROCESSOR column in ollama ps is the answer. It will say 100% GPU, 100% CPU, or a split like 40%/60% CPU/GPU when the model was too large to fit entirely in VRAM. A split is not an error, but it is much slower than full GPU — drop to a smaller model if you see one.

While the model is loaded, watch utilisation from a second terminal:

watch -n1 nvidia-smi

Step 3: Join Your Devices to a Private Network

This is the step that makes remote access safe. Rather than opening a port, we give the server a private IP that only your own devices can route to. Tailscale is the least-effort way to do this — it builds encrypted WireGuard tunnels directly between your devices with no port forwarding.

On the server:

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

Follow the printed URL to authenticate. Then install Tailscale on your laptop and phone from tailscale.com/download and sign in with the same account.

Find the server's private address:

tailscale ip -4

You'll get something in the 100.x.y.z range. Note it, and note the machine's Tailscale hostname (tailscale status shows it) — with MagicDNS you can use the name instead of the IP.

If you'd rather own the whole stack, a self-hosted WireGuard server achieves the same result with more control and more maintenance. The requirement either way is identical: a private interface to bind to.

Step 4: Bind Ollama to the Private Interface

Now, and only now, we change the bind address. Edit the service:

sudo systemctl edit ollama.service

Add the following, substituting your own Tailscale IP:

[Service]
Environment="OLLAMA_HOST=100.x.y.z:11434"
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_KEEP_ALIVE=-1"

What each one does:

• OLLAMA_HOST — binds to the private address specifically. Not 0.0.0.0, which would also bind the public interface. This single choice is what keeps the server off the scanners.
• OLLAMA_CONTEXT_LENGTH — raises the context window from the 4096-token default. Costs memory; 8192 is a sane starting point.
• OLLAMA_KEEP_ALIVE=-1 — keeps models resident in memory indefinitely instead of unloading after five minutes. On a dedicated server this is what you want: no cold-start delay on the first request of the day.

Apply and verify:

sudo systemctl daemon-reload
sudo systemctl restart ollama
ss -tlnp | grep 11434

The listen address should now be your 100.x.y.z address and nothing else. Confirm the public interface is not listening:

ss -tln | grep -c '0.0.0.0:11434'

That must print 0. If it prints anything else, stop and fix it before continuing.

Step 5: Call It From Another Device

From your laptop — connected to the same private network, anywhere in the world:

curl http://<server-hostname>:11434/api/generate -d '{
  "model": "qwen3.5:9b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

Point any OpenAI-compatible tool at it by setting the base URL:

1from openai import OpenAI
2
3client = OpenAI(
4    base_url="http://<server-hostname>:11434/v1/",
5    api_key="ollama",  # required by the SDK, ignored by Ollama
6)
7
8completion = client.chat.completions.create(
9    model="qwen3.5:9b",
10    messages=[{"role": "user", "content": "Say this is a test"}],
11)
12print(completion.choices[0].message.content)

To use the CLI from your laptop against the remote server, set the same variable client-side:

export OLLAMA_HOST=http://<server-hostname>:11434
ollama ls
ollama run qwen3.5:9b

The CLI is just an API client. With that variable set, every command runs against the server while the model stays on the server's hardware.

Step 6: Preload Models So the First Request Is Fast

OLLAMA_KEEP_ALIVE=-1 keeps a model loaded once it has been loaded, but the first request after a reboot still pays the load cost. A tiny systemd unit fixes that.

1sudo tee /etc/systemd/system/ollama-preload.service >/dev/null <<'EOF'
2[Unit]
3Description=Preload Ollama models into memory
4After=ollama.service
5Requires=ollama.service
6
7[Service]
8Type=oneshot
9ExecStartPre=/bin/sleep 5
10ExecStart=/usr/bin/ollama run qwen3.5:9b ""
11RemainAfterExit=yes
12
13[Install]
14WantedBy=multi-user.target
15EOF
16
17sudo systemctl daemon-reload
18sudo systemctl enable --now ollama-preload

The empty prompt loads the model and exits immediately. Combined with KEEP_ALIVE=-1, the model is warm from boot onward.

Step 7: Fail Closed With a Firewall

Binding to the private interface is the real protection. A firewall is the backstop for when something changes it — a config edit, a package upgrade, a future you in a hurry.

If you followed the UFW tutorial, add an explicit deny:

sudo ufw deny 11434/tcp
sudo ufw status numbered

Traffic over the Tailscale interface (tailscale0) bypasses this rule, so your own devices keep working while any packet arriving on the public interface is dropped. Belt and braces, and it costs nothing.

Verify from outside: from a machine not on your private network, curl http://<server-public-ip>:11434 must hang or refuse. If it answers, you have a problem to fix right now.

Common Issues

curl from the laptop times out — check the private network first with tailscale status on both machines, then ping <server-hostname>. If ping works but the port doesn't, Ollama is bound to the wrong address; re-check ss -tlnp | grep 11434 on the server.

ollama ps shows CPU despite a working nvidia-smi — Ollama probes for the GPU at startup, so a driver installed after Ollama started won't be seen. sudo systemctl restart ollama. If it persists, check journalctl -u ollama | grep -i gpu for the detection error.

Model reloads on every request despite KEEP_ALIVE=-1 — an API request can override the server default by passing its own keep_alive field. Check what your client library sends.

The Tailscale IP changed and Ollama won't start — binding to a literal IP is brittle if the address ever changes. Either disable key expiry for the machine in the Tailscale admin console, or bind to tailscale0 by its current address and add a systemd dependency on tailscaled.service so ordering is correct at boot.

Out of memory when a second model loads — KEEP_ALIVE=-1 means nothing ever unloads, and OLLAMA_MAX_LOADED_MODELS allows three concurrently by default, so two or three large models will happily exhaust RAM. Set Environment="OLLAMA_MAX_LOADED_MODELS=1" to pin a single model, or use a finite keep-alive like 30m instead.

Frequently Asked Questions

Why not just put Ollama behind Nginx with a password?

You can, and it's a legitimate pattern — a reverse proxy with basic auth or mTLS in front of 127.0.0.1:11434. It's more moving parts (TLS certificates, renewal, an auth store) for the same outcome as a private network, and a misconfigured proxy fails open in a way a private network doesn't. If you need to share the server with people outside your own devices, the proxy becomes the right answer.

Can I run this on a cheap cloud GPU instance?

Yes, and the economics need checking honestly. A GPU instance capable of running a 27B model comfortably costs real money per hour, and if it's idle most of the day you'll spend more than a hosted API would have cost. Self-hosting wins on privacy, on predictable cost at steady high volume, and on not being rate-limited. It rarely wins on raw price for bursty personal use.

How many people can share one Ollama server?

Few — by default OLLAMA_NUM_PARALLEL is 1, so requests to a model queue one behind another. Raise it (Environment="OLLAMA_NUM_PARALLEL=4" in the same systemd override) and a small team using it occasionally is fine. What you can't tune away is the missing continuous batching, so a team hammering it concurrently will still queue badly — that's the signal to move to a purpose-built inference server. I've written about exactly where that threshold sits.

Does this work with a Raspberry Pi?

A Pi 5 with 16 GB runs small models (2B–4B) at a few tokens per second — genuinely usable for classification and extraction, painful for chat. There's no GPU acceleration path, so everything is CPU-bound. It's a fine always-on endpoint for automation, not a good interactive assistant.

What about running Ollama in Docker or on Kubernetes?

Docker works well and the official image supports GPU passthrough with --gpus=all. Kubernetes is overkill for a single-node personal server — but if you're already running a cluster and want GPU scheduling, autoscaling, and multi-tenancy, that's a different architecture entirely and Ollama probably isn't the right server; see deploying an LLM on Kubernetes.

Tear Down

Remove the preload unit and revert Ollama to localhost-only:

sudo systemctl disable --now ollama-preload
sudo rm /etc/systemd/system/ollama-preload.service
sudo rm /etc/systemd/system/ollama.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart ollama

Remove Ollama and its models entirely:

sudo systemctl stop ollama && sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service
sudo rm $(which ollama)
sudo rm -r /usr/share/ollama
sudo userdel ollama && sudo groupdel ollama

Leave the private network in place if you use it for anything else; otherwise sudo tailscale logout and remove the machine from the admin console.

Official References

• Ollama FAQ — OLLAMA_HOST, OLLAMA_MODELS, OLLAMA_KEEP_ALIVE, and systemd environment configuration
• Linux install and systemd — the unit file, manual install, and journalctl logs
• OpenAI compatibility — supported endpoints and parameters
• Tailscale install — per-distro packages
• Model library — available models and tags

Next steps: work out which models your hardware can actually hold in Local LLM VRAM Requirements, and harden the box itself with Harden SSH Access.