Self-Hosted Ollama Homelab: 3 Mistakes Running Local LLMs
We thought setting up a self-hosted Ollama homelab for DevOps assistance would take an afternoon. Three OOM crashes, one exposed API endpoint, and a silent CPU fallback later, here’s what actually works — and what we’d do differently from day one.
Context: Why We Wanted a Local LLM for Homelab Automation
Our homelab runs a fairly typical self-hosted DevOps stack: Gitea for source control, Drone CI for pipelines, Proxmox for VM management, and a handful of Docker Compose services scattered across two physical hosts. Day-to-day tasks involve writing Ansible playbooks, debugging systemd units, generating Dockerfiles, and occasionally reverse-engineering someone else’s Terraform. All of that involves a lot of back-and-forth with language models.
The problem with cloud-based LLM APIs — ChatGPT, Claude, Gemini — is that they require pasting real infrastructure configs into a browser. Hostnames, internal IP ranges, secrets that slipped into a playbook comment, service account names. None of that should leave the network. We also have air-gapped lab segments that physically can’t reach the internet, and we wanted consistent tooling across both environments.
We chose Ollama for a few concrete reasons: it ships as a single binary, it manages model downloads via a clean CLI, and it exposes an OpenAI-compatible REST API on port 11434 out of the box. That last point matters — it means any tool that supports the OpenAI API (Continue.dev in VS Code, shell scripts using curl, custom Python utilities) works against Ollama without modification. The plan was straightforward. The execution was not.
Mistake 1: Assuming Any GPU Would “Just Work” with Ollama
The first host we tried had an NVIDIA GTX 1060 3GB installed. We ran ollama run llama3, watched it load, and immediately noticed something was wrong. Inference was crawling at roughly 2 tokens per second. We expected 25 or more. No error. No warning. The model just loaded silently into system RAM and ran entirely on CPU.
We burned two hours checking the wrong things — model quantization, RAM speed, thermal throttling — before running ollama ps and seeing 100% CPU next to the loaded model. Then we checked the logs with OLLAMA_DEBUG=1 ollama serve and found the real message:
msg=”no GPU detected”
The GTX 1060 was visible to nvidia-smi. The driver was loaded. But Ollama requires CUDA 11.8 or higher, and that host was running CUDA 11.4. The mismatch was silent from the client’s perspective — Ollama simply fell back to CPU without surfacing any error to the caller.
Watch out for this: nvidia-smi showing your GPU does not mean Ollama will use it. Always verify with OLLAMA_DEBUG=1 and look for offload layers: 32/32 in the output. That line confirms full GPU acceleration. Anything less means partial or zero offload.
The fix required three steps: updating the CUDA toolkit to 11.8, pinning nvidia-container-toolkit to version 1.14.3 (earlier versions had inconsistent behavior with Docker runtime detection), and ensuring /etc/docker/daemon.json had “default-runtime”: “nvidia” set. After that, the same model ran at 28 tokens per second on the same hardware. Same model, same host, completely different experience.
We also verified the CUDA version properly this time:
# Check CUDA version — must be >= 11.8 for Ollama GPU support
nvcc –version
# Confirm Docker is using the nvidia runtime
docker info | grep -i runtime
# Verify GPU offload after starting Ollama with debug logging
OLLAMA_DEBUG=1 ollama serve 2>&1 | grep -i “offload layers”
Mistake 2: Running Ollama Directly on the Host Without Resource Limits
Once GPU acceleration was working, we got greedy and pulled a larger model. codellama:13b seemed like the right tool for generating longer Ansible roles and reviewing Dockerfiles with more context. We pulled it, ran it, and about forty seconds into the first inference request, everything went sideways.
The OOM killer fired. It took down our Gitea container mid-push. A developer on the team lost a commit they hadn’t pushed yet. The codellama:13b model in Q4_K_M quantization needs roughly 8.5GB of VRAM — and on a 16GB host already running a dozen services, the model load spiked total RAM consumption to 14GB instantaneously. The kernel picked Gitea as the most expendable process. It wasn’t.
The root cause was straightforward: we were running Ollama directly on the host with no cgroup constraints, no memory limits, and no model eviction policy. In Ollama v0.1.38 and later, OLLAMA_MAX_LOADED_MODELS controls how many models stay resident in memory simultaneously. In earlier versions, there’s no eviction control at all. We hadn’t set either OLLAMA_MAX_LOADED_MODELS or OLLAMA_NUM_PARALLEL, so Ollama defaulted to greedy behavior — loading whatever was asked, keeping it resident, and accepting parallel requests that multiplied memory pressure.
Watch out for this: OLLAMA_NUM_PARALLEL defaults to 4 in recent versions. On a constrained homelab host, four simultaneous inference requests against a 7B model will absolutely OOM your other services. Set it to 1 unless you have dedicated hardware.
The correct approach was to containerize Ollama with explicit resource caps and isolate it from critical services on a dedicated Docker network. We moved everything into Docker Compose, set mem_limit: 10g, and added environment variables to enforce single-model, single-request behavior.
Mistake 3: Exposing the Ollama API Without Authentication
This one is embarrassing to admit. The default ollama serve binds to 127.0.0.1, which is fine for bare-metal installs. But our Docker Compose setup used ports: “11434:11434” without thinking through what that means on a flat home network. It published the port to every interface on the host — which sits on the same physical switch as our IoT VLAN due to a pfSense VLAN rule we’d misconfigured months earlier.
Ollama has no built-in authentication as of v0.1.x. Anyone who could reach port 11434 could call GET /api/tags to enumerate loaded models, send inference requests, or — and this is the one that actually worried us — trigger a POST /api/pull to download a multi-gigabyte model and exhaust disk space. There’s no rate limiting either. A single unauthenticated pull request from a compromised IoT device could fill a volume.
We also had OLLAMA_HOST=0.0.0.0 set inside the container, which is necessary for the container’s internal binding but easy to misread as “only affects the container namespace.” It doesn’t change the fact that the Docker port mapping exposes it externally.
The fix was an Nginx reverse proxy with HTTP Basic Auth sitting in front of Ollama, combined with binding the proxy’s published port to 127.0.0.1:8080 on the host. Remote access goes through Tailscale, which restricts it to trusted nodes only. For the health check endpoint used by monitoring tools, we carved out an auth-exempt location /api/tags block in Nginx.
What We Do Differently Now: A Stable, Reproducible Ollama Stack
The current setup avoids all three mistakes. Here’s the full Docker Compose configuration we run in production on the homelab. It uses a pinned Ollama image, GPU passthrough via the nvidia runtime, hard memory caps, and an Nginx auth proxy — all wired together with a named volume for model persistence.
This Docker Compose file defines the full hardened stack. Read the inline comments carefully — several of the environment variables are non-obvious in their effect:
# docker-compose.yml — Production-hardened Ollama stack for homelab
# Tested with Ollama v0.1.38, Docker 24.x, nvidia-container-toolkit 1.14.3
version: “3.9”
services:
ollama:
image: ollama/ollama:0.1.38 # pinned — avoid surprise API changes
container_name: ollama
restart: unless-stopped
runtime: nvidia # requires nvidia-container-toolkit
environment:
– OLLAMA_HOST=0.0.0.0 # bind inside container; proxy handles external auth
– OLLAMA_NUM_PARALLEL=1 # prevent concurrent request memory spikes
– OLLAMA_MAX_LOADED_MODELS=1 # evict previous model before loading new one
– OLLAMA_DEBUG=0 # set to 1 temporarily to verify GPU offload layers
volumes:
– ollama_models:/root/.ollama # persist models across container rebuilds
networks:
– ollama_internal # isolated from critical services network
deploy:
resources:
limits:
memory: 10g # hard cap — prevents OOM-killing neighbors
reservations:
devices:
– driver: nvidia
count: 1
capabilities: [gpu]
healthcheck:
test: [“CMD”, “curl”, “-f”, “http://localhost:11434/api/tags”]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s # model server takes ~15s to initialize
ollama-proxy:
image: nginx:1.25-alpine
container_name: ollama-proxy
restart: unless-stopped
ports:
– “127.0.0.1:8080:80” # bind to loopback only; use Tailscale for remote access
volumes:
– ./nginx/ollama.conf:/etc/nginx/conf.d/default.conf:ro
– ./nginx/.htpasswd:/etc/nginx/.htpasswd:ro
networks:
– ollama_internal
depends_on:
ollama:
condition: service_healthy
volumes:
ollama_models:
driver: local
networks:
ollama_internal:
driver: bridge
internal: false # set true if Ollama should have no outbound internet
The Nginx config below handles streaming inference correctly. The two most common mistakes people make here are forgetting proxy_buffering off (which causes the response to appear all at once instead of streaming) and leaving proxy_read_timeout at the default 60 seconds, which causes 504 errors mid-generation on longer prompts:
# nginx/ollama.conf — Reverse proxy with Basic Auth for Ollama API
# Generate .htpasswd: htpasswd -c nginx/.htpasswd devops
# Set proxy_read_timeout high — streaming inference takes time
server {
listen 80;
server_name _;
# Basic auth — replace with mTLS for higher-security environments
auth_basic “Ollama API”;
auth_basic_user_file /etc/nginx/.htpasswd;
location / {
proxy_pass http://ollama:11434;
proxy_http_version 1.1;
# Critical: streaming responses require these headers
proxy_set_header Connection ”;
proxy_buffering off;
proxy_cache off;
# Increase timeouts — default 60s causes 504 on long generations
proxy_read_timeout 300s;
proxy_connect_timeout 10s;
proxy_send_timeout 300s;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
# Health endpoint — exempt from auth for monitoring tools
location /api/tags {
auth_basic off;
proxy_pass http://ollama:11434/api/tags;
}
}
For model selection, we settled on two workhorses that fit within 6GB VRAM with quantization: codellama:7b in Q4_K_M for quick completions and inline code suggestions, and mistral:7b-instruct-v0.2-q4_K_M for longer reasoning tasks like reviewing a Terraform plan or explaining a failing systemd unit. The 13B models sound appealing but they’re genuinely impractical on homelab hardware unless you have a dedicated GPU with 10GB+ VRAM.
We also pin models using digest references in our Modelfiles — FROM llama3:8b-instruct@sha256:abc123… — so a weekly ollama pull cron job doesn’t silently swap out a model we’ve tuned prompts for. Model versioning matters more than most people realize when you’re building automation on top of LLM output.
For observability, Ollama has no native Prometheus endpoint, so we rely on ollama ps and ollama list in a lightweight cron-based check, plus the Docker healthcheck defined in the Compose file. It’s not elegant, but it catches the cases that actually happen: the model server failing to start, or a model getting stuck in a loading state. More details on our homelab monitoring approach are over at kuryzhev.cloud.
Running a self-hosted Ollama homelab is genuinely useful once the stack is stable. The self-hosted Ollama homelab we have now handles dozens of DevOps assistance requests per day with zero data leaving the network, predictable resource usage, and a setup that survives container restarts and host reboots without manual intervention. Getting there required making all three of these mistakes first — hopefully this saves you the OOM crashes.
Related
More Docker Compose production patterns and service hardening
Homelab monitoring, alerting, and observability setups
Container and network security configurations for self-hosted stacks
Fuente: Artículo original