{"id":3175,"date":"2026-06-26T07:48:09","date_gmt":"2026-06-26T07:48:09","guid":{"rendered":"https:\/\/tucumandevelopers.com\/index.php\/2026\/06\/26\/self-hosted-ollama-homelab-3-mistakes-running-local-llms\/"},"modified":"2026-06-26T07:48:09","modified_gmt":"2026-06-26T07:48:09","slug":"self-hosted-ollama-homelab-3-mistakes-running-local-llms","status":"publish","type":"post","link":"https:\/\/tucumandevelopers.com\/index.php\/2026\/06\/26\/self-hosted-ollama-homelab-3-mistakes-running-local-llms\/","title":{"rendered":"Self-Hosted Ollama Homelab: 3 Mistakes Running Local LLMs"},"content":{"rendered":"
Originally published on kuryzhev.cloud
\nWe thought setting up a self-hosted Ollama homelab for DevOps assistance would take an afternoon. Three OOM crashes, one exposed API endpoint, and a silent CPU fallback later, here’s what actually works \u2014 and what we’d do differently from day one.
\nContext: Why We Wanted a Local LLM for Homelab Automation
\nOur homelab runs a fairly typical self-hosted DevOps stack: Gitea for source control, Drone CI for pipelines, Proxmox for VM management, and a handful of Docker Compose services scattered across two physical hosts. Day-to-day tasks involve writing Ansible playbooks, debugging systemd units, generating Dockerfiles, and occasionally reverse-engineering someone else’s Terraform. All of that involves a lot of back-and-forth with language models.
\nThe problem with cloud-based LLM APIs \u2014 ChatGPT, Claude, Gemini \u2014 is that they require pasting real infrastructure configs into a browser. Hostnames, internal IP ranges, secrets that slipped into a playbook comment, service account names. None of that should leave the network. We also have air-gapped lab segments that physically can’t reach the internet, and we wanted consistent tooling across both environments.
\nWe chose Ollama for a few concrete reasons: it ships as a single binary, it manages model downloads via a clean CLI, and it exposes an OpenAI-compatible REST API on port 11434 out of the box. That last point matters \u2014 it means any tool that supports the OpenAI API (Continue.dev in VS Code, shell scripts using curl, custom Python utilities) works against Ollama without modification. The plan was straightforward. The execution was not.
\nMistake 1: Assuming Any GPU Would “Just Work” with Ollama
\nThe first host we tried had an NVIDIA GTX 1060 3GB installed. We ran ollama run llama3, watched it load, and immediately noticed something was wrong. Inference was crawling at roughly 2 tokens per second. We expected 25 or more. No error. No warning. The model just loaded silently into system RAM and ran entirely on CPU.
\nWe burned two hours checking the wrong things \u2014 model quantization, RAM speed, thermal throttling \u2014 before running ollama ps and seeing 100% CPU next to the loaded model. Then we checked the logs with OLLAMA_DEBUG=1 ollama serve and found the real message:
\nmsg=”no GPU detected”
\nThe GTX 1060 was visible to nvidia-smi. The driver was loaded. But Ollama requires CUDA 11.8 or higher, and that host was running CUDA 11.4. The mismatch was silent from the client’s perspective \u2014 Ollama simply fell back to CPU without surfacing any error to the caller.
\nWatch out for this: nvidia-smi showing your GPU does not mean Ollama will use it. Always verify with OLLAMA_DEBUG=1 and look for offload layers: 32\/32 in the output. That line confirms full GPU acceleration. Anything less means partial or zero offload.
\nThe fix required three steps: updating the CUDA toolkit to 11.8, pinning nvidia-container-toolkit to version 1.14.3 (earlier versions had inconsistent behavior with Docker runtime detection), and ensuring \/etc\/docker\/daemon.json had “default-runtime”: “nvidia” set. After that, the same model ran at 28 tokens per second on the same hardware. Same model, same host, completely different experience.
\nWe also verified the CUDA version properly this time:
\n# Check CUDA version \u2014 must be >= 11.8 for Ollama GPU support
\nnvcc –version<\/p>\n

# Confirm Docker is using the nvidia runtime
\ndocker info | grep -i runtime<\/p>\n

# Verify GPU offload after starting Ollama with debug logging
\nOLLAMA_DEBUG=1 ollama serve 2>&1 | grep -i “offload layers”
\nMistake 2: Running Ollama Directly on the Host Without Resource Limits
\nOnce GPU acceleration was working, we got greedy and pulled a larger model. codellama:13b seemed like the right tool for generating longer Ansible roles and reviewing Dockerfiles with more context. We pulled it, ran it, and about forty seconds into the first inference request, everything went sideways.
\nThe OOM killer fired. It took down our Gitea container mid-push. A developer on the team lost a commit they hadn’t pushed yet. The codellama:13b model in Q4_K_M quantization needs roughly 8.5GB of VRAM \u2014 and on a 16GB host already running a dozen services, the model load spiked total RAM consumption to 14GB instantaneously. The kernel picked Gitea as the most expendable process. It wasn’t.
\nThe root cause was straightforward: we were running Ollama directly on the host with no cgroup constraints, no memory limits, and no model eviction policy. In Ollama v0.1.38 and later, OLLAMA_MAX_LOADED_MODELS controls how many models stay resident in memory simultaneously. In earlier versions, there’s no eviction control at all. We hadn’t set either OLLAMA_MAX_LOADED_MODELS or OLLAMA_NUM_PARALLEL, so Ollama defaulted to greedy behavior \u2014 loading whatever was asked, keeping it resident, and accepting parallel requests that multiplied memory pressure.
\nWatch out for this: OLLAMA_NUM_PARALLEL defaults to 4 in recent versions. On a constrained homelab host, four simultaneous inference requests against a 7B model will absolutely OOM your other services. Set it to 1 unless you have dedicated hardware.
\nThe correct approach was to containerize Ollama with explicit resource caps and isolate it from critical services on a dedicated Docker network. We moved everything into Docker Compose, set mem_limit: 10g, and added environment variables to enforce single-model, single-request behavior.
\nMistake 3: Exposing the Ollama API Without Authentication
\nThis one is embarrassing to admit. The default ollama serve binds to 127.0.0.1, which is fine for bare-metal installs. But our Docker Compose setup used ports: “11434:11434” without thinking through what that means on a flat home network. It published the port to every interface on the host \u2014 which sits on the same physical switch as our IoT VLAN due to a pfSense VLAN rule we’d misconfigured months earlier.
\nOllama has no built-in authentication as of v0.1.x. Anyone who could reach port 11434 could call GET \/api\/tags to enumerate loaded models, send inference requests, or \u2014 and this is the one that actually worried us \u2014 trigger a POST \/api\/pull to download a multi-gigabyte model and exhaust disk space. There’s no rate limiting either. A single unauthenticated pull request from a compromised IoT device could fill a volume.
\nWe also had OLLAMA_HOST=0.0.0.0 set inside the container, which is necessary for the container’s internal binding but easy to misread as “only affects the container namespace.” It doesn’t change the fact that the Docker port mapping exposes it externally.
\nThe fix was an Nginx reverse proxy with HTTP Basic Auth sitting in front of Ollama, combined with binding the proxy’s published port to 127.0.0.1:8080 on the host. Remote access goes through Tailscale, which restricts it to trusted nodes only. For the health check endpoint used by monitoring tools, we carved out an auth-exempt location \/api\/tags block in Nginx.
\nWhat We Do Differently Now: A Stable, Reproducible Ollama Stack
\nThe current setup avoids all three mistakes. Here’s the full Docker Compose configuration we run in production on the homelab. It uses a pinned Ollama image, GPU passthrough via the nvidia runtime, hard memory caps, and an Nginx auth proxy \u2014 all wired together with a named volume for model persistence.
\nThis Docker Compose file defines the full hardened stack. Read the inline comments carefully \u2014 several of the environment variables are non-obvious in their effect:
\n# docker-compose.yml \u2014 Production-hardened Ollama stack for homelab
\n# Tested with Ollama v0.1.38, Docker 24.x, nvidia-container-toolkit 1.14.3<\/p>\n

version: “3.9”<\/p>\n

services:
\n ollama:
\n image: ollama\/ollama:0.1.38 # pinned \u2014 avoid surprise API changes
\n container_name: ollama
\n restart: unless-stopped
\n runtime: nvidia # requires nvidia-container-toolkit
\n environment:
\n – OLLAMA_HOST=0.0.0.0 # bind inside container; proxy handles external auth
\n – OLLAMA_NUM_PARALLEL=1 # prevent concurrent request memory spikes
\n – OLLAMA_MAX_LOADED_MODELS=1 # evict previous model before loading new one
\n – OLLAMA_DEBUG=0 # set to 1 temporarily to verify GPU offload layers
\n volumes:
\n – ollama_models:\/root\/.ollama # persist models across container rebuilds
\n networks:
\n – ollama_internal # isolated from critical services network
\n deploy:
\n resources:
\n limits:
\n memory: 10g # hard cap \u2014 prevents OOM-killing neighbors
\n reservations:
\n devices:
\n – driver: nvidia
\n count: 1
\n capabilities: [gpu]
\n healthcheck:
\n test: [“CMD”, “curl”, “-f”, “http:\/\/localhost:11434\/api\/tags”]
\n interval: 30s
\n timeout: 10s
\n retries: 3
\n start_period: 20s # model server takes ~15s to initialize<\/p>\n

ollama-proxy:
\n image: nginx:1.25-alpine
\n container_name: ollama-proxy
\n restart: unless-stopped
\n ports:
\n – “127.0.0.1:8080:80” # bind to loopback only; use Tailscale for remote access
\n volumes:
\n – .\/nginx\/ollama.conf:\/etc\/nginx\/conf.d\/default.conf:ro
\n – .\/nginx\/.htpasswd:\/etc\/nginx\/.htpasswd:ro
\n networks:
\n – ollama_internal
\n depends_on:
\n ollama:
\n condition: service_healthy<\/p>\n

volumes:
\n ollama_models:
\n driver: local<\/p>\n

networks:
\n ollama_internal:
\n driver: bridge
\n internal: false # set true if Ollama should have no outbound internet
\nThe Nginx config below handles streaming inference correctly. The two most common mistakes people make here are forgetting proxy_buffering off (which causes the response to appear all at once instead of streaming) and leaving proxy_read_timeout at the default 60 seconds, which causes 504 errors mid-generation on longer prompts:
\n# nginx\/ollama.conf \u2014 Reverse proxy with Basic Auth for Ollama API
\n# Generate .htpasswd: htpasswd -c nginx\/.htpasswd devops
\n# Set proxy_read_timeout high \u2014 streaming inference takes time<\/p>\n

server {
\n listen 80;
\n server_name _;<\/p>\n

# Basic auth \u2014 replace with mTLS for higher-security environments
\n auth_basic “Ollama API”;
\n auth_basic_user_file \/etc\/nginx\/.htpasswd;<\/p>\n

location \/ {
\n proxy_pass http:\/\/ollama:11434;
\n proxy_http_version 1.1;<\/p>\n

# Critical: streaming responses require these headers
\n proxy_set_header Connection ”;
\n proxy_buffering off;
\n proxy_cache off;<\/p>\n

# Increase timeouts \u2014 default 60s causes 504 on long generations
\n proxy_read_timeout 300s;
\n proxy_connect_timeout 10s;
\n proxy_send_timeout 300s;<\/p>\n

proxy_set_header Host $host;
\n proxy_set_header X-Real-IP $remote_addr;
\n }<\/p>\n

# Health endpoint \u2014 exempt from auth for monitoring tools
\n location \/api\/tags {
\n auth_basic off;
\n proxy_pass http:\/\/ollama:11434\/api\/tags;
\n }
\n}
\nFor model selection, we settled on two workhorses that fit within 6GB VRAM with quantization: codellama:7b in Q4_K_M for quick completions and inline code suggestions, and mistral:7b-instruct-v0.2-q4_K_M for longer reasoning tasks like reviewing a Terraform plan or explaining a failing systemd unit. The 13B models sound appealing but they’re genuinely impractical on homelab hardware unless you have a dedicated GPU with 10GB+ VRAM.
\nWe also pin models using digest references in our Modelfiles \u2014 FROM llama3:8b-instruct@sha256:abc123… \u2014 so a weekly ollama pull cron job doesn’t silently swap out a model we’ve tuned prompts for. Model versioning matters more than most people realize when you’re building automation on top of LLM output.
\nFor observability, Ollama has no native Prometheus endpoint, so we rely on ollama ps and ollama list in a lightweight cron-based check, plus the Docker healthcheck defined in the Compose file. It’s not elegant, but it catches the cases that actually happen: the model server failing to start, or a model getting stuck in a loading state. More details on our homelab monitoring approach are over at kuryzhev.cloud.
\nRunning a self-hosted Ollama homelab is genuinely useful once the stack is stable. The self-hosted Ollama homelab we have now handles dozens of DevOps assistance requests per day with zero data leaving the network, predictable resource usage, and a setup that survives container restarts and host reboots without manual intervention. Getting there required making all three of these mistakes first \u2014 hopefully this saves you the OOM crashes.
\nRelated
\nMore Docker Compose production patterns and service hardening
\nHomelab monitoring, alerting, and observability setups
\nContainer and network security configurations for self-hosted stacks<\/p><\/div>\n

Fuente: Art\u00edculo original<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Originally published on kuryzhev.cloud We thought setting up a self-hosted Ollama homelab for DevOps assistance would take an afternoon. Three OOM crashes, one exposed API endpoint, and a silent CPU fallback later, here’s what actually works \u2014 and what we’d do differently from day one. Context: Why We Wanted a Local LLM for Homelab Automation […]<\/p>\n","protected":false},"author":1,"featured_media":2648,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[41],"tags":[],"class_list":["post-3175","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devto"],"jetpack_publicize_connections":[],"_links":{"self":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/3175","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/comments?post=3175"}],"version-history":[{"count":0,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/3175\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media\/2648"}],"wp:attachment":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media?parent=3175"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/categories?post=3175"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/tags?post=3175"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}