Computers
Peripherals
Components
Supplies
Audio Visual
Mobile Tech
Software
Networking
My products
Product guidesLOCAL AI
Running local AI in a homelab
Where the GPU should live, how to pass it through, what to put in front of it, and how to make a local model a proper service rather than a desktop toy.
Running a language model on your desktop is a good way to find out whether you like it. Running one in your homelab is what makes it useful. The difference is that it stops being an application you open and becomes a service that is simply there — reachable from your phone, from Home Assistant, from a script on another box, at three in the morning, without anyone having to be sitting at a particular machine.
This article is about that transition: where to put the GPU, how to get it into a virtual machine, what to run in front of the model, and the security question that a surprising number of guides skip over entirely.
Where should the GPU live?
Three reasonable answers, in increasing order of faff.
Bare metal on a dedicated box
Simplest, and there is no shame in it. An old desktop with the card in it, Ubuntu or Debian, Ollama installed, done. No passthrough, no IOMMU groups, nothing to go wrong. The cost is that the machine is now a single-purpose appliance, which in a homelab full of virtualised everything can feel like a waste of a perfectly good host.
A container on the host
If the GPU host is already running Docker, running Ollama in a container with the GPU exposed to it is straightforward and avoids passthrough entirely. It keeps the model runner isolated and easy to update, while the driver stays on the host where it is simplest to manage. For most people this is the right balance.
Passed through to a virtual machine
The most flexible and the most work. The card is handed wholesale to a VM, which then owns it exclusively. This is what you want if the GPU host is also running a dozen other guests and you would rather the AI workload were isolated, snapshottable, and independently rebootable.
GPU passthrough on Proxmox, briefly
The official Proxmox wiki page is the authority here and worth reading properly, but the shape of it is:
Hardware. You need an IOMMU with interrupt remapping — Intel VT-d or AMD-Vi. Server-grade boards handle this considerably better than consumer ones.
Firmware. Enable IOMMU (or VT-d) in the BIOS. On AMD platforms it is often on by default.
Kernel. Add intel_iommu=on for Intel on older kernels, and optionally iommu=pt for performance.
Modules. Load vfio, vfio_iommu_type1 and vfio_pci via /etc/modules, then run update-initramfs -u -k all and reboot.
IOMMU groups. The card has to sit in its own group. Related functions and bridges sharing the group is fine; unrelated devices are not. If the grouping is wrong, physically moving the card to a different slot frequently fixes it — this is the single most useful piece of passthrough folklore.
Binding. Bind the card to vfio-pci by device ID, or blacklist the native driver in /etc/modprobe.d/ so the host never claims it.
One consequence catches people out: once the GPU belongs to the guest, you lose framebuffer output through NoVNC and SPICE for that VM. Plan on a physical monitor for the initial setup, or SSH in and never look at a console again.
External GPUs are a legitimate answer
Worth knowing if your server is a mini PC with no room for a full-length card. For inference specifically, the link between host and GPU barely matters — once the model is loaded into VRAM, only token data crosses it. XDA measured over 70 tokens per second running gpt-oss-20b on an RTX 4080 in a Thunderbolt 5 enclosure attached to a handheld. Compare that with gaming over the same link, which typically costs 5 to 15% of performance.
The link does matter in two places: the initial load of tens of gigabytes into VRAM, and any situation where the model does not fit and is spilling across the link on every token. If it fits, an eGPU over Thunderbolt or USB4 is a perfectly sensible way to attach a big card to a small server — and it can be passed through to a VM the same as any other PCIe device.
The stack in front of the model
Ollama on its own gives you an HTTP API on port 11434. That is genuinely all some people need. Most will want a bit more.
Open WebUI is the standard companion: a self-hosted web interface with multi-user accounts, conversation history, document knowledge bases and model switching. It talks to Ollama natively and to anything else speaking the OpenAI API, and ships Docker images including a combined one with Ollama bundled. Do read the licence before deploying it widely — since version 0.6.6 there is a branding-protection clause, with an exemption for deployments of fifty or fewer users in a rolling thirty-day window. Every home deployment is comfortably inside that; not every small business one is.
Behind that, whatever you already use: a reverse proxy for TLS and a sensible hostname, and your existing remote-access arrangement rather than a port forward.
The security bit nobody mentions
By default Ollama binds to localhost. Setting OLLAMA_HOST=0.0.0.0 opens it to the network, which is the whole point of running it in a homelab — but it is worth being explicit about what you have just done.
Ollama has no authentication. None. Anyone who can reach port 11434 can use your GPU, run any prompt they like, and pull or delete models. There is no user model, no token, no rate limit.
On an isolated home VLAN that is an acceptable risk. What it must never be is a port forwarded to the internet. Put a reverse proxy with authentication in front of it, or — much simpler — do not expose it at all and reach it over a private overlay network such as Tailscale or WireGuard when you are away from home. The same applies to Open WebUI, which does at least have accounts, but which still should not be the only thing between the internet and your hardware.
Storage, and a mistake worth avoiding
Models are large and multiply quickly. Ollama stores them in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and under the user profile on Windows, and the OLLAMA_MODELS environment variable moves that wherever you like. Set it before you start downloading, not after you have filled a root partition.
There is a temptation to point it at the NAS. Resist it, or at least think it through. Model loading reads tens of gigabytes sequentially, and doing that over a gigabit link adds minutes to every cold start. Local NVMe for the models you actually use, and the NAS for the archive, is the arrangement that will annoy you least.
What to actually connect it to
This is where a homelab earns its keep, because the model becomes just another internal service that other things can call.
Home Assistant. A local conversation agent means voice control that keeps working when your internet does not, and does not send your household's speech to anyone.
Automation platforms. n8n, Node-RED and similar can call an OpenAI-compatible endpoint like any other HTTP node — summarising feeds, classifying incoming documents, drafting text on a schedule.
Notes and documents. Point Open WebUI's knowledge base at your own material and you have private retrieval over documents you would never upload to a hosted service.
Development tooling. Most editor AI plugins accept a custom base URL. A coding model on your own hardware, with no per-token cost, changes how freely you use it.
The common thread is that OpenAI-compatible endpoint. Once you internalise that your model is reachable at http://your-box:11434/v1/ and that a very large amount of software will happily talk to it, the possibilities stop being about the model and start being about plumbing — which is the fun part.
Monitoring and the practical annoyances
ollama ps is the first thing to check when something feels slow: a PROCESSOR column showing anything other than 100% GPU means part of the model is running on the CPU. Beyond that, the same tooling you already run — a dashboard tile, an uptime check on port 11434, and whatever tells you about GPU temperature and power draw.
Two things will irritate you eventually. Cold starts: the first request after a model has been unloaded takes as long as it takes to read the model off disk. Model juggling: if you have several models and not enough VRAM for all of them, requests queue while idle models are evicted. Neither is a fault, but both are worth knowing about before you conclude something has hung.
Honest limits
A single consumer GPU serves one person well and several people badly. Concurrency is a genuinely different engineering problem, and if you are thinking about a small team rather than a household, a purpose-built serving stack such as vLLM is a better starting point than Ollama.
And the power bill is real. A card drawing 300 to 500 W under load, in a machine that is on permanently, is not free — although in practice inference is bursty and the card idles most of the time, so the figure is usually far lower than the headline TDP suggests.
Where to go from here
If you are still choosing hardware, The best GPUs for local AI right now covers the current market and How much VRAM do you actually need covers the sizing. If you have the hardware and want the software set up properly, start with the Ollama guide.