Running local AI in a homelab

Running a language model on your desktop is a good way to find out whether you like it. Running one in your homelab is what makes it useful. The difference is that it stops being an application you open and becomes a service that is simply there — reachable from your phone, from Home Assistant, from a script on another box, at three in the morning, without anyone having to be sitting at a particular machine.

This article is about that transition: where to put the GPU, how to get it into a virtual machine, what to run in front of the model, and the security question that a surprising number of guides skip over entirely.

Where should the GPU live?

Three reasonable answers, in increasing order of faff.

Bare metal on a dedicated box

Simplest, and there is no shame in it. An old desktop with the card in it, Ubuntu or Debian, Ollama installed, done. No passthrough, no IOMMU groups, nothing to go wrong. The cost is that the machine is now a single-purpose appliance, which in a homelab full of virtualised everything can feel like a waste of a perfectly good host.

A container on the host

If the GPU host is already running Docker, running Ollama in a container with the GPU exposed to it is straightforward and avoids passthrough entirely. It keeps the model runner isolated and easy to update, while the driver stays on the host where it is simplest to manage. For most people this is the right balance.

Passed through to a virtual machine

The most flexible and the most work. The card is handed wholesale to a VM, which then owns it exclusively. This is what you want if the GPU host is also running a dozen other guests and you would rather the AI workload were isolated, snapshottable, and independently rebootable.

GPU passthrough on Proxmox, briefly

The official Proxmox wiki page is the authority here and worth reading properly, but the shape of it is:

• Hardware. You need an IOMMU with interrupt remapping — Intel VT-d or AMD-Vi. Server-grade boards handle this considerably better than consumer ones.

• Firmware. Enable IOMMU (or VT-d) in the BIOS. On AMD platforms it is often on by default.

• Kernel. Add intel_iommu=on for Intel on older kernels, and optionally iommu=pt for performance.

• Modules. Load vfio, vfio_iommu_type1 and vfio_pci via /etc/modules, then run update-initramfs -u -k all and reboot.

• IOMMU groups. The card has to sit in its own group. Related functions and bridges sharing the group is fine; unrelated devices are not. If the grouping is wrong, physically moving the card to a different slot frequently fixes it — this is the single most useful piece of passthrough folklore.

• Binding. Bind the card to vfio-pci by device ID, or blacklist the native driver in /etc/modprobe.d/ so the host never claims it.

One consequence catches people out: once the GPU belongs to the guest, you lose framebuffer output through NoVNC and SPICE for that VM. Plan on a physical monitor for the initial setup, or SSH in and never look at a console again.

External GPUs are a legitimate answer

Worth knowing if your server is a mini PC with no room for a full-length card. For inference specifically, the link between host and GPU barely matters — once the model is loaded into VRAM, only token data crosses it. XDA measured over 70 tokens per second running gpt-oss-20b on an RTX 4080 in a Thunderbolt 5 enclosure attached to a handheld. Compare that with gaming over the same link, which typically costs 5 to 15% of performance.

The link does matter in two places: the initial load of tens of gigabytes into VRAM, and any situation where the model does not fit and is spilling across the link on every token. If it fits, an eGPU over Thunderbolt or USB4 is a perfectly sensible way to attach a big card to a small server — and it can be passed through to a VM the same as any other PCIe device.

The stack in front of the model

Ollama on its own gives you an HTTP API on port 11434. That is genuinely all some people need. Most will want a bit more.

Open WebUI is the standard companion: a self-hosted web interface with multi-user accounts, conversation history, document knowledge bases and model switching. It talks to Ollama natively and to anything else speaking the OpenAI API, and ships Docker images including a combined one with Ollama bundled. Do read the licence before deploying it widely — since version 0.6.6 there is a branding-protection clause, with an exemption for deployments of fifty or fewer users in a rolling thirty-day window. Every home deployment is comfortably inside that; not every small business one is.

Behind that, whatever you already use: a reverse proxy for TLS and a sensible hostname, and your existing remote-access arrangement rather than a port forward.