Computers
Peripherals
Components
Supplies
Audio Visual
Mobile Tech
Software
Networking
My products
Product guidesLOCAL AI
Ollama: a beginner's guide
Installing it, pulling your first model, understanding what the tags actually mean, and turning it into a service the rest of your network can use.
If you have decided to try running a language model on your own hardware, Ollama is almost certainly where you should start. It is the shortest path from "nothing installed" to "a model answering questions on my machine", and — more importantly for anyone with a homelab — it quietly turns that model into an ordinary network service that everything else can talk to.
This guide covers installation, pulling models, what the tags on the model library actually mean, where the files land, how to expose it to the rest of your network safely, and how to build a customised model of your own. If you have not read Getting started with local AI yet, that sets the context for what follows.
What Ollama actually is
Ollama is a model runner. It fetches quantised open-weight models from a central library, manages them on disk, loads them into GPU or CPU memory, and serves them over a local HTTP API. It is MIT-licensed and runs on macOS 14 and later, Windows, Linux and Docker, with packages in Homebrew, Pacman and Nix as well.
Underneath, it is not one engine but three. It began as a friendly wrapper over llama.cpp, which is still a supported backend. In May 2025 the project shipped its own inference engine, initially to handle multimodal models — giving each model a self-contained implementation rather than patching shared code every time a new vision architecture appeared. And in March 2026 it added MLX on Apple Silicon as a third runner, which is the significant one if you are on a Mac with 32 GB or more of unified memory.
You do not need to care which engine is being used day to day. It is worth knowing because it explains why Ollama sometimes supports a new model before other llama.cpp-based tools do, and why Mac performance improved sharply in 2026.
Installing it
Get it from ollama.com/download. There is a desktop application for macOS and Windows, added in mid-2025, which gives you a chat window, drag-and-drop of text and PDF files, and image input. On Linux the install script is the usual route, and there is an official Docker image if you would rather it lived in a container — which, in a homelab, it probably should.
On the GPU side, Ollama supports NVIDIA via CUDA (compute capability 5.0 and above, driver 550 or newer), AMD via ROCm 7, Apple via Metal, and has a Vulkan fallback on Windows and Linux. If you have several GPUs and want to pin it to one, the usual environment variables apply — CUDA_VISIBLE_DEVICES for NVIDIA, ROCR_VISIBLE_DEVICES for AMD.
Your first model
One command pulls and runs a model:
ollama run gemma4:12b
That downloads the model if it is not already present and drops you into a chat prompt. ollama list shows what you have, ollama rm removes one, and ollama pull downloads without starting a conversation. The full catalogue is at ollama.com/library.
Reading the tags
This is the part that trips people up. A tag like gemma4:12b or granite4.2:8b-q8_0 encodes two separate things.
The parameter count — 8b, 12b, 27b. Roughly, how big the model is.
The quantisation — q4_K_M, q8_0, bf16, and increasingly nvfp4. How much precision has been squeezed out of each weight to make the file smaller.
If you do not specify a quantisation, you get Q4_K_M, which is the sensible default and what almost everyone runs. The library page for each model lists the exact file size of every tag, and that number is the one that matters: the download size is very close to the memory the model will need. Granite 4.2 8B is 5.3 GB at Q4_K_M and 9.3 GB at q8_0. Gemma 4 12B is 7.6 GB at Q4_K_M and 24 GB at bf16. Check the tag list before you pull, not after.
Google also publishes quantisation-aware-trained builds of Gemma 4, tagged qat, which are trained with the eventual quantisation in mind rather than squashed afterwards — and are usually a little smaller than the equivalent Q4_K_M as well.
Where the files live
Models are large and you will accumulate them faster than you expect. By default they land in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and C:\Users\%username%\.ollama\models on Windows. Set the OLLAMA_MODELS environment variable to move that somewhere with more room — on a NAS-backed volume or a dedicated SSD, for instance. Do this before you download twenty gigabytes onto your root partition rather than after.
Talking to it over HTTP
This is where Ollama stops being a chat toy and starts being infrastructure. It listens on port 11434, bound to localhost by default. It exposes its own API at /api/chat and /api/generate, and — far more usefully — an OpenAI-compatible API at http://localhost:11434/v1/.
That compatibility layer is the single most valuable thing about Ollama. Chat completions, completions, models, embeddings and responses endpoints all work, which means any tool built to talk to OpenAI can usually be pointed at your own machine by changing one base URL. Client libraries insist on an API key; Ollama ignores whatever you give it. A few things are not supported — logprobs, tool_choice, and stateful conversations on the responses endpoint — but for the overwhelming majority of software, it simply works.
Exposing it to the network
By default nothing outside the machine can reach it. Setting OLLAMA_HOST=0.0.0.0 changes that, and the official docs cover putting it behind Nginx or a Cloudflare Tunnel.
Do think about what you are doing here. Ollama has no authentication of its own. Anyone who can reach port 11434 can use your GPU, load and delete models, and read anything you send it. On a trusted home VLAN that is fine. On anything wider, put a reverse proxy with authentication in front of it, or reach it over a private network such as Tailscale instead of opening a port.
Watching what the GPU is doing
ollama ps shows what is currently loaded and, in a PROCESSOR column, how the model has been split between GPU and CPU. "100% GPU" is what you want. A split like "48%/52% CPU/GPU" means the model did not fit in VRAM and part of it is running on the processor, which is why it suddenly feels slow. The fix is a smaller model or a heavier quantisation, not more patience.
If several models are loaded and there is not enough memory for all of them, new requests queue until idle models are unloaded. Worth knowing before you conclude something has hung.
Modelfiles: making a model your own
A Modelfile is a short text file that describes a customised model, in a format deliberately reminiscent of a Dockerfile. The instructions are few and mostly self-explanatory:
FROM — the base. An existing Ollama model, a GGUF file, or Safetensors weights. This is the only required line.
SYSTEM — a baked-in system prompt. The most common reason people write a Modelfile at all.
PARAMETER — temperature, context length, top-k, top-p and friends, fixed rather than passed on every call.
TEMPLATE — the full prompt template, in Go template syntax, if you need to change how messages are assembled.
ADAPTER — a LoRA adapter to apply on top of the base.
MESSAGE — seeded conversation history, useful for few-shot behaviour.
Build it with ollama create my-model -f ./Modelfile and run it like any other. To see how an existing model is put together, ollama show --modelfile prints its Modelfile — which is by far the best way to learn the format.
This is the mechanism behind most of the genuinely useful local AI setups we have seen: a general model, a tightly written system prompt, a fixed context length, and a sensible temperature, saved once and reused everywhere.
A note on cloud models
Ollama now also offers cloud-hosted models, pulled with tags ending in -cloud, which run on Ollama's infrastructure while keeping the same local commands and tooling. It is a reasonable convenience if you occasionally need something far larger than your hardware can hold, but it is worth being clear-eyed about it: that is not local inference, and your prompts are leaving the building. Cloud features can be disabled entirely if you want the local-only guarantee, and if privacy is your reason for being here, you probably should.
What to do next
Two obvious directions. Put Open WebUI in front of it for a proper multi-user chat interface with document knowledge bases — it talks to Ollama natively and ships a combined Docker image. Or start pointing real tools at that OpenAI-compatible endpoint and see what having a private model on your own network changes. We work through the second in Running local AI in a homelab.
And if you are still deciding what hardware to put underneath it, start with How much VRAM do you actually need before reaching for the GPU buying guide.