Getting started with local AI

There is a version of "running AI locally" that amounts to a party trick: a tiny model dribbling out sentences on a laptop, impressive for roughly ninety seconds. That is not what this is about. On a mid-range gaming GPU bought in the last few years you can now run open-weight models that are genuinely useful for summarising, drafting, classifying, answering questions about your own documents and writing a decent amount of code — entirely on hardware you own, with no API key and no per-token bill.

This is the starting point for our Local AI collection. It covers what local AI actually is, the one number that decides what you can run, which software to install first, and — just as importantly — what local models are still bad at.

What "local AI" actually means

In practice it means downloading an open-weight model, which is a file containing the trained parameters of a neural network, and running it through an inference engine on your own CPU or GPU. Nothing leaves the machine. There is no service to sign up to and nothing to rate-limit you.

Two words there are doing a lot of work. Open-weight means the maker has published the trained parameters for anyone to download. It does not automatically mean open source. Some models ship under Apache 2.0 or MIT; others arrive under bespoke licences with conditions attached, and the difference matters the moment you want to use one commercially. Inference means running a finished model to produce output, as opposed to training it — an enormously more expensive activity that you are not going to be doing at home.

Why bother

The honest answer is that it depends what you want out of it.

• Privacy. Nothing you type is sent anywhere. For anything involving client documents, personal notes, internal infrastructure or code you have not published, this is not a small consideration.

• A different cost shape. You pay once for hardware and then in electricity. That is worse than an API for occasional use and better for anything that runs constantly — a summariser chewing through RSS feeds, a classifier tagging documents overnight, a voice assistant that answers a hundred times a day.

• Availability. No outages, no deprecation notices, no model being retired underneath you. A model you have downloaded keeps working exactly as it did the day you downloaded it, indefinitely.

• Control. You choose the system prompt, the sampling parameters, the context length and the model itself, and you can pin all of it. Nothing changes unless you change it.

• Because it is interesting. This is a legitimate reason and most of us are not going to pretend otherwise.

What local AI is not is a drop-in replacement for a frontier model. The gap has narrowed a great deal, but a 27B model running on a single consumer card is not going to match the best hosted models on hard reasoning, obscure factual recall or very long, complicated tasks. Going in expecting a free ChatGPT is the fastest route to disappointment. Going in expecting a fast, private, endlessly available workhorse that is genuinely good at a wide range of ordinary tasks is much closer to reality.

Memory is the number that decides everything

Almost every "can I run this?" question reduces to memory. A model has to be held in memory in its entirety while it runs. If it fits in your GPU's VRAM, it runs fast. If it does not, your software will either spill part of it into system RAM — which works, but slows down dramatically — or refuse to load it at all.

Raw parameter counts make this sound worse than it is, because almost nobody runs models at full precision. Quantisation reduces the numerical precision of each weight, most commonly to about four bits, which shrinks the file by roughly three quarters for a quality loss that is small and, for most everyday work, difficult to notice. The near-universal default is a format called Q4_K_M, and unless you have a specific reason to do otherwise it is the right thing to start with.

The practical rule, and it is a good one: the memory you need is roughly the size of the file you downloaded, plus headroom for the conversation itself. Model catalogues publish those file sizes directly, so you can check before you commit to a download.

What that looks like in practice

• 8 GB of VRAM comfortably runs models in the 3B to 9B range. IBM's Granite 4.2 8B is 5.3 GB at Q4_K_M.

• 12 to 16 GB opens up the class most people should actually be aiming at. Google's Gemma 4 12B is 7.6 GB; OpenAI's gpt-oss-20b is 14 GB and its own model card puts it inside 16 GB of memory.

• 24 GB — an RTX 3090, 4090 or a used equivalent — runs the 27B to 35B class that represents the current sweet spot for local work. Qwen3.8 27B is 18 GB at Q4_K_M.

• Beyond that you are into unified-memory machines or multiple cards, and into models that are interesting rather than practical for most people.

Memory bandwidth is the second number, and it is the one that sets how fast words come out. We cover both in more detail in How much VRAM do you actually need for local AI?.