You do not need a cloud subscription, an API key, or a supercomputer to chat with an AI model. With a reasonably modern laptop, you can download an open-weight model and run it on your own machine — offline, with your prompts never leaving the room. This guide covers what you need, the three main ways to do it, and where local models fall short.
One honest note: this guide is built from official documentation and reputable setup guides, not from a hands-on test on my own machine. Commands and requirements are as documented; exact speed will vary with your hardware.
With ChatGPT or Claude, your prompt travels to a company's servers and a remote model answers. Locally, the model is a file on your hard drive, executed by a runtime on your CPU or graphics card.
That only works with open-weight models — models whose trained weights are published for anyone to download, such as Meta's Llama, Mistral, Google's Gemma, and Qwen. Closed models like ChatGPT and Claude cannot be downloaded at any price. You will also meet quantization: weights compressed from 16-bit to 4-bit numbers, shrinking a model by roughly three quarters. Rule of thumb: a 4-bit quantized model needs about 0.6 GB per billion parameters plus headroom — so 7 billion parameters fit in roughly 5–6 GB, 13 billion in about 9–10 GB, and 70 billion in around 40 GB.
Sources: PromptQuorum's local LLM guide, llama.cpp explained
Sources: LM Studio system requirements, minimum PC requirements for AI tools
Ollama is the simplest on-ramp: a free, open-source runtime that downloads, manages, and serves models with a handful of commands. Install it from its website, via Homebrew on a Mac (brew install ollama), or on Linux with its install script (curl -fsSL https://ollama.com/install.sh | sh). Then ollama run llama3.2 downloads the model (about 2 GB) and opens a terminal chat. ollama pull mistral:7b fetches without running; ollama list shows what you have. Ollama also runs a local server on port 11434 for other apps. Internet is needed only for downloads; afterwards everything runs offline.
Sources: Ollama made easy (Hugging Face), Getting started with Ollama
LM Studio is a desktop app for macOS, Windows, and Linux that does the same job with buttons instead of a terminal: search, download, load, chat. It picks the right backend automatically — Apple's MLX on Apple Silicon, llama.cpp with CUDA or Vulkan on PCs — and CPU-only inference is supported, though slower on larger models. Its documentation recommends 16 GB of RAM and 4 GB of video memory. Like Ollama, it can serve an OpenAI-compatible local API.
Source: LM Studio local LLM guide
Underneath both tools sits llama.cpp, the free, MIT-licensed C/C++ engine that made consumer-hardware inference practical. It works directly with GGUF model files from Hugging Face: llama-server -m model.gguf --port 8080 gives you a browser chat UI plus a local API. No registry, no hand-holding — just more flags and control. Pick it when you outgrow the friendlier tools.
Source: What is llama.cpp?
Local models are useful for drafting, summarising your own documents, coding help, brainstorming, and anything you would rather not send to a server. They are not miniature frontier models: a 7-billion-parameter model reasons less well, knows less, and hallucinates at least as readily — sometimes more — than the big ones. It has no live web access by default, and it is no doctor, lawyer, or financial adviser. Think of it as a private assistant with a small library, not an oracle: inside that boundary it is handy; outside it, it will confidently invent things.
The real privacy win is simple: with a true local setup, your prompts and documents never leave your machine. But check the fine print: some tools now offer cloud-hosted models alongside local ones, and those do send your data out. Graphical tools may include telemetry; check settings before pasting anything sensitive. And running a model locally says nothing about how it was trained — only about where your conversation happens now.
Local AI exists partly by accident. On February 24, 2023, Meta released its first LLaMA model to researchers only. Nine days later, on March 3, a torrent of the model's weights was posted to 4chan's technology board — and that same day, someone filed a pull request asking for the magnet link to be added to Meta's official repository documentation. Takedowns followed, but the weights had already spread, seeding a wave of open fine-tunes within weeks. Ten days after LLaMA's release, Georgi Gerganov's llama.cpp showed a 7-billion-parameter model running on an ordinary MacBook CPU — the moment laptop AI stopped being theory.
Sources: Wikipedia, Llama (language model), llama.cpp explained
Tonight, in fifteen minutes: install Ollama, run ollama run llama3.2, and ask it to summarise something you wrote recently. Notice that it works with your Wi-Fi off — and where it stumbles. Then, if your RAM allows, pull a second model and compare answers to the same question; the gap between a 3-billion and a 7-billion model is the fastest education in what "model size" buys. Terminal-shy? Try the same experiment in LM Studio. If it feels slow, drop a model size before blaming your machine — picking the right size is most of the skill.
Related on Poptani Labs: the open-weights war behind these models, what AI agents can actually do, and the cloud frontier models they are measured against. More on the Practical Lab category page.