Running a language model on your own computer means no subscription, no data leaving your machine and no dependence on an internet connection. Ollama is one of the simplest ways to do it: a free, open-source tool that downloads, manages and runs open-weight models with a single command, on Windows, macOS and Linux.

Step 1 — Check your hardware

What you can run depends mainly on memory:

  • 8 GB RAM: small models of a few billion parameters, compressed (quantized).
  • 16 GB RAM: comfortable for models around 7–14 billion parameters.
  • 32 GB+ RAM or a GPU with lots of video memory: larger models and faster responses.

Apple Silicon Macs use unified memory efficiently. On PCs, an NVIDIA or AMD GPU speeds things up considerably, but CPU-only works for small models — just more slowly. See our guide to quantization for why compressed models fit in less memory.

Step 2 — Install Ollama

Download the installer from the official website for your operating system and run it. On Linux, the website provides an install script. After installation, Ollama runs in the background.

Step 3 — Run your first model

Open a terminal and run a model from the library, for example a small Llama, Gemma, Qwen or Mistral model:

ollama run llama3.2

The first time, Ollama downloads the model (from a few hundred megabytes to many gigabytes). Then you can chat directly in the terminal. Type /bye to exit.

Useful commands:

ollama list          # models installed
ollama pull qwen2.5  # download without running
ollama rm llama3.2   # delete a model
ollama ps            # models currently loaded

Step 4 — Choose the right model

Browse the model library for options. Rules of thumb:

  • Start with a small, recent general model for chat.
  • Try coding models for programming help.
  • Use vision-capable models to describe images.
  • If responses are too slow, choose a smaller model or a more compressed variant.

Step 5 — Add a friendly interface

The terminal is fine for testing. For a ChatGPT-like experience, install a web interface such as Open WebUI, which connects to Ollama and adds conversation history, file uploads and multiple models. Several desktop apps also support Ollama.

Step 6 — Use the local API

Ollama exposes an HTTP API on your machine (by default on port 11434), including an OpenAI-compatible endpoint. Many tools and libraries can point to it instead of a cloud provider, so you can build private applications or connect coding assistants such as Continue.

What to expect

Local models are excellent for drafting, summarizing, brainstorming, private document questions and learning. On the hardest reasoning and coding tasks, the largest cloud models are still stronger. Many people use both: local for private or routine work, cloud for the toughest problems.

Common mistakes to avoid

  • Downloading the biggest model first. A model that doesn’t fit in memory will be painfully slow or fail. Start small and scale up.
  • Exposing the API to your network unintentionally. Keep the server bound to your own machine unless you add authentication.
  • Ignoring licenses. Each model has its own license; check it before commercial use.
  • Judging local AI by one model. Models differ a lot; try two or three for your tasks.

Troubleshooting

  • Very slow responses: try a smaller model or a more heavily quantized version.
  • Out of memory: close other applications or choose a model that fits your RAM or GPU memory.
  • Model gives poor answers: try a different model family or a larger size if your hardware allows.
  • Can’t connect from another app: check that the local server is running and the port is correct.

Related: quantization explained.

Key takeaways

  • Ollama is a free, open-source tool to download and run open-weight models locally.
  • Memory is the main constraint; 16 GB of RAM handles many popular 7–14B models.
  • A single command such as “ollama run” downloads and starts a model.
  • Web interfaces like Open WebUI add a familiar chat experience.
  • The local API lets your own apps and tools use models privately.

Frequently asked questions

Is Ollama free?

Yes. Ollama is open-source and free to use; the models it runs have their own licenses, most of which allow free use.

Does it work without internet?

After downloading a model, yes. Chatting with local models requires no internet connection.

Is my data private?

Prompts and responses are processed on your computer. As with any software, keep it updated and install models only from trusted sources.