What is Ollama? Complete Guide to Local AI Models (October 2026)

Ollama is a free, open-source (MIT-licensed) tool for running large language models (LLMs) like Llama, DeepSeek, Qwen and Gemma on your own computer. Local use is free and unlimited. Ollama Cloud adds a free tier that includes a small amount of monthly usage for a set of starter models, plus paid plans from $20/month.

Ollama works through a command-line tool, a desktop app or an OpenAI-compatible API. When a model outgrows your machine, you can run Ollama on a dedicated cloud GPU from $0.35/hr.

What Is Ollama?

Ollama is an open-source platform for running and managing LLMs locally or through hosted cloud models. It packages model weights, configuration and a prompt template into a single Modelfile. Pulling and running a model takes one command.

Ollama gives you full control over your models and data, unlike proprietary cloud AI services. The same CLI and API work for local models and for models routed through Ollama's cloud.

Who Makes Ollama?

Ollama Inc. is an independent, venture-backed startup based in Palo Alto, California. Jeffrey Morgan (CEO) and Michael Chiang co-founded the company. The pair previously built Kitematic, an early graphical interface for Docker.

Ollama first launched around Meta's Llama models, and the name appears to nod to them. Today it runs models from Google, Microsoft, Mistral, DeepSeek, Alibaba (Qwen), Moonshot AI (Kimi) and other labs.

Ollama's design reflects the founders' Docker background: you pull and run models much like container images.

Is Ollama Free? Ollama Pricing Explained

Yes, Ollama is free to download and use. Running models on your own hardware is unlimited and needs no account. The software is released under the MIT license, so commercial use is also free. You pay only for Ollama Cloud, which runs models on Ollama's servers instead of your GPU.

Ollama Cloud moved to credit-based pricing on August 31, 2026. Each paid plan includes a monthly pool of usage credits, spent at per-token rates that vary by model. Usage continues at the same rates after the included credits run out. The old 5-hour session and weekly limits no longer apply, although existing subscribers can keep their legacy plans.

Ollama Cloud plans as of October 2026:

Unused Ollama Cloud credits do not roll over. Any plan, including Free, can buy extra credits to unlock all models on a pay-as-you-go basis. Ollama's pricing page lists per-model token rates.

Ollama Cloud or a Dedicated GPU: Which Costs Less?

Ollama Cloud is usually cheaper for occasional use, since you pay per token and manage no hardware. A dedicated GPU can be cheaper for long sessions, batch jobs and custom models. The $20 Ollama Pro subscription equals about 57 hours on a Thunder Compute RTX A6000 (48GB VRAM) at $0.35/hr. It also equals about 18 hours on an NVIDIA A100 80GB at $1.09/hr, with no per-token metering.

A dedicated GPU runs any model and precision that fits in VRAM, including fine-tuned GGUF files (llama.cpp's model format) and full-precision FP16 tags. The trade-off is setup: you size, start and stop a GPU instance, while Ollama Cloud needs only a sign-in.

Ollama vs Cloud AI: How Do They Compare?

Ollama wins on privacy, cost control and offline use. Cloud AI services like ChatGPT and Claude win on setup speed and simplicity.

Choose Ollama when your data can't leave your infrastructure, API bills are climbing or you need to work offline. Choose cloud AI when you need frontier reasoning or built-in multimodal features without setup.

How Ollama Works

Ollama runs as a background service that downloads models, loads them into GPU or system memory and serves them through a local API. It handles model weights, configuration and runtime libraries for you.

Setting Up Ollama

Ollama installs and runs a first model in three steps:

1. Download the installer from the official Ollama website.
2. Run the installer and open your terminal.
3. Run a model: ollama run llama3.2

After setup, type a prompt and the model responds, much like ChatGPT or Claude.

CLI, API and Desktop App

Ollama offers three interfaces to the same local engine and model library: a CLI, a REST API and a desktop app.

The CLI lets you prompt models directly and manage them:

ollama run llama3.1:8b                              # start an interactive chat
ollama run llama3.1:8b "Explain KV cache briefly"   # send a single prompt and exit
ollama pull llama3.1:8b                             # download a model
ollama list                                         # show installed models
ollama ps                                           # show loaded models and the GPU/CPU split
ollama rm llama3.1:8b                               # delete a model

The REST API serves models to other applications at http://localhost:11434. Scripts, IDE coding assistants and chat interfaces all talk to Ollama through the REST API, which also accepts OpenAI-style requests.

The Ollama desktop app adds a chat window with file and image drag-and-drop for . On Linux, Ollama is command-line only.

Ollama's REST API and OpenAI Compatibility

Ollama exposes a local REST API at http://localhost:11434 that mirrors the OpenAI Chat Completions API. Tools built for OpenAI (LangChain, LlamaIndex, CrewAI, Continue) usually work with Ollama by changing only the base URL. Claude Code and other Anthropic-compatible tools connect via ANTHROPIC_BASE_URL=http://localhost:11434.

This API compatibility is one of the main reasons developers adopt Ollama over running llama.cpp directly. The same endpoint serves local and cloud-tagged models.

Ollama Cloud vs Local Models

Ollama Cloud models run on Ollama's servers instead of your GPU, through the same CLI and API as local models. Cloud models carry a :cloud suffix (such as kimi-k2.6:cloud or deepseek-v4-pro:cloud). Run ollama signin once, then call them like any local model. Every cloud request needs an internet connection.

Ollama Cloud suits models too large for your hardware and fast inference without GPU management. Local models never send data off your machine. Ollama Cloud requests go to Ollama's servers, which run primarily in the US and Europe, plus Singapore for a limited set of Qwen models. Ollama says it retains no data and does not log or train on prompts.

The ollama launch Command

ollama launch starts a coding agent in one command, with Ollama as the model backend:

ollama launch claude
ollama launch codex-app
ollama launch opencode
ollama launch openclaw

ollama launch differs from ollama run in scope. ollama run opens a chat with a single model. ollama launch starts an agent such as Claude Code, Codex, OpenCode (an open-source coding agent) or OpenClaw (an open-source personal AI assistant) on top of the model. Both commands work with local and cloud models.

Ollama System Requirements

Ollama's hardware requirements vary significantly based on the model you run and its precision.

Model Parameters

Note

Parameters: numerical values a model learns during training.

A model's name states its parameter count (7B, 13B, 70B). More parameters usually mean more capable responses and higher memory requirements. All parameters must load into VRAM or system RAM when a model runs.

Choosing a Precision: FP16, Q8 or Q4

Most default Ollama tags, like llama3.1:8b, are 4-bit (Q4_K_M) builds sized to fit laptops and 8GB GPUs. This 4-bit default favors convenience over quality.

Q8 is the best balance for most Ollama workloads. 8-bit quantization halves memory use with minimal quality loss compared with full precision. FP16 delivers full quality and fits cases where outputs must match the original model, such as evaluation baselines or fine-tune comparisons.

Q4 cuts memory to roughly a third of FP16, but quality loss grows on smaller models, reasoning-heavy work, and long-context tasks. Studies show the largest quality drops in those cases. Use Q4 only when nothing else fits.

To choose a precision, run a model's full tag instead of the default. Each Ollama model page lists available builds in its Tags tab:

ollama run llama3.1:8b-instruct-q8_0    # 8-bit
ollama run llama3.1:8b-instruct-fp16    # full precision

VRAM Requirements by Model Size

FP16 needs about 2GB of VRAM per 1B parameters, and Q8 needs about 1GB. Add 10-20% for the context window (KV cache). Q4 needs roughly half the Q8 figure.

Why VRAM Is King for AI

System RAM transfers data at 20-70GB/s, whereas VRAM moves data at 350-4,800GB/s. That bandwidth gap makes GPU inference much faster than CPU inference.

When a model exceeds VRAM, Ollama offloads the remaining layers to system RAM. Offloading prevents a crash but sharply cuts tokens/second.

Run ollama ps while a model is loaded to check its placement. A reading of 100% GPU means the whole model fits in VRAM.

Read next

Compare GPUs by workload and budget in the Thunder Compute guide to the best GPU for LLM work.

How Much Disk Space Does Ollama Need?

The Ollama installer is small: about 190 MB on macOS and 1.5GB on Windows. Models take far more space. An 8B model needs about 5GB at Q4, 8.5GB at Q8 and 16GB at FP16. 70B models range from about 43GB at Q4 to over 140GB at FP16.

Available Models on Ollama

Ollama's model library spans lightweight assistants to frontier reasoning models.

Thunder Compute has step-by-step guides for running large open models with Ollama:

• How to run DeepSeek R1 with Ollama: the full 671B reasoning model on cloud GPUs
• How to run Kimi K2 with Ollama: Moonshot AI's 1T-parameter open-weight coding model
• How to run Llama 4 with Ollama: Scout and Maverick, Meta's multimodal MoE family

Ollama vs LM Studio vs llama.cpp vs vLLM

Ollama, LM Studio, llama.cpp and vLLM all run open models, and each fits a different job.

Use Cases and Applications

Ollama supports development, business and education use cases.

Development: Developers use Ollama for code completion, programming assistance, document analysis and research on proprietary datasets. Local models carry no API costs or rate limits.

Business Applications: Regulated industries benefit most from local deployment, because Ollama keeps inference on infrastructure they control. Customer service chatbots, internal knowledge-base search, content generation and legal document analysis all run without exposing data to third-party servers.

Education: Schools and universities can use Ollama to avoid ongoing cloud costs for teaching AI concepts, student projects and academic research.

Why Run Ollama in the Cloud?

Cloud GPUs remove the trade-off between model size and quality that local hardware forces. An 8GB GPU runs an 8B model at Q4, but not a 32B model at Q8 or an 8B model at FP16. Closing that gap with owned hardware is expensive: an NVIDIA H100 runs from around $25K and a dedicated RTX PRO 6000 around $13.25K as of June 2026.

Thunder Compute offers a pre-configured Ollama template on the RTX A6000, NVIDIA A100 and NVIDIA H100, billed per minute with no upfront cost. A 48GB RTX A6000 runs a 32B model at Q8 or a 14B model at FP16. Prompts stay on an instance you control instead of going to a third-party API.

Step-by-Step: Running Ollama on Thunder Compute

Thunder Compute's one-click Ollama template installs NVIDIA drivers and CUDA automatically, with nothing to configure.

Step 1: Create a GPU Instance

Create a Thunder Compute instance with this configuration:

• GPU: NVIDIA RTX A6000*
• Size: 6 vCPUs (48GB RAM)*
• Template: Ollama
• Disk: 100GB

*Adjust GPU and disk based on the model and precision you plan to run.

The Ollama template ships with Ubuntu 22.04, NVIDIA drivers and the Ollama binary.

Step 2: Connect

Connect to the live instance with the Thunder Compute extension for VS Code or Cursor, or with your OS's CLI.

Step 3: Start Ollama

Run start-ollama to launch Open WebUI. The command returns a link to the Open WebUI interface.

Step 4: Add a Model

The Ollama template ships with no models pre-installed.

• Head over to the Ollama Models page.
• Select a model.
• Go back to Open WebUI.
• Click Select a model.
• Type the model name to begin the download.

Download time depends on model size.

Step 5: Start Chatting

The model is now live. Send it a complex prompt, ask it to write a Python script or use it to analyze a dataset.

Last Thoughts on Ollama

Ollama runs capable open models on your own machine for free, with an optional cloud tier for larger models. Start locally for full control over your data and models. When your hardware runs out, launch the Ollama template on Thunder Compute. A 48GB RTX A6000 costs $0.35/hr, billed by the minute.

FAQ