How to Run Ollama on a Budget: A Step-by-Step Guide
The demand for local Large Language Models (LLMs) has skyrocketed, and Ollama has emerged as the go-to tool for running models like Llama 3, Mistral, and Phi-3 with incredible ease. However, as models grow in complexity, the hardware sitting under your desk might not always be up to the task.
Read on to learn how to run Ollama without breaking the bank or overheating your computer. This guide will cover everything from system requirements to deploying on high-performance cloud GPUs for just $0.27/hour.
What is Ollama?
Ollama is an open-source framework designed to let users run LLMs locally. It packages model weights, configurations, and datasets into a unified "Modelfile," making the setup process as simple as running a single command.
Ollama System Requirements
The system requirements for running Ollama vary greatly based on the selected model.
Model parameters
First, we need a simple definition: parameters are the numerical values that the model has learned during its training process.
The number in a model’s name (like 7B, 13B, or 70B) represents how many billions of these parameters the model contains. The more parameters in a model, the more nuanced its responses generally are.
When a model runs, all parameters must be loaded into your VRAM or RAM so the processor can access them instantly.
A rule of thumb for estimating memory requirements for 4-bit quantization (the standard compression used by Ollama): For every billion parameters, you need around 1.2 GB of VRAM/RAM.
Can I use Ollama without a GPU?
The short answer is yes. Ollama supports CPU-only execution. However, be prepared for significant wait times. A CPU might only produce a few words per second, making long-form content generation or complex coding tasks frustratingly slow.
Why VRAM is King for AI
While you can run Ollama on system RAM (CPU mode), it is significantly slower. System RAM typically transfers data at 20–100 GB/s, whereas a VRAM moves data at over 2,000 GB/s.
If your model size exceeds your available VRAM, Ollama will "offload" the remaining layers to your system RAM. This prevents a crash, but you will notice a massive drop in tokens-per-second (the speed at which text appears). To keep things snappy, always aim some VRAM overhead.
The Ollama Hardware Reference Guide (2026)
How to Use Ollama: The Quick Start
If you are running on a local machine (Mac, Linux, or Windows Preview), here is how to use Ollama in three steps:
<ol><li><a href="https://ollama.com/download">Download the installer</a> from the official Ollama website.</li><li>Run the installer and open your terminal.</li><li><strong>Run a Model:</strong> Type the following command: <code>ollama run llama3</code></li></ol>
Where does Ollama store models?
Once you start downloading models, you might notice your disk space shrinking. Where does Ollama store models?
<ul><li><strong>macOS:</strong> <code>~/.ollama/models</code></li><li><strong>Linux:</strong> <code>/usr/share/ollama/.ollama/models</code></li><li><strong>Windows:</strong> <code>C:\Users\<username>\.ollama\models</code></li></ul>
Why Run Ollama in the Cloud?
While local execution is great for small models, professional-grade performance requires serious hardware. If you don't have $30,000 for an NVIDIA H100 or even $12,000 for a dedicated RTX PRO 6000 rig, the cloud is your best friend.
Thunder Compute provides a simple way to run Ollama using pre-configured templates. This allows you to access enterprise-grade GPUs like the NVIDIA A100 or NVIDIA H100 for a fraction of the cost of buying hardware.
Step-by-Step: Running Ollama on Thunder Compute
Thunder Compute simplifies the process by offering a one-click template for Ollama. You don't have to worry about NVIDIA drivers or CUDA versions; it's all ready out of the box.
Step 1: Create a GPU Instance
Create a Thunder Compute Instance with the following configuration:
<ul><li><strong>Instance mode:</strong> Prototyping</li><li><strong>GPU:</strong> NVIDIA RTX A6000*</li><li><strong>Size:</strong> 4vCPUs (32GB RAM)*</li><li><strong>Template:</strong> Ollama</li><li><strong>Disk:</strong> 100GB</li></ul>
*This is just a starting point, consider the model you want to run and adjust this accordingly.
This environment comes pre-loaded with Ubuntu 22.04, NVIDIA Drivers, and the Ollama binary.
Step 2: Connect
Once your instance is live, connect using VSCode or the CLI of your operating system.
Step 3: Start Ollama
Once the connection is set up, run the command start-ollama. This will initialize Open WebUI and provide you with a link to access the platform.
Step 4: Add a model
To make installation fast, this template includes no models.
<ul><li>Head over to the <a href="https://ollama.com/search">Ollama Models</a> page.</li><li>Select one of the models.</li><li>Go back to Open WebUI.</li><li>Click <strong>Select a model</strong>.</li><li>Type the name of your desired model.</li></ul>
This will download the model to the instance and will take some time depending on model size.
Step 5: Start chatting!
Your model is now live. You’re no longer limited by your laptop's cooling fans or your RAM's patience.
Fire off a complex prompt, ask it to write a Python script, or let it brainstorm your next big project.
Conclusion
Learning how to run Ollama is the first step into the world of private AI. While you can use Ollama without a GPU, the experience is vastly improved with the right hardware.
Don't let high hardware costs limit your AI projects. With Thunder Compute's specialized templates, you can get a professional Ollama environment running on the world’s most powerful GPUs.