The Windmill Club

The local option: Ollama and the GPU question

We mentioned local as an option earlier, and it deserves a fuller, honest explanation. The most common way to run a model locally is with a tool like Ollama, which makes it surprisingly easy to download and run models on your own machine. No cloud, no subscription, no per-token cost — your model runs entirely on your hardware.

The honest catch is hardware — specifically the graphics card (GPU). Running a capable model locally is demanding:

• ✓A good GPU is the key — the more powerful your graphics card, the larger and smarter the model you can run smoothly. A gaming or workstation GPU with plenty of video memory is ideal.
• ✓Plenty of RAM — a capable local setup wants a healthy amount of system memory alongside the GPU.
• ✓An ordinary laptop can struggle — integrated graphics and modest RAM limit you to smaller models that may not feel as capable as cloud options.

None of this is meant to scare you off — local is a genuinely rewarding path for privacy and control. But it's honest to say that a serious local setup wants a serious machine. If your computer is ordinary, you may be happier with a cloud model, or you may run a smaller local model for everyday tasks and use the cloud for the heavy lifting. Many people do exactly that.

Try it for yourself: the local reality check

All of this talk about models, local, and cloud is useful — but abstract. The best way to make it real is to see it with your own eyes. So before you install anything else, here's an optional hands-on detour: install Ollama, run a couple of models on your actual computer, and let the results teach you what we've been describing. It only takes a few minutes, and it will make everything that follows click.

Step 0: Install Ollama

Ollama is the easiest way to run a model on your own machine. If you don't have it installed yet, we have a full, step-by-step guide — how to install Ollama, download models, and run your first local conversation. It covers the install, finding and comparing models, and common pitfalls, so you can get set up without a hitch.

Once Ollama is running, it automatically detects your hardware — if you have a compatible graphics card, it will use it; if not, it falls back to your CPU. Either way, it's ready to go with no extra configuration. Then come back here and try the three acts below.

Act 1: A heavyweight model, a simple question

Open a terminal and run a large local model, for example:

(If that specific model name has changed, just run ollama list or visit Ollama's model library to find a large one.) Now ask it something simple that requires current, live information:

Watch what happens. A local model has no internet connection and no tools — it only knows what it was trained on, up to some past date. So it cannot possibly know the current time in Belize. It will either guess, make something up, or tell you it doesn't know. It might even feel slow, because a large model strains the hardware on a typical computer.

The honest lesson: a bigger local model is not automatically more useful. It doesn't have access to live information, and on ordinary hardware it can be slow. "Big" and "free forever" sound great in theory — but in practice, a heavyweight local model alone can't do much of what you'll actually want.

Act 2: A lightweight model, a genuinely hard question

Now run a small local model, for example:

Ask it a genuinely difficult reasoning question — something multi-step that requires careful thought. For example:

A small model will often answer this incorrectly — it may confidently say "10 cents" (the classic intuitive-but-wrong answer) without catching its own mistake. Watch for a confident-but-wrong response.

The honest lesson: a lightweight model is fast and free, but it lacks the depth for genuinely hard reasoning. You get what you pay for in capability.

Act 3: The fair middle path — how a framework changes things

So far this might sound like "local is limited, full stop." But that's not the honest whole story — and it wouldn't be fair to stop there.

The limits you just saw come from the model working on its own. But a model rarely works alone. It runs inside a framework — a program that gives it tools, memory, and connections. And that's where something called RAG comes in.

RAG stands for Retrieval-Augmented Generation. In plain terms, instead of relying only on what the model learned during its training (which goes stale), a RAG-enabled system goes and fetches current information before it answers, then grounds its response in that fresh data. So the very limitation you saw in Act 1 — a model that can't know the current time — is one that a framework with RAG can overcome. It reaches out, finds the answer, and builds a better reply.

The honest, encouraging conclusion: for a user with relatively modest needs who cares about privacy and cost, a reasonably capable lightweight local model — running inside a framework like Hermes that gives it retrieval and tools — can be genuinely viable. You don't necessarily need a giant model or the cloud. The framework is what turns a modest model into a useful partner.

So now you've seen, with your own eyes, the honest reality of local models: a heavyweight is limited without live data, a lightweight is limited in depth, and a framework like Hermes is what bridges those gaps. When you're ready, continue on to install Hermes itself and put all of this into practice.