A Local AI at Home
An AI assistant that runs on my own PC, not in the cloud: a web interface for the whole family and a terminal agent that works on code — on a 12 GB graphics card, without the official support you'd expect.
The idea
The best AI assistants live in the cloud, and every question leaves the house. I wanted to try the alternative: an AI that runs in my own home, on a PC I already have, for two uses. One is a chat-style web interface, for me and the family, with documents to consult. The other is a terminal agent that works on a project — reads files, edits them, compiles, runs tests — in C, C++, Java, .NET, Python and Node, this very website included.
The problem: the “wrong” graphics card
The PC has a Ryzen 9, 128 GB of RAM and a Radeon RX 6700 XT with 12 GB. The RAM is plenty;
the 12 GB of video memory is the real limit. And AMD’s compute software (ROCm) doesn’t
officially support this card on Windows, and under WSL it only works on newer cards. So the
inference engine runs natively on Windows using Vulkan, through llama.cpp, and
everything else lives in WSL.
Choosing models follows the same constraint: MoE (“mixture of experts”) models activate
only a small part of their weights for each word, so the “experts” can sit in RAM and only
the core on the card. A llama.cpp option decides precisely what goes where, which is why I
preferred it over simpler tools.
How it’s built
- Engine:
llama.cpp(Vulkan) driven by llama-swap, which loads and unloads models on demand behind a single port. It starts by itself when the PC boots, even with no login, as a scheduled task. The PC is only switched on when needed. - Models: one with 35 billion parameters (3 active) for everyday use, and a much bigger one, 125 billion (6 active), for hard problems. On the first I now measure about 35 tokens a second writing (it was 16-20 before a change, see below) and over 230 a second reading a long prompt; the second is slower (11-13) but reasons better. A useful finding: raising the prompt-reading batch size gave +50% almost for free.
- Web interface: Open WebUI in a container, connected to the engine. Accounts are created by the admin, open registration is off; newcomers stay “pending” until I approve them.
- Development environment: an Ubuntu distribution on WSL2 with C/C++ compilers, JDK, .NET, Python, Node and a Docker engine, with repositories cloned on the Linux filesystem (much faster than Windows’).
- Network: reachable only from the home network and the second home’s, with firewall rules opening just the ports needed. Away from home, through the VPN. The PC wakes up remotely with Wake-on-LAN.
- HTTPS without opening anything: the web interface has an address with a valid certificate, following the same logic as my password manager: the address exists only in the home DNS and the certificate is obtained through DNS validation (a delegation to a dynamic-DNS service), without exposing the PC to the Internet. An nginx proxy sits in front of Open WebUI, and the interface itself listens only on the PC.
Claude Code or OpenCode? A real test
The interesting question was which agent to use with a local model. Claude Code has the experience I already know, and the engine speaks the Anthropic protocol too, so no translator is even needed. The alternative is OpenCode, an open agent built to work with any model. Instead of trusting reputation, I gave both the same two tasks, with the same model, on isolated copies, with an hour’s limit:
| Task | Claude Code | OpenCode |
|---|---|---|
| A script to find broken links on the site | 24 minutes, works but with one false positive | 5.5 minutes, correct |
| A C++ program with CMake and tests, checked by a script the models can’t see | not finished in an hour | 24 minutes, correct |
The difference wasn’t down to configuration — prompt caching worked in both — but to how the agent works: Claude Code, designed for Claude models, makes more turns and more rewrites, and with a slower local model that costs dearly. OpenCode is the main client. I also use it from the Mac on local projects, and hand it the memory I’d already prepared for Claude (global and per-project instruction files), most of which it already reads on its own.
A caveat about the test. While Claude Code was working on the C++ task, the graphics card was reset twice by the Windows watchdog (see below): that result is therefore at least partly skewed, and isn’t a final verdict. The rest of the picture holds: OpenCode completed both tasks correctly, including afterwards.
A graphics card that kept resetting
Windows checks that the graphics card answers within two seconds, and if it doesn’t, restarts it. With models split between the card and RAM, some Vulkan operations took longer: the card was reset (seven times in one day) and the request in progress failed with a “device lost” error. The fix was to lengthen those timeouts in the Windows registry, and after a reboot it hasn’t reset again.
Double the speed, with a limit
With multi-token prediction (MTP) the model proposes two words at a time and the model checks them: about 85% are accepted, and writing goes from 16 to 33-37 tokens a second. The price is that it handles one request at a time, so the others queue. In the comparison, with a single run, the agent’s variability (how many attempts it makes) weighed more than generation speed: the C++ task with OpenCode succeeded in 36 minutes instead of 24. I chose MTP as the main configuration, keeping the non-MTP version as a fallback. The bottleneck that remains is prompt reading, and there it would take a card with more memory.
Searching documents, and writing them
Open WebUI can answer questions about uploaded documents, and to do that it must “understand” the text: it needs an embedding model. The default one is built for English, so I replaced it with a small multilingual one that runs on the processor next to the chat model without unloading it; search combines meaning with word matching. A catch: this version stores all settings in its database and environment variables only count at first start, so I wrote a script that changes them with the container stopped, after a safety copy. That’s also how I found the interface’s address was still the old one, without HTTPS.
To test document writing I asked both models for the same usage guide for the family. The result is a single guide, converted to PDF and Word with pandoc, where I corrected the places the models had invented details — a useful reminder: they write well, but the facts need checking.
Backups for a PC that sleeps at night
A fixed-time backup makes no sense on a machine that’s off most of the time. Here a timer fires fifteen minutes after boot and then every four hours: it copies the accounts, history, uploaded documents and agent configuration to the NAS (with a safe copy of the SQLite databases, keeping the last 14), and skips the round if the NAS doesn’t answer. If a backup fails, an email arrives — I tested it with a simulated fault. Then I actually verify it: complete contents and an integrity check on the databases. Only the models stay out, since they can be re-downloaded.
Access to handle with care
OpenCode also has a web interface, handy, but it grants access to a shell, files and the GitHub key. So I keep it listening only on the PC itself, with a password, and open it from the Mac through an SSH tunnel: it’s never exposed to the network.
Realistic expectations
These models are fine for contained edits, explanations, documentation, tests and scripts. On long jobs across many files they’re clearly below cloud models: they get tool calls wrong more often and lose the thread after many steps. It’s not a replacement, it’s a complement that costs only electricity.
What I learned
- One run isn’t a verdict: with an agent, timings change more with the path it takes (how many attempts, how many rewrites) than with the model’s speed.
- Check before describing: in the family guide I’d written that everyone signs up on their own, while open registration was switched off. Before explaining how a service works, look at the real configuration.
- Scripts running an agent need stdin closed: launched without an input, the agent waits forever.
- The next step is hardware: prompt reading, the agents’ bottleneck, really only improves with a card that has more video memory. Before buying one I made a sheet of the whole PC, with an inventory script, and found two limits the spec list doesn’t show: the power supply handles a big card on its own but not alongside the old one, and the radiator mounted at the front of the case takes about five or six centimetres off the stated clearance, so the longest cards almost certainly won’t fit. Better to measure first than to return a parcel.
Status
Running since 17 October 2026 (and already tweaked the same day): engine, models, web interface with HTTPS, development environment, two agents (one on the Mac too) and backups with alerts are up, and the comparison is done. The family usage guide exists; automatic updates remain. The configuration is all in a private repository, without the models, which can be re-downloaded.
Stack
llama.cpp (Vulkan) · llama-swap · Open WebUI · OpenCode · Claude Code · WSL2 · Docker · Windows 11 · PowerShell · Bash