This is the latest post documenting my tech stack built around using coding agents safely. The previous posts were:
- Part I: Intro and Rationale
- Part II: Linux
- Part III: Python/Django
- Part IV: HTMX
- Part V: VSCode and Copilot
- Part VI: Bulma
- Part VII: Sqlite
- Part VIII: uv
- Part IX: tooling
- Part X: networking
- Part XI: Sandboxing
- Part XII: Sandboxing update (Docker)
Up until this past week, I had been using cloud based AI, but the endgame of this setup was to get onto local AI. In Part II, I mentioned that I moved to Linux on a Framework Desktop. That machine is using an AMD Ryzen AI Max+ with 64GB of unified memory. This is definitely good enough to run some interesting local models, but I do regret not getting a 128GB back in December when I bought it. When prices come down again, I will be upgrading.
For hosting models, I use Ollama. I tried a bunch of models. For each one, I gave it a simple command as a way to “eval” it: “Build and run nebulas tests”. I’ll explain more in the next post when I write about my coding agent (Pi.dev). Of the ones I tried, only qwen3-coder-next could do this without more help.
Here are some random technical notes about the setup:
- For a Framework, everything you need to know is documented well this AMD Playbook about Ollama.
- The main difference between an AMD setup and others is how you allocate memory to the GPU. A Ryzen uses Unified memory (shared between the CPU and CPU), which is similar to how the Apple M series chips work. In the playbook, it explains how to do that. I decided to set the minimum to 512k via the BIOS and then used
amd-ttmto set the shared memory pool to 48GB for now. - With the above setup and the qwen-coder-next model, I get 93% of the model running on the GPU and 7% on the CPU. This is tolerable, but it would be better to be at 100%.
- However, I am getting 36 tokens/sec which is more than fast enough.
In the next post, I’ll explain how I sandbox Pi in the Docker and connect it to Ollama.