Dev Stack, Part XIII: Local AI

This is the latest post documenting my tech stack built around using coding agents safely. The previous posts were:

Up until this past week, I had been using cloud based AI, but the endgame of this setup was to get onto local AI. In Part II, I mentioned that I moved to Linux on a Framework Desktop. That machine is using an AMD Ryzen AI Max+ with 64GB of unified memory. This is definitely good enough to run some interesting local models, but I do regret not getting a 128GB back in December when I bought it. When prices come down again, I will be upgrading.

For hosting models, I use Ollama. I tried a bunch of models. For each one, I gave it a simple command as a way to “eval” it: “Build and run nebulas tests”. I’ll explain more in the next post when I write about my coding agent (Pi.dev). Of the ones I tried, only qwen3-coder-next could do this without more help.

Here are some random technical notes about the setup:

For a Framework, everything you need to know is documented well this AMD Playbook about Ollama.
The main difference between an AMD setup and others is how you allocate memory to the GPU. A Ryzen uses Unified memory (shared between the CPU and CPU), which is similar to how the Apple M series chips work. In the playbook, it explains how to do that. I decided to set the minimum to 512k via the BIOS and then used amd-ttm to set the shared memory pool to 48GB for now.
With the above setup and the qwen-coder-next model, I get 93% of the model running on the GPU and 7% on the CPU. This is tolerable, but it would be better to be at 100%.
However, I am getting 36 tokens/sec which is more than fast enough.

In the next post, I’ll explain how I sandbox Pi in the Docker and connect it to Ollama.

Dev Stack, Part XII: Sandboxing Update (Docker)

Late last year, I completely changed my dev stack to Python on Linux with some other things. I wrote a series about it at the time:

In Part XI, I described using KVM (a full virtualized VM) and specifically not Docker. At the time, I wrote:

Docker: I am not that comfortable with it, and it seems like running GUIs (like VSCode) is not trivial. It would be more efficient with sharing installed software, but wasting disk space is just not an issue.

After using a VM for several months, I have decided to bite the bullet and learn Docker more and get comfortable with it. I’m not trying to run VSCode in it, just terminal based coding agents (which I will describe in a future post).

My setup is simple. I started with a Dockerfile based on node because I wanted to use Pi.dev and that is installed via npm. I gave it the minimal tools needed to build my project (a game built with C and Raylib), which is just gcc, X11, OpenGL, and a few other things.

To harden the security, I created a docker private network and put the Docker into that network. From there it has no internet or network access at all. I opened up one port so that the Docker could talk to Ollama on my host machine.

In the next post of this series, I’ll elaborate on the entire local AI and Pi.dev setup.

An Infinite Game is Like a Bad Habit

If an activity is an infinite game, then you don’t need motivation to do it. It’s more likely you need motivation to not do it too much. Like smoking.

If you play infinite games that are good for you, then that’s fine. I did my first 5k in 2005 and first marathon in 2009. I was addicted to running and got a lot of health benefits. Better than that, I ate as much as I wanted. I loved running, and I thought I would do it forever.

But all of that running did have downsides, and I had to give it up. It was hard. Last year, I was sidelined for six months because I injured a knee trying to re-incorporate running into my routine. That was enough to make me quit (I hope). I still think about doing it, though.

Bad habits are hard to break because you get an immediate benefit. The bad part usually comes in the long-term. When designing an infinite game, we need to keep the immediate benefit part. It’s best if that benefit is obvious when you are playing the game. Or at worst, you see improvement the next time you play.

This is easy at the beginning, which helps you get hooked. Couch to 5K is a good example. But as you improve, it gets harder for a small gain to seem significant. This happens for bad habits too, which is how you can eventually end up with a two-pack a day smoking habit.

Right now, I am trying to figure out an infinite game to learn Spanish. I really think that there’s no downside to do this, but I am really struggling to find a fun game with immediate benefits. My improvements are imperceptible to me. Maybe I just need to play longer each day for now to get more of a fix.

Identities Under Failure

Associating goals with a personal identity works for me. It becomes a North Star. Something I can always strive for and a source of motivation. Thinking of myself as a programmer causes me to program even when there is no reason to. I believe an identity is what you do. So, a runner runs, a writer writes, and I did the things I had to do to become those things.

If an identity is leverage on the way to success, then, like other forms of leverage, it can compound failure as well. Warren Buffet clarified Charlie Munger’s quote on leverage this way:

“My partner Charlie says there is only three ways a smart person can go broke: liquor, ladies and leverage,” Buffett said during a 2018 CNBC interview. Then came the twist: “Now the truth is — the first two he just added because they started with L — it’s leverage.”

Leverage is just a way of multiplying a force, so this makes sense. Identities can compound failure in multiple ways.

You want an identity that others decide. This one is common, and even though I know this is bad, I am not immune. I know that I would like to have achievements that come with recognition, and I know that I might not reach that. At some point, when I accept that, it might make me quit.
You want an identity that harms you to achieve. For me that’s being a runner. My knees and ankles can’t handle it any more and last year I was sidelined for months because I was too stubborn to give it up. I have since picked something more sensible to strive for: being “Strong”. I think I can ride that out for a couple of decades.
You want an identity, but not only if it’s easy. I worry that this is what is holding me back from working at understanding Spanish. I hope that my demotivation is that my plan is bad and I need to work on finding a better one.
You want an identity for money. I can’t say money is not in the mix for me, but I see people fail or give up doing things that they only wanted because of the money. I can’t deny that it does also seem to work for some. I don’t think it’s a bad way to get started, but most high paying activities only pay off to the skilled and that often requires work that isn’t always paid (or worse for this problem, costs money).
You want an identity that you don’t really want. For me that was “home owner”, which I thankfully don’t care about any more. This happens a lot to kids who internalize their parents’ wishes for them.

The main way these work against you is they turn setbacks into quitting. For me, the main way I check this for myself is by really asking myself if I want the identity for external reasons (like recognition or money) or if I want it because of the change it will make in me and for the fun of working at it. For self-esteem. Something I experience mostly in my own mind. Can I reframe what I call it in that way?

In some cases, quitting might be good. Better to fail fast at something you don’t want than to toil away at it for years. Even if you get it.

An Identity is What You Do

I think of Identities as Forever Goals that I might never fully achieve or could always “be more of”. So, while I am definitely a “Programmer” and am proud of that ability and my accomplishments, I can see that there is a lot I could still learn and achieve. My other aspirational identities like “Writer” or “Spanish speaker” might never be realized. I’m ok with that as long as keeping trying.

To me, my identity is what I do, not what I am. If I keep writing, even on this blog, even if no one reads it, then I am doing what a writer does. That’s enough.

I have to keep reminding myself because I am struggling with Spanish, and it’s mostly because I don’t do the work. It won’t matter how much I want it or think about it or hope I get better. The truth is that I haven’t fully embraced the work I need to do. Yet.

The Inner Game of Spanish

In my post reviewing The Inner Game of Tennis, I wrote that I thought the book applied to any physical activity. I also think it could apply to language acquisition, because we speak and listen with our automatic mind. You don’t want a judging mind to be evaluating how well you are pronouncing words and understanding. But if it’s not judging, then what?

The trick described in The Inner Game is to give your judging mind something to do. In tennis, Gallwey recommends having your thinking, judging self track the ball’s spin or think “bounce”, “hit” when appropriate. You could have it just track the position of your arm right before you hit. The key is that it is something objective.

I want to use this technique to get better at Spanish comprehension. One way I do that is to watch Dreaming Spanish videos to get comprehensible input (native language without translation, just above my level). It’s hard. I can feel my mind wander as it judges that I am not doing well enough.

Here are some things I am going to try out to give my thinking mind something to do:

Observe the mouth of the speaker: I am hoping I’ll see some connections to the sounds they are making. This is easier with a video than in real life, of course.
Sketchnote the content of the video: I am specifically trying to avoid translation. I want the Spanish words to evoke pictures in my mind.
Transcribe: Not every word, but phrases here and there. I will probably combine this with sketchnoting.

The goal is to get my mind to stop doing meta-thinking about how well I am understanding, and if that’s good enough. I want it to provide neutral, objective information to my automatic brain. I’ll share more after I’ve been trying it a while.

Review: The Inner Game of Tennis

I just finished The Inner Game of Tennis [ad] by Timothy Gallwey, which has been on my reading list for over a decade. I happened to see a copy featured in a bookstore and bought it as a plane read. It’s short and sweet at under 200 pages, which I love.

I should say to start that it’s not about tennis. Well, it is, but everything in this book is applicable to every sport and possibly every physical activity. I’m not sure if it applies to mental activities yet, but it might. I definitely think it applies to language learning. The thesis is that your inner self judges you while you try to do something and causes you to do it poorly. The “Inner Game” is a set of techniques to quiet this voice (he calls it Self 1) so that the body (Self 2) can play and get into a flow state.

I think it works for language learning because you should just be engaged in the conversation or content, not judging your accent or fretting over whether you got every word. It’s possibly a more physical than mental activity. Gallwey’s methods, which are based on observing experts and yourself is similar to the comprehensible input method of language acquisition (which is how we all learned our native language, so we know it works).

This book reminded me of Thinking, Fast and Slow [ad] by Daniel Kahneman, which also separates the self into System 1 and System 2 thinking, but with the line drawn in a slightly different place. To start with, he reverses the numbers, making System 1 the automatic system and System 2 the more conscious, logical, methodical system. Even so, they are not exactly the same, but close enough.

Gallwey’s book is about the things that the automatic system does well (sports) because it’s fast. Kahneman’s book is more about suppressing the automatic, heuristic-using System 1 when you should be doing the work to understand and analyze using System 2. His worry is that the fast system will give you a plausible (but incorrect) answer, and then you won’t do the work to find the right one. Gallwey thinks your judging self will slow down the automatic response you need at the speed of sports. Or, in my case, language acquisition.

I am sure that this book is the most important one I read this year (maybe in multiple years). I suspect that it will have the same kind of lasting impact on my thoughts and behavior as 4DX [ad] and Thinking in Systems [ad] have.

A Game’s Fun Comes from Its Rules

Baseball’s infield fly rule seems strange at first. It says that when there are runners on first and second with less than two out, and the ball is popped up to the infield, then it’s an automatic out, even if the fielder drops it. Without the infield fly rule, the fielder might let the ball drop on purpose. Then they could turn an easy double-play at second and third base. A pop-up would turn into two outs in a way that isn’t fun for the players or fans because it takes no skill. The rule is there to make the game more fun.

Professional sports are filled with rules like this. Basketball has goal tending, soccer and hockey have offsides, and in football, you can’t catch a pass if you run out of bounds. Rules like this often come after their absence was exploited. For example, goaltending was introduced in 1944 to stop a 6’ 10” center from making the game less fun for everyone (else).

So, when designing an infinite game, you might need seemingly arbitrary rules. Probably it’s better to notice when the game is too easy and figure out a way to make it harder. Based on the examples above, you should also look for times when you play the game in an odd way that gets you rewards with no skill or work. It might seem fun at first, but easy games get boring, and then you’ll quit.

Forever Goals Aren’t SMART

In the last post, I wrote about Forever Goals, which are outcomes I will always want. It’s something I can approach or maintain, not achieve. They are not SMART goals (i.e. Specific, Measurable, Achievable, Relevant, and Time-Bound). For example, a SMART goal is phrased like this: “losing 5 pounds by the end of the year by doing 4 hours of cardio per week and tracking calories.” A Forever Goal is more like “be fit.” A Forever Goal could have a logo. It should make a good tattoo.

I think a SMART goal is more likely to succeed, especially in the short-term. But I fail a lot of personal SMART goals. For some of my aspirations, it hasn’t caused me to give up. I think it’s because I know I don’t really care about the SMART version of the goal. I care about the Forever version. If done right, a Forever Goal can succeed … forever.

Let’s compare the two.

In a SMART goal, the Relevancy is how much you want the goal. Relevancy is baked into the idea of a Forever Goals, so no difference there. Forever goals are also Measurable in the sense that there are measures that I want to increase or maintain, like my Personal Record (PR) lift or mile time. The difference is that the goal is the direction, not necessarily a number. A number can be achieved, but a direction is forever.

The similarities end there because Forever Goals aren’t Time-bound or Achievable and not particularly Specific. If you stop there, this is a good enough reason for them to fail. You could certainly support a Forever Goal with milestones that are SMART, which I have done. But now I prefer designing an infinite game instead.

The issue with SMART milestones is that failing them is de-motivating. If I do the cardio and track my calories, but don’t lose 5 pounds, I might be discouraged. This is true even if you become more fit. Personally, I fail almost all of my fitness milestones. But, I do still work out regularly. I do generally eat healthy. The measures of my fitness are mostly trending positive. I think the reason is that I have found a way to make doing the right things fun. Working out, cooking, and even grocery shopping are games to me.

I have also found programming to be incredibly fun and do it for enjoyment. I never made any kind of SMART milestone to get better at programming, but have no problem doing the work that makes me better at it.

SMART-like goals were imposed on me by employers, so I guess they were in the mix. If the only reason to program was to meet the goals of my employers, I think I would have given up a while ago (or not have been as successful or happy).

This is on my mind as I struggle to figure out a way to make learning Spanish more fun. I do the work (mostly), but it’s subject to wild swings in motivation. I have been relying on habits and discipline to make progress, which I’ll discuss next.

Forever Goals

I like infinite games. These are things you do over and over because they are fun and you want to get better at them. You want to get better so that you can make the game more fun and want to play it more. The game is the point. But I often do have a secondary reason to play them: to make progress on a forever goal.

My podcast, Write While True, is one example. Its name is even a nod to infinite games because it’s an infinite loop. The goal is not to get listeners or make money. The podcast’s forever goal is to make me into a better speaker. Now that I have a co-host, it’s also to make me a better listener and conversation partner.

This blog is another example. I mostly write just to write. But it has the secondary effect of making it more likely that I could express these opinions more eloquently later. The forever goal is something like: to organize my thoughts. This blog has also led to serendipitous opportunities, and I would always want that.

I would also say that cooking is an infinite game for me. I enjoy the activity of cooking, and I do want to do it more to get better at it. But, of course, I have a forever goal of eating regularly and being healthy, which this game supports. Similarly, I go to Crossfit, which has lots of infinite games embedded in it and supports my forever goal of becoming fit.

I also have forever goals that I haven’t figured out a supporting infinite game. One that I struggle with right now is being fluent in Spanish. I started last year, and while I have gotten better, it’s a struggle. The studying I do is not a fun game.

To be clear, I am not looking for something like DuoLingo, which is gamified, but not a game. I use it, and it’s fine, but I wish I could turn off streaks, trophies, badges, leader boards, gems, and treasure chests. These features don’t make the app more fun. They are slapped on. This is also true of Fluent Forever, but not as much. Parts of that app are genuinely fun.

In the next post, I’m going to think out loud about this problem a little more.