QA is Valuable When Releasing is Expensive

In 1992, at my first job, to release the software, someone took a few days to make 100’s of floppies, and then we all stuffed envelopes so we could mail them all over the world. Our customers were banks who only adopted new versions carefully. Releasing was expensive, so releasing with a critical bug was unthinkable. It’s not surprising that the person who did the most QA work was one of the leaders of the company who was a former customer and had deep domain expertise. To help him, we built custom test automation tools.

These days, releasing is so cheap, that it feels like software companies are fine with releasing a bug. Between the time that Apple sent out iPhone 15’s and the time most arrived, they released iOS 17.0.2, which fixed a critical bug transferring data. Apple has a support doc showing how to recover, so we know it happened to customers, but most people were guided by the installer to update their OS, and it was fine, I guess.

That’s an OS release, which used to be hard. Meanwhile, the cost of releasing a web app has been near zero for a while, and it’s a joke in the industry that the customers are the QA team.

If I were looking to do QA in this environment, I’d seek out software that is expensive to release.

Write While True Episode 37: The Passive Voice Was Used

Podcast: Play in new window | Download

Today I want to talk about one of the rules that’s in Strunk and White. It’s actually in almost every book about writing, which is to use the active voice and to prefer it over the passive voice.

Transcript

September 2023 Blog Roundup

This month I published four episodes of my Podcast. We are in the middle of season three, which has been about the basic building blocks of writing: words, sentences and paragraphs.

I rediscovered PlantUML. I had dismissed it because I thought the setup was too complex. Then I realized that I mostly want diagrams in Confluence and that there was a plugin that let me do that easily. There are also lots of online editors, so there’s no reason to run it locally.

I made journals with prompts for guiding morning pages. To do that, I added a lot more features to Page-o-Mat (a tool for creating journals). Then, I used it to create the cover and inner-page PDFs that I used to make the book on Amazon KDP.

I wrote some more posts on software job hunting

Speaking of how little it takes to stand out, I celebrated Post #500 by sharing some of the ways this blog has impacted my career even though it’s not widely read. The more you put out there, the more chance you have of being found, but anything is better than nothing.

It takes very little to stand out

One thing I tell mentees, particularly those in college looking for internships or their first job, is that they have to think about what the job application process looks like from the other side. When an employer opens a job position for an entry-level position, they will be inundated with resumes. The vast majority of them are indistinguishable from each other. They have the same classes, GPAs in the same range, similar job histories. You have to be honest with yourself with whether you’d stand out in that pile.

What you should do depends on the situation, but the general idea is to do more than your peers. The good news is that your peers (when you are entry level) are pretty clueless. It takes some effort, but it’s doable.

Do research, follow up, network, find peers that were successful and get intel on interviewing. Just don’t assume that you are done once you send in your resume.

Prompt Engineering is a Dead End

LLM chatbots are bad at some things. Some of this in intentional. For example, we don’t want chatbots to generate hate speech. But, some things are definitely not intentional, like when they make stuff up. Chatbots also fail at writing non-generic text. It’s amazing that they can write coherent text at all, but they can’t compete with good writers.

To get around some of these limitations, we have invented a field called “prompt engineering”, which use convoluted requests to get the chatbot to do something it doesn’t do well (by design or not). For example, LLM hackers have created DAN prompts that jailbreak the AI out of its own safety net. We have also seen the leaked prompts that the AI companies use to set up the safety net in the first place. Outside of safety features, prompt engineers have also found clever ways of trying to get the LLM to question its own fact assertions to make it less likely that it will hallucinate.

Based on the success of these prompts, it looks like a new field is emerging. We’re starting to see job openings for prompt engineers. YouTube keeps recommending that I watch prompt hacking videos. Despite that, I don’t think that this will actually be a thing.

All of the incentives are there for chatbot makers to just make chatbots better with simple prompts. If we think chatbots are going to approach human-level intelligence, then we’ll need prompt engineers as much as we need them now for humans, which is “not at all.”

Prompt engineering is not only a dead end, it’s a security hole.

Triangle Estimates

I estimate using time, not points. I do this even though I have mostly worked on product development teams inside of companies and not as a contractor for clients.

But, estimation on product development is not as high-stakes an activity as it is if you are billing based on it. The judgement of a successful product development project is based on whether it moves some business needle, not whether it was under budget. The margin on successful product development dwarfs reasonable overruns. There is also a lot of appetite for risk, where we know that we’ll win some and lose some.

That’s not true if the work is for clients. If they pay by the hour, they expect the estimate to be accurate. But there are very few (if any) big software projects that can be accurately estimated. When you’re wrong, you either have to eat it or have a very difficult conversation.

I don’t have a good answer, but I would start with triangle (or three-point) estimates. To use them, you estimate a most-likely estimate and then a best and worst case on either side. The three numbers describe a distribution (see the wikipedia page for how to combine them). The result is an estimate with a error range and a confidence interval. I would recommend sharing that instead of a single number.

The only task management system that I have seen that offers this option is DevStride, which carries three-point estimates throughout its system.

Stopping at a Good Part

When I’m reading non-fiction, I usually progress chapter by chapter, purposefully stopping at the end of each one so that I can have time to process what I have just read. It’s a good time to write a note with my reaction to it. Today, I ended up in a very long and dense chapter and found a different kind of stopping point.

I am reading The Sense of Style by Steven Pinker. After a while, I got to a new section in this long chapter that would have been a good place to stop, but I just kept going. About a page into it, he made a great point that I want to remember. I was excited to keep reading. But I put the book down.

I purposefully turned back a page so I have to read that part again. I know that the passage I just read is very propulsive and will make me want to read whatever follows, so I want it to be at the very beginning of the next session. It’s also keeping an open loop (in a good way) that makes me think about what I just read.

It reminds me about the way I purposefully leave a unit test broken so that I know what to do when I return to the code.

When Someone is Wrong on the Internet

I have a policy never to write a negative reply to an opinion on the Internet. But I still sometimes have negative reactions. At first, I try to let it go. That works a lot, but not always.

If I find myself thinking about it the next day, then I need to do something just to get it out of my head. In Reframing Anxiety, I wrote about how I’ve come to see anxiety as as asset. I see my anxiety as the flip-side to conscientiousness, which I need to be successful. There’s another way anxiety is working for me now.

Part of what’s happening when you read social media and see an opinion you disagree with is that you imagine that you are in a live debate with that person and that you are losing. You imagine that everyone can see this, so (if you are prone to anxiety) your brain will keep it in your head. You think you can solve it with the perfect remark. The problem is that both sides of the argument think this, so it quickly escalates.

What I am doing instead is using that energy to write my own post here that expresses my opinion on the subject. I write it in a positive tone. I don’t refer to the original post. I don’t post it on social media. It’s just here on my site outside of the conversation.

My inability to let it go helps me fulfill my personal commitment to write every day and I’m grateful for that.

Write While True Episode 36: Stack of Paragraphs

Podcast: Play in new window | Download

The advice applies to any kind of writing. It resonated with me because I feel like I might be having that problem in my written work. That sometimes my writing feels like a stack of paragraphs. I am feeling the lack of propulsion that John and Aline described.

Transcript

Generating Podcast Episode Ideas

Tomorrow, I will record and publish episode 36 of Write While True. I have not given a lot of thought about the content yet except that I have the topic.

For each episode, all I want to do is end with a takeaway that I have learned about writing better, It feels like there should be a limitless number of topics, so I’m not worried about running out, but I still need to think of them.

To make it more focused, I have been using “seasons” to set a theme. At some point in the week, something that fits in the theme comes to me. Sometimes it’s from something I’m reading, or maybe another podcast, or it just pops into my head from some past bit of writing advice I saw somewhere.

Sometimes I get an idea that is not on theme. For that, I just make a card on my podcast Trello board. Eventually, there will be enough cards in some other theme that I can use to start a new season.

In a way it’s a lot like James Webb Young’s Technique for Producing Ideas. He recommends exposing yourself to both random things and the problem you are trying to solve. At some point, a new idea will pop into your head, since new ideas are just novel combinations of old ideas.

Then, you refine it, because the idea alone is only a seed, and not good enough on its own.