Category Archives: Software Business

Pre-define Your Response to the Dashboard

A few days ago, I wrote about using Errors Per Million (EPM) instead of success rate to get better intuition on reliability. I also recently said that Visualizations Should Generate Actions. Sometimes it’s obvious what to do, but if not, you can think through the scenarios and pre-define what actions you would take.

Here’s an example. This is a mock up of what a dashboard showing EPM over time might look like. The blue line is the EPM value on a date:

The three horizontal lines set levels of acceptability. Between Green and Yellow is excellent, between Yellow and Red is acceptable, and above Red is unacceptable. When we did this, we thought about using numbered severity levels (like in the Atlassian incident response playbook), but we decided to use Green/Yellow/Red for simplicity and intuition.

We also pre-defined the response you should have at each level. It was something like this:

LevelResponse
GreenNone
YellowThere must be at least one item in the current sprint with high priority to address this until the level is back to Green. It can be deployed when the current sprint is deployed.
RedAt least one person must be actively working to resolve the issue and doing hot fix deploys until the level is back to Yellow.

The advantage of this was that these actions were all pre-negotiated with management and product managers. This meant that we could just go ahead and fix things (at a certain level) instead of items getting lost in the backlog.

When we created this dashboard, we were in the Red, but we knew that going in. We worked to get ourselves Green and in practice, we were rarely not Green. This is another reason to pre-define your response, as it becomes too hard to remember how to handle situations that rarely happen.

How I Use JIRA and Trello Together

I started using JIRA for issue tracking when I worked at Trello (at Atlassian), and I still use it now. JIRA does everything I need in managing software projects, but I never send people outside of my team to JIRA because it’s not easy for casual users. For that I use Trello.

I have a Trello board for each project I am managing that is meant to be a high-level summary of that project. It is useful for onboarding and getting its current status easily. It has links to JIRA, Confluence (for specifications), Atlas (for status) and Figma.

This Trello board is the first place I send a new team member to help with onboarding. If someone has a question in Slack about the project, I make sure that it was something you could find out on the board and then link them to it there. The board is a kind of dashboard and central hub of the project.

These hub boards are curated, so I don’t try to use any automations to bring things over. If I think you need more information, I send you directly to the source.

JIRA is useful to the people that work on the project every day. I use Trello for those that just check in weekly or monthly.

Visualizations Should Generate Actions

Yesterday, I shared a heatmap visualization that I used to target manual testing time. I chose a heatmap to show this data because you can tell what you need to do just by looking at it.

In this example

A heat map the test status of iOS devices across different features in an app

It’s pretty clear that you should get an iPhone 13 with iOS 15 on it and start testing everything. You could also explore board creation on all devices. If the entire heatmap were green, you would know that you had probably covered most areas.

It would be easy to write a program that took this same data and generated a to-do list instead. Maybe that would be preferable, but people like visual dashboards, and it’s easier to see the why behind the task if you have a sense of the underlying data.

But, that’s a big clue to whether your dashboard visualization works. If you could easily generate a to-do list just by looking at it, then it probably works. If you look at your dashboard and have no response, it might look pretty, but it’s not doing its job.

Use Heatmaps for iOS Beta Test Coverage

At Trello, I built a simple visualization for understanding coverage of our app during Beta periods. We used Mode to analyze data, and so I used their Heatmap.

Here’s a recreation in Google Sheets:

Along the top was each device family and OS. Individual devices were grouped based on how likely they were to be similar in testing (based on size, version, etc). I used this list of Apple device codes (which were logged with analytic data).

Along the left side were the most important screens and features. It was a much longer list that was generated from analytic categories.

The center of the visualization was a heat map based on how much usage that feature got on that device (at the cross-section) normalized against how much usage it got in production. So, if a cell was green, it meant that it was tested a lot when compared to how much it was used in production. If a cell was red, it meant it was under tested.

Often, entire vertical columns would be near red because the combination of device/OS wasn’t used much by our beta testers. So, we could direct our own efforts towards those devices and turn an entire column from red to green.

We also made sure new features would get their own row. These could also be red because beta testers might not know about them. We could similarly target those areas on all devices. These features could not be normalized against production usage (since they were not in production yet), so we use a baseline usage as a default.

Mode kept snapshots of the heatmaps over time. We could watch it go from nearly all red at the beginning of the beta period to more green by the end. I can’t say we could get the entire heatmap to be green, but we could at least make sure we were testing efficiently.

Just Started a New Software Engineering Job? Fix Onboarding

If you are about to start a new job as a software engineer, the way to have a big impact day one is to go through the onboarding with the intention to generate a list of improvements that you work on over time.

Here are some things to look for

  1. Incorrect or outdated information. If you find these, just fix them as you find them.
  2. Missing entry-point documentation. Even teams that have good documentation often do not have a document that is useful if you know nothing. Often they don’t go back and make a good “start here” kind of document.
  3. Manual steps that could be scripted. Don’t go overboard, but if you see some quick wins to automate the dev setup steps, it’s a good first PR. It’s a tech debt payoff that is timed perfectly.
  4. Dev Setup Automation bug fixes. If anything goes wrong while running the scripts to set up your machine, fix the bug or try to add in any detection (or better error messages) that would have helped diagnose the issue.

There is usually a lot of tech-debt in onboarding code and documents because no one really goes through them. Sometimes underlying things just change and tech debt happens to you. You are in a unique position to make it better for the next person and have some impact right away.

WWDC 2023 Wishlist

WWDC23 is next week, so I put together a wishlist. I last did this in 2021, where I broke it down to watchOS, iOS, and developer tools. Whenever I write these wishlists, they are very centered on the work I am doing in the moment and what I need to help me. This year, I am doing less Apple device development, but I use the devices a lot and here are the things I am thinking about.

Headset

There are a lot of rumors that Apple will release an AR/VR headset. It seems like it will cost about $3,000, have an external battery pack, and come with a new framework.

This rumor has been around a while. For the Fall 2021 Apple event (when we really thought a headset would be coming), I wrote:

So, the main thing I’d hope for is something in AR. I’ve written about how I think AR could make apps more like games, and I do think that there’s space for a workout AR device. I would love to extend Sprint-o-Mat to make it feel like you’re in a race against the pace-runner. It would also be a good addition to Fitness+, which could extend to outdoor activities.

So, while I do have development ideas for an AR headset and would love to try one while running, it’s not worth $3,000 for me. If it’s a gaming device, I am not interested.

If the headset could somehow help me in my work (make me a more productive software engineer), then I would be more interested. GitHub Copilot seems to do a good enough job just in VSCode’s interface, but I could imagine being immersed in a VR world with even more heads up information. It would be interesting if there is some kind of meeting space VR, but since I mostly work alone, it would not be worth it to me.

I continue to be worried about headsets that have cameras. I think that it’s inherently creepy out in the real world and dangerous if camera access is extended to apps. I wrote about some ideas for Socially acceptable cameras in AR that I hope are in this headset if they are meant to be worn in public.

New Mac Hardware

If they release new hardware, I am in the market for a new MacBook Air. I love mine, but it’s an M1, so it only has 16GB. I wouldn’t mind expanding on that. I am holding out for a better camera. This seems impossible in the razor thin lid of the MacBook Air. I would be ok with some kind of camera array and a notch, if that’s what it took.

watchOS

My watch needs are driven by my app, Sprint-o-Mat. Aside from the AR features I mentioned above, I am pretty happy with where it is right now and don’t think there’s anything more I need in watchOS for it.

iOS/iPadOS

I hope that Apple adds more safeguards against device theft. One thing they could do is autolock the device if it moves out of connection with the watch. And they obviously need to do something about the fact that the device password gives too much access to iCloud and the Apple ID.

As for a system-wide feature, the biggest thing I miss on iOS is a clipboard manager. Even if they just kept a clipboard history and exposed an API, so that apps could fill the gap, I would be satisfied.

tvOS

I have an Apple TV, HomePods, and my wife and I both have AirPods (all made by Apple). But for some reason, the Apple TV insists on being connected to the TV audio by default. There seems to be no way to get to stay on the HomePods.

Making Sausage and Delivering Sausage

There’s an article about DevEx, a new developer productivity methodology, in ACM Queue. If you subscribe to the Pragmatic Engineer newsletter, there was an interview with the article’s authors last week. This is the latest methodology from the people behind DORA and SPACE.

DORA’s measurements were grounded in externally visible outcomes.

  • Deployment Frequency
  • Mean Time to Recovery
  • Change Failure Rate
  • Lead Time

The idea was to pick things that engineers could actually control.. Even though the elements of DORA are not directly translatable to business outcomes, they are still understandable to external stakeholders.

In SPACE, these metrics are still one kind that we collect, but SPACE also recognizes that there are other things besides Performance and Activity metrics (the P and A of SPACE). It also considers Satisfaction, Communication, and Efficiency, which are more internal to the team.

In DevEx, the emphasis is on internal metrics: Flow, Cognitive Load, and Feedback Loops.

I want to say upfront that I completely agree that these things do help engineers deliver software better and faster. But they are hard to share outside of the team. It’s how the sausage is made. The business ultimately needs to deliver sausage.

Aside from the rest of the business not understanding or caring about these metrics, I also worry that they will try to get too involved in them. Engineering leadership should care a great deal about the cognitive load of the members of their teams, and should work to lower it, but they need to find a better way to express that outside of engineering if they do.

I know the DevEx authors know this, and emphasis on these “making sausage” metrics doesn’t mean that they don’t also think externally visible performance isn’t important (they did after all design DORA and SPACE). But if you deliver on, for example, long flow states, but there isn’t more useful software on servers, you have failed in the business objective. This is the same thing I said about Story Points—they are too far removed from things people outside of engineering care about:

[…] regular people just translate [story points] to some notion of time anyway, and in that regard, I am very regular. If you are going to take some random number I provide and assign a date to it, I’d much rather I do the translation for you.

To the extent that you report directly on DevEx, try to emphasize the parts outsiders can help with. Frequency of meetings and speed of external feedback loops (especially from product management) are good examples of that.

Large Language Models are a Sustaining Innovation

In the Innovator’s Dilemma [amazon affiliate link], Clay Christensen made a distinction between “disruptive” and “sustaining” innovations. Disruptive innovations favor new entrants and new categories of products because incumbents are harmed by adopting the innovation, and so they resist. Sustaining innovations favor incumbents because it improves their products and margins, and they have the resources and incentives to adopt them.

With this in mind, I think that Large Language Models will be readily adopted by incumbents who will be successful with them. To be clear, I’m not talking about OpenAI vs. Google, but their customers, mostly B2B SaaS companies who will be using LLMs to enhance their own software, not providing LLMs to others.

There are two advantages that incumbents have that will hard to overcome.

The first is that LLMs readily embed into established software. GitHub Copilot is the model. The “copilot” experience is being extended to Microsoft’s Office suite, and I think it fits well in almost any kind of software.

The second advantage is access to proprietary data. Incumbents already have customers and their data and can generate better content using that data in their training sets. A new entrant would be stuck with just public sources which is “ok” for some categories, but in the long tail of B2B SaaS companies would be anemic at best.

This is playing out for VSCode right now. Microsoft controls proprietary data (the private code in GitHub) and has the best content creating software. Their first iteration of enhancing that with LLMs is just a better VSCode. I use VSCode every day and adopting GitHub Copilot was seamless. It took seconds to install, see the benefit, and give over my credit card.

The case for a disruptive innovation is easier to make with things like Webflow, which obsolete the editor and establish a new proprietary datasource (their customers’ projects). This might happen to VSCode, but not to Microsoft, since it has its own no-code solutions (the Power Platform). So even this might not be disruptive.

Magnets for Innovation Needles in a Haystack

Each department in a business collects and spreads performance information throughout their organization. They know their own numbers well and are incentivized to improve them. For successful businesses, all of that data is telling them to keep doing what they are doing.

But, new kinds of information aren’t as easy to collect and spread, which is a problem because new information drives innovation.

The beginning of a new market regime, where your products don’t match the market, starts out insignificantly small. You might have a customer satisfaction of 97%, but the innovation might be living in that 3%. Or it might be pointing to an uninteresting niche that you should ignore. It’s hard to know if outliers are noise or a new signal.

If this problem is like finding a needle in a haystack, then we need a metal detector or, even better, a magnet.

In Noticing Opportunities Using an AI Agent I wrote that AI’s ability to consume and synthesize a lot of disparate information can be used to bring opportunities to you. I gave two examples of when I did it consciously and unconsciously in job searches.

For this to work, the organization needs to pour a lot of its internal data into a training set. They should start with sales conversation transcripts and customer support tickets. If there are any user communities, all of their posts should be put in as well. If their competition has public forums, bring those in. Each interaction in the dataset may be an anecdote, but if you have enough of them, at some point you need to accept that there are clusters worth exploring.

Humans still need vet what the AI finds, so I’d try to use this to generate experiments that can validate (or invalidate) a hypothesis.

One thing you’ll want to know is how big the potential new market is. Since these might not be your customers yet, the experiment might need to a content strategy rather than a product one.

It’s fitting then, that the concept of a Lead Magnet already exists. The normal use is to get a lead to start a sales process with. For an innovation, use lead magnets to research and validate a new market.

Sell Your Learning Plan, Not Your Roadmap

The reason software companies adopted agile practices was so that they could adapt their roadmaps to whatever they learned along the way.

If you learn something and do not change the roadmap, then there was no point to learning it. You might as well be doing a waterfall process. I think everyone accepts this. After all, waterfall is a myth. No one really thinks that the roadmap is set in stone.

But the business needs to plan. Marketing has events that need lead time to develop messages and material. Sales has prospects that want to know where the product is headed. Support, Training, and the Documentation team need time to get ready for new releases.

Some of this can be solved by developing support material alongside product development. For example, an embedded Product Marketing Manager can be closely following the trajectory of the roadmap and be developing messages at the same time. Even better, they can be helping to shape the roadmap from insights they get from their customer engagement.

The hard part is what to do with prospects. If you’re Apple, you never talk about what’s coming until you are ready to sell, but most of us can’t do that. Our prospects will certainly have unmet needs, and if you know you are actively working on them, it’s tough not to tell them. But, what should you say?

The truth is you don’t know the roadmap beyond the next few sprints. You are actively hoping that you learn something that will change it. New information is highly correlated with innovation.

But you are not changing your mission as easily. You know which customers you want to serve and what jobs of theirs you want to help with. So, talk about that and not the specific features you currently think will help. And talk about what you do know is coming (if you want). Even in a learning organization, the short-term roadmap should be pretty solid.

Product Development might not be able to commit to exact features, but they should commit to a learning plan, and that’s the roadmap that is safer to talk about.