One of my projects at Trello was looking into and fixing a reliability problem we were having. Even though it seemed solid in testing, we were getting enough support tickets to know that it must be worse than we thought. We collected data on its success rate and found out that it was successful 99.8% of the time. To move forward with doing more work on it, I had to convince our team and management that 99.8% was bad.

At the time, there was a company-wide push at Atlassian to improve reliability, and there was a line-manager assigned to oversee it across Trello teams, so I spoke to him. I showed him the data, but also told him that there’s an overall feeling on our team that it’s affecting our customers more than it seems.

He suggested that I flip the ratio and instead look at the data as Errors Per Million. When you do that, with the same data, you get 2,000 errors per million attempts. This particular thing happened around 2 million times per day, so that was 4,000 errors per day. That partially explained the issue, because it wouldn’t take much for the support tickets to get out of control. Luckily, not all 4,000 were of the same severity, and many were being retried. Still, that number is too high.

The other thing I found after more analysis, was that the errors were not evenly spread across the user base. They tended to cluster among a smaller cohort, which was experiencing a much worse EPM than then rest of the users.

With that in hand, we greenlit a project to address this by targeting the most severe problems. We found several bugs. Eventually we got the success rate to 99.95%.

It’s not clear that 99.95% is four times better than 99.8%, but the equivalent EPM after the fixes is 500 (as opposed to 2,000 before). What surprised me is how different a reaction a high EPM got compared to equivalent numbers. 2,000 EPM is literally the same as 99.8% success and 0.2% failure rates, but the latter two seem fine. Even if I say we get 2 million attempts per day, it’s hard to intuitively understand what that means.

When I said 2,000 EPM, we instinctively felt that we failed 2,000 users. When I said we get 2 million attempts, everyone knows to double EPM to get 4,000 incidents per day, and we felt even worse. That simple change in reporting made all the difference in our perspective.

Getting a pull request reviewed quickly is often under the author’s control. This is great news because, according to DevEx, you should reduce feedback loops to increase developer productivity. There are other feedback loops that a developer experiences, but pull requests happen all of the time. You can have a big impact on productivity if they happen faster, and a big reason they don’t is because of the commits in the PR.

At Trello, during a hackathon, someone did an analysis on all of the PRs on all of the teams to see if they could get some insights. At the time, we probably had about 10 teams of about 7-10 developers each.

One thing they looked at was median time to approve a PR based on team, and there were two teams that were far outliers (with smaller waits for a PR to be approved). They went further to look at the PRs themselves and noticed that they generally had fewer commits and the commits themselves were smaller. The number of pull requests per developer-week was much higher than other teams.

I was on one of those teams, and the other one was very closely aligned with us (meaning we had a lot of shared processes and rituals). When we very small, 5 total developers, we were basically one team with a shared lead. The style of PR on these teams was very intentional. When I was onboarded, I was given very specific instructions on how to make a PR.

The essence of what we did was to completely rewrite the commits before making a pull-request to “tell a story”. I wrote about the details in Construct PRs to Make Reviewing Easy.

As a reviewer, this made it very easy to do approvals, which we could fit it in at almost any time. With all of us doing this, many PRs were approved within an hour and most in a few hours. A really good time to do some reviewing was right after submitting a PR, which made the throughput reach a steady-state. The PR list was rarely very long.

I worked in this style for the 6+ years I was on this team and know that it contributed to a high level of personal work satisfaction. Even though I had to wait for others to approve my work, I felt that my own productivity was largely under my control.

Related: Metrics that Resist Gaming

For almost my entire life (and before that all the way back to the dawn of baseball), the stats on the back of a baseball card were unchanged. If you got the box scores for your favorite player, you could calculate their stats yourself with a pencil. That’s not necessarily good. These stats were simple and misleading.

For example, it was clear in the 90’s that on-base percentage was more important than batting average. This got expanded on in the money ball era. Computers were brought in to analyze players, and so analyzing players was now subject to Moore’s Law, which can be simplified to say that we double computer power every 18 months. We’ve had about 20 doublings since then.

What the Moore’s Law of baseball? The number of stats is doubling every 18 months, all enabled by modern compute power.

There’s a stat called WAR or Wins Above Replacement, which tries to tell you how many wins a player adds to their team relative to the average player at their position (who has a WAR of 0). To calculate WAR for a single player you need every outcome from every player. It’s so complex, that we can’t agree on the right way to do it, so we have a dozen variants on it.

Stats like Exit Velocity, Launch Angle, Spin Rates, Pitch Tunneling, and Framing are only possible to know because of high-speed cameras and advanced vision processing enabled by Moore’s law. We’re not limited to describing what has happened already—some broadcasts put pitch-by-pitch outcome predictions on the screen.

Even with all this advancement, it still sometimes feels like we’re still at the dawn of this era. As a fan, these don’t feel like the right stats either. No one will be put in the hall of fame because they hit the ball hard a lot of times.

Just need a few more doublings, I guess.