Category Archives: Tech Debt

Invest 10% of a team (not of each dev) to pay back tech debt

If you budget 10% of your team’s time to paying down technical debt, there are a few ways you could do it.

Make sure 10% of the story points of each sprint are technical debt related
Assign every other Friday (10% of 2 weeks) to everyone paying down technical debt (see this article for a story about Tech Debt Friday)
Assign 10% of the team to spend 100% of their time paying down technical debt (rotating who this is every quarter or so, or at project boundaries)

I’ve done some variation on all of these ways, and in my experience, #3 works the best. When I was at Trello, my team allocated more like 30%, but tech debt was lumped together with anything “engineering driven”, which was more than just debt payoff (e.g. tooling).

The main reason #3 works better is how companies typically review and reward developers. Something that is 10% of your work is never going to show up on your review. Over time this is generally a disincentive to do it. But, if you are supposed to spend 100% of your time on something, then it has to show up on your review.

Making this someone’s full time job for a quarter means that they can plan bigger projects with more impact. It’s hard to get a PR done in one day, so it takes about a month to get anything deployed at all. When you work on it full-time, you can deploy much more frequently. When I was on a tech-debt project, I would use the first week to deploy some extra monitoring that could measure impact or catch problems.

It allows devs to get into the zone, which is really helpful in giant refactoring or restructuring/rewrite slogs. If you only get one day every two weeks, you have to reacquaint yourself with anything big, which eats into that one day quickly.

It will also make it more likely that this debt paydown is localized. This makes it easier to test that it hasn’t caused regressions.

Finally, (for managers) it’s easier to measure that you are actually spending your budget correctly because you don’t have to monitor individual stories over time. You just need to track how developers were allocated over time.

Use Your First Commit to Fix CRAP

The CRAP metric combines cyclic complexity and test code coverage to flag functions that are both complex and under tested so that you can see which functions are risky to change.

There are extensions for many IDEs to get you the metric directly or that will show the parts (test coverage and complexity). But you don’t really need them, because you know CRAP-y code when you see it—run unit tests to see if the function is under test and eyeball the complexity by counting up the branches and logical sub-expressions—you can stop counting at about four, because more than that is probably CRAP.

So, if you have to change a CRAP-y function, you could start the PR by trying to lower the score.

The first step to reduce CRAP scores is to add tests. Complex functions are often hard to test, but I would add any tests you can to start, because they help with the next step.

Next, lower complexity by refactoring the function down into simpler parts. The tests you just added will make sure you do it right, but these should be simple mechanical refactors that might even be automatable by your IDE. If they are not trivial, you need to add more tests. Do not restructure or rewrite code unless that’s the goal of the PR—all of your changes should preserve the observable behavior of the code.

I start a lot of PRs this way. It’s a good way to get warmed up, and you know that you are improving the code base in a place that benefits the most from it. You are paying technical debt down right before an interest payment was due.

Dependency Based Tech Debt

One of the things I am doing right now is ripping Bootstrap out of App-o-Mat. The site was mostly built in 2013 and at the time, Bootstrap was good about hiding all the float/clear/positioning nonsense from CSS and giving you a simple grid system. Once I had it, I used everything else it brought along. The site markup is pretty coupled to Bootstrap.

My plan is to build it on just a very small amount of hand-written CSS. Modern CSS has flex and grid layouts built-in, and so re-implementing what Bootstrap offered for layout is not that hard (especially for my limited usage).

When I’m done, my only frontend dependency will be on highlight.js, which syntax colors my code samples.

Ripping out dependencies is a recurring theme for me when a codebase starts to age. I wrote about this in Tech Debt Happens to You.

And I think that this is the main source of tech debt, not intentional debt that you take on or the debt you accumulate from cutting corners due to time constraints. The debt that comes with dependency and environment changes.
Being able to bring code into your project or build on a framework is probably the only thing that makes modern programming possible, but like mortgages, they come with constant interest payments and a looming balloon payment at some point.
There are some you can’t avoid, like the OS, language, and probably database, but as you go down the dependency list, remember to factor in the debt they inevitably bring with them.

Of course, in 2013, the trade-off was probably right for me. I really didn’t want to build anything complex with CSS as it existed. Today, I be hard-pressed to take on any CSS framework dependency (I looked at Tailwind).

It’s not that they aren’t useful—it’s that I don’t have time to keep up with the dependency management problem they will bring with them, and they don’t promise any kind of long-term backwards compatibility (like SQLite does, for example).

Tech Debt Happens to You

In the original ANSI C, there are a bunch of library functions that use internal static variables to keep track of state. For example, strtok is a function that tokenizes strings. You call it once with the string, and then you call it with NULL over and over until you finish reading all of the tokens. To do this, strtok uses static variables to keep track of the string and an iterator into it. In early C usage, this was fine. You had to hope that any 3rd party library calls you made while iterating tokens weren’t also using strtok because there could only be one iterator at a time.

But when threads were introduced to UNIX and C, this broke down fast. Now, your algorithms couldn’t live in background threads if they used strtok. This specific problem was solved with thread-local variables, but the pervasive use of global state inside of C-functions was a constant source of issues when multi-threading became mainstream.

The world was switching from desktop apps to web apps, so now a lot of your code lived in a multi-threaded back-end that serviced simultaneous requests. This was a problem because we took C-libraries out of our desktop apps and made them work in CGI executables or NSAPI/ISAPI web-server extensions (similar to Apache mod_ extensions)

To make this work, we had to use third-party memory allocation libraries because the standard malloc/free/new/delete implementations slowed down as you added more processors (from constant lock contention). Standard reference-counting implementations used normal ++ and -- which aren’t thread-safe, and so we needed to buy a source code implementation of stl that we could alter to use InterlockedIncrement/InterlockedDecrement (which are atomic, lock-free, and thread-safe).

As the world changed around us, we could keep moving forward with these tech-debt payments.

Also, this was slow-paced problem—strtok/malloc/etc were written in the 70s and limped through the 90s. That’s actually not that bad.

But, the world doesn’t stop. Pretty soon, it was just too weird to implement back-ends as ISAPI extensions. So, you pick Java/SOAP because CORBA is just nuts, and well, that’s wrong because REST deprecates that, and then GraphQL deprecates that, and you picked Java, but were you supposed to wait for node/npm? Never mind what’s going on on the front-end as JS and CSS frameworks replace each other every 6 months. Even if you are happy with your choice, are you keeping your dependencies up to date, even through the major revisions that don’t follow Substitutable Versioning?

And I think that this is the main source of tech debt, not intentional debt that you take on or the debt you accumulate from cutting corners due to time constraints. The debt that comes with dependency and environment changes.

Being able to bring code into your project or build on a framework is probably the only thing that makes modern programming possible, but like mortgages, they come with constant interest payments and a looming balloon payment at some point.

There are some you can’t avoid, like the OS, language, and probably database, but as you go down the dependency list, remember to factor in the debt they inevitably bring with them.

Timing Your Tech Debt Payments

It’s impossible to ignore that developers have a visceral reaction against tech debt. Even if they agree that it’s worth it. That’s because they are the ones that need to service the debt.

Tech debt is a cost similar to real-life debt like a mortgage. If you can use tech debt to bring forward revenue and growth, you can pay off the debt later.

But, until then, the interest must be paid.

So, when you are calculating the cost of taking on some debt, a factor in that calculation is how much future work is going to happen on that code. The more work you do, the more interest you pay. If you fix bugs or add features to debt-laden code, you are servicing the debt by making an interest payment. If you refactor, you are paying off principal, and future interest payments are lowered, but that only matters if there are going to be future interest payments.

If you have a system that works and doesn’t need any changes, the fact that it has tech debt doesn’t matter.

To carry the analogy forward, some mortgages have penalties for early payment. Paying off tech debt also has a penalty, usually in QA and system stability.

This is why my favorite time to pay off tech debt is just before a major feature is being added to indebted code. You are trading off the looming interest payments (which will balloon) and your penalty is already being incurred, because you need to QA that whole area again anyway.

In Defense of Tech Debt

I’m a fan of tech debt if used properly. Just like real debt, if you can pull some value forward and then invest that value so that it outgrows the debt considerably, it’s good.

Mortgages and tuition-debt can possibly do this. Credit card consumption debt does not. If your tech debt looks like the former, do it.

For example, if can can close a big enterprise deal with some tech debt, and the alternative is another round of VC to “do it right”, I think it’s obvious to hack away. When you close that deal, your valuation goes up. Maybe you don’t even need to raise.

The decision depends on the specifics. Tech debt isn’t “bad”, it’s a cost. Calculate the cost.

It can be worth it.