We are the Dev Teams of
  • brands
  • ebay_main
  • ebay
  • mobile

Payback Time - Dealing With Technical Debt

by Mateusz Szczap
in My Two Cents

Technical Debt Title (Free Images UK)The term "technical debt" means different things to different people, but one thing is for sure: you have to pay your dues sooner or later. But how do you explain that to your product owner, who only cares about making the customer happy and cannot understand why the dev team should spend so much time refactoring, upgrading, code-cleaning, without producing any value to the customer that's directly observable?

So much has been written about technical debt that I may fail to surprise you.

Technical Debt is one of those terms that means something different to different people. For some, it means lack of unit tests or regression tests, for others it means wrong object oriented design, yet for others it means slow performance of the product and using legacy technologies, which are no longer supported and maintained. Quite often production bugs that are hard to fix are also classified as technical debt.

On many occasions, when we developers propose a "repayment of technical debt" to product owners or other business stakeholders, we are asked for  justification, i.e. what customer value will this change bring. In some cases, it is easy to justify it, as a some new technology brings new features or allows to fix certain bugs that were previously swept under the rug.

In other cases, it is not so easy. We developers intuitively feel that a change will improve something, but many times we cannot find a immediate customer value the improvement will bring. My analysis on this brought me to the conclusion that in some cases of technical debt repayment, there is no direct customer benefit - the benefit is indirect and delayed.

I have also observed that people perceive reality in different ways. People that focus on product and customers often cannot see an immediate burning need to replace old legacy technology if it somehow works. They lack an expertise in this area. They do not know how painful it is to work with legacy code, that some bugs cannot be fixed, that DevOps (engineers responsible for server operations) have to spend a considerable amount of time each day just to keep the system running.

Trabant - the most popular car brand in former East Germany (Wikimedia Commons)Imagine that some legacy platforms are like a Trabant – they work fine, but in order for the customer to avoid problems while driving, the people from the DevOps department have to drive behind her with two additional Trabants. When the customer’s "Trabbi" breaks down, they offer her a new one as a replacement. The customers rarely notice a problem, but extra monies have to be spent for the people to drive behind each customer.

This analogy is the best I can come up with to present my idea: The customer is not the only stakeholder.

Discussions about technical debt are difficult for a number of reasons:

  • resolving the debt may prove difficult and time consuming
  • it may be risky on its own, may introduce more bugs and technical debt
  • it may not always bring 100% success – sometimes the usefulness of an improvement can only be assessed after it has been released to production
  • it may take more time and be less important than certain features, because the competition is ahead and users do not benefit directly
  • it may lead to a fragmented platform, because different teams use different technologies, due to of different visions on how to move the platform forward
  • proving the value of an improvement for the customers is often difficult

It is clear, however, that if we do not repay technical debt over time, there is a point at which a platform gets so unstable, so expensive to run or introduce a feature to, that a company may even go bankrupt over it.

This is why some companies embrace an engineering-driven culture, Google being a good example. You may think that I consider the engineering-driven company a perfect company, but that is not the case. There have been some very good engineering companies that did many things well, but at the end of the day forgot about making money, because they were so focussed on doing the engineering right. Needless to say, a balance needs to be found between those two.


So, what stakeholders profit from resolving technical debt?

DevOps as stakeholders  the people at the front line, responsible for running the servers and infrastructure:

"Old and no longer maintained legacy technologies are a particular pain to work with, we spend extra time every day to keep things running, thus we cannot focus on more strategic improvements to the platform, which would indirectly benefit our users or reduce the cost of operating our server farm."

"We have to define NFRs (non functional requirements) for the software to meet the requirements of our infrastructure. These requirements are often de-scoped, because the dev team is focusing on feature delivery. Properly implemented NFRs allow our engineers to monitor the applications not only on the technical level, but also on the business level, e.g. the number of orders a day."

"Our customers will not appreciate it if our software sometimes works, sometimes doesn’t, even if it works 90% of the time."

"A new technology would allow us to run fewer servers, which saves operating cost."

Developers as stakeholders:

"We sometimes notice a 'code smell' and we would like to improve the product. It is hard for us to quantify the direct customer value, we can only say that when performing such an improvement, we have a good gut feeling that it will speed up our development efforts later on."

"If I improve my daily development process, I have more time to deliver quality products with more features."

"Bad code can sometimes spread like a virus: if a bad solution to a common problem is implemented, when a new feature is requested, further new bad solutions are introduced, based on the bad fundaments."

"Some improvements we can do immediately, because they are easy. Others, quite often critical, because based on bad fundaments, require us to constantly justify the effort to the business units and prove the value of them."

"Modern technologies may enable customer-visible features later that are not easy to quantify and see immediately."

Talent acquisition as stakeholders:

"In addition to factors like a nice working atmosphere and a good salary, to hire most talented and ambitious engineers, we need to convince them that it is worth their while to work for us in regard to technology. It is difficult to 'sell them' a dated technology stack that other companies are moving away from."

Technical Debt Improvement Requires Confidence in the Business Domain and Technical Domain

This is a very important thing that is often missed. Technical debt repayment requires people to understand the platform. Being fairly new to the company, I personally do not feel comfortable to improve some areas of our platform because I lack in-depth knowledge. Technical debt repayment projects require confidence and business domain knowledge. Developers who have this confidence are hard to replace, e.g. by outside contractors.

Bigger Technical Debt Repayment Should Be a Project

Small refactorings should be done as part of the day-to-day work on the code base, but more difficult technical improvements should be full-scale projects with technical product owners, such as tech leads or architects.

Technology-driven projects should be proper projects with the same priority as those driven by customer value, and executed in the same way.

My Proposal

Since it is often hard to prove an immediate customer value, as I clearly showed above, I propose a contract of trust between product owners and developers, so that technology decisions can be left to the developers.

Rather than constantly require developers and DevOps to justify their projects, let us split the development streams, so that some time is always reserved for technical product owners to improve the platform as a whole.

There are various ways to solve this. The team could agree to spend half of its time on resolving technical debt and the other half on building new features. Another way would be setting up a dedicated team to improve the platform's technology.

The important point here is that technical debt repayment cannot be only analyzed in regard to immediate customer value and doesn't need to be constantly proven by technical departments. I propose a culture of mutual trust: The technology team trusts product management to make the best decisions in regard to customer value and revenue growth – product management trusts the developers to make the best technical decisions to improve the platform as whole.

Please note: While I am an employee of eBay Inc, the opinions stated in this article are my own.

Technical debt title graphics: Free Images UK
Trabant photograph: ignis, Wikimedia Commons
Trabant logo: Flominator, Wikimedia Commons

technical debt, devops, opinion, programming