How to put technical debt on your roadmap

Over the years, Honeycomb has grown, steadily attracted more customers, and faced the challenge of scaling. As the company has weathered these growing pains, it has overcome new hurdles, learned, and adapted. This growth process has had a variety of side effects, each pushing the company's systems to their limits in unique and unpredictable ways. Despite these challenges, the organization's infrastructure remains robust, albeit showing signs of strain as it encounters new boundaries.

In my presentation at QCon San Francisco 2023, I detailed how I overcame the challenge of doubling my business.

Initially, it was essential to determine how to create a compelling business case so that the technology project could be prioritized within the schedule.

understand your priorities

There will always be a surplus of work compared to the ability to perform. Engineers focused on product roadmap items (new features and enhancements) are often faced with the challenge of how to prioritize the work needed to maintain existing operations. there is.

Scheduled work by product managers may not match the immediate needs for maintaining the infrastructure. From a product manager's perspective, prioritizing technical needs over roadmap tasks may seem to call into question the engineering team's focus. For example, in a startup environment, cost-cutting measures such as his 20% reduction in hosting costs may be ignored in favor of projects that are considered to provide more value to end-users and potential customers.

Prioritization process

Sophisticated tools help product managers integrate numerous ideas and inputs to create proposals for engineering evaluation. From customer feedback to ongoing ideas and recognition for a particular business, this information must be integrated to represent the services, solutions, or problems your business addresses. By reframing each feedback and idea as a problem for the business to solve, a comprehensive dataset is formed. This highlights the different challenges your business faces, how those challenges impact your customers, and opportunities for growth and improvement, all of which are curated by your product organization.

After understanding these ideas, we can categorize them into three groups. Some have clear benefits, some have little benefit, and some are vaguely in between. During a workshop led by Jeff Patton, I was introduced to a graph called the Constable truth curve that he presented at QCon San Francisco in 2013.

For the majority of ideas that fall into the blue middle area, tools should be leveraged with the goal of clarifying ideas and leading to more concrete decisions. Effective frameworks, such as the Opportunity Canvas and the Learning Canvas, to help structure understanding of the problem, identify who benefits and who doesn't, assess costs, and foster discussion around these aspects. there is.

The main objective here is to assess the impact on users, how important they value the change, and whether frequently used features will be enhanced to provide a clear return on investment for the project. It is to do. This process is central to product management and focuses on thoroughly answering these important questions.

Software costs are higher

Addressing technical debt is costly and emphasizes the need to consider engineering efforts beyond the level of individual contributors. Companies often use revenue per employee metrics as a measure of the value each employee contributes to the company's success. According to some simple math, engineers, who typically make up about 30 to 35 percent of a company's workforce, can each be expected to generate about $1 million in revenue for their efforts. For the best-performing companies, this number can jump from $2 million per engineer to $5 million per engineer.

It's important to recognize that product organizations spend a lot of effort cherry-picking features and deciding which ones to prioritize. To consider technical debt, such as a database upgrade, you should evaluate its business value and potential benefits using the same criteria applied to other features.

A specific approach involves working and prioritizing based on building a solid business case. This involves balancing the benefits of taking on the work with the costs of performing it. These decisions must begin through a framework of the business's current priorities and overarching goals to ensure that our efforts are always aligned with what's most important to the business.

What is technical debt?

Identifying technical debt is complex. This includes not only customer-facing features like new features and bug fixes, but also behind-the-scenes work like toolchains, testing, and compliance that only becomes apparent when issues arise. Additionally, operational aspects such as CI/CD processes, training, and incident response are important non-code components of systems management.

Emily Nakashima, VP of Engineering at Honeycomb, published the following graph in Anything But Tech Debt.

While technical debt is often thought of narrowly as code refactoring and dependency upgrades, it actually encompasses a wide range of tasks needed to adapt a product to new business requirements. The term “technical debt” itself has many different interpretations, which can prevent clear communication. A more effective approach might be to discuss the work in terms of business impact, thereby facilitating integration into the product roadmap.

Change language

The process of determining how to prioritize and schedule engineering tasks ranges from the specific work that needs to be done, such as a database upgrade, to understanding why these tasks are essential and how to implement them. It's important to shift your focus to linking it to your business goals. For example, an engineer might insist on upgrading a database because it is nearing the end of its lifespan. However, the urgency becomes clear when you consider the business implications, such as hosting providers refusing to support older databases, threatening product continuity.

Emphasizing the direct impact on business operations, such as potential downtime, provides a compelling case for prioritizing such upgrades.

As an engineering team, we have the unique ability to generate and utilize data from our systems, and even modify these systems to glean new insights. This feature allows you to substantiate your point of view in discussions and move from speculative arguments to data-based conclusions.

Service level objectives (SLOs) are a recommendation for connecting technical metrics with business value, primarily because they encapsulate user experience metrics and provide a tangible way to demonstrate the impact of technical decisions on business outcomes. It stands out as a tool. InfoQ also has a talk by Liz Fong-Jones about the pitfalls of measuring SLOs. Past incidents also provide useful indicators.

Engineers also have access to metrics that reveal the capabilities of the service, allowing them to conduct chaotic experiments. By intentionally limiting services, you can simulate stress conditions and uncover potential problems before they escalate into actual incidents.

Additionally, gathering qualitative feedback through surveys about engineering experience, especially regarding technical debt, can provide valuable insights. For example, areas of the codebase that are perceived as “ghost graveyards” due to their complexity can hinder progress. Addressing these areas will not only wipe out technical debt but also drive business progress.

Let me explain the points I've made so far by comparing them with my experience from a year and a half ago. Since the organization is a Software as a Service (Saas) company, Annual Recurring Revenue (ARR) serves as a core business metric. Understanding the importance of her ARR around customer acquisition, upgrades, and churn allowed her to prioritize actions that were not directly related to technical tasks, such as database upgrades. By working with our sales and product teams, we were able to gain detailed insight into customer behavior and requirements. By translating this information into tangible technical metrics, we were able to directly link engineering efforts to business value and guide infrastructure planning and optimization.

We asked each engineering team to start with a comprehensive overview and assess the service by identifying potential bottlenecks and scalability. They evaluated how their services related to our sales goals, considering relevant factors such as uptake rates. This comprehensive process, supported by our SRE team, resulted in standardized reports detailing service features, dependencies, and scalability challenges.

For example, our core API service, which is critical to ingesting customer telemetry, underwent analysis to identify its scaling limitations and dependencies. By comparing these findings with our sales forecasts, we determined that it was urgent to address certain scalability issues, such as database load issues, in order to meet our growth goals. This approach not only identified our immediate technical priorities, but also influenced our strategy for future scalability testing and tuning.

Connection with business case

The legitimacy of this project quickly became clear. I was able to convincingly communicate to the sales team that they needed to address database load issues in order to meet their sales goals. This made it easy to approve schedules for the necessary work.

This systematic approach was applied across all services, allowing an organized Asana committee to categorize tasks based on urgency and relevance. This exercise also revealed certain engineering challenges, particularly one called “.explode” highlights important issues regarding scalability. Such insights will not only inform immediate action plans, but also foster discussion of broader engineering practices and scalability considerations, highlighting a balanced approach to system design and prioritization. To do.

close the loop

In this way, we effectively leveraged business priorities to address critical technical issues and schedule necessary fixes while deferring less urgent tasks. As this scheduling phase nears completion, additional steps beyond simple planning are required. It's about celebrating accomplishments. Highlighting the success of solving these projects emphasizes the critical role of engineering teams in solving business challenges and driving growth. This evaluation not only justifies our efforts but also strengthens the basis for future technology projects.

Looking back at past incidents, we have seen significant improvements in our service capabilities and tangible benefits from engineering advances. This is a reminder of our progress and the value of visualizing often-overlooked technical work to improve organizational agility and product quality.

summary

In summary, to effectively incorporate technical debt resolution into your roadmap, it is important to align it with the core needs of your business and the potential impact of addressing such debt. It is essential to champion the success of these projects as essential to business growth and articulate their value beyond mere technical fixes.

Use data to demonstrate the need and outcomes of these projects. Once completed, it is important to communicate the positive effects achieved and highlight how the business has surpassed previous constraints by overcoming these challenges.