Development Productivity in the Age of Generative AI

“Productivity is being able to do things that you were never able to do before.”

—anon

With generative AI grabbing almost every organisation’s attention, watching how they use the technology is interesting. Today most AWS customers are focused on achieving productivity gains (e.g., making information easier to acquire and digest for customer service staff). A second, much smaller set of customers are looking at more disruptive uses of generative AI but not to the point of redefining industries. These use cases range from rethinking customer experiences at Accor and Booking.com to accelerating pharmaceutical research using synthetic data. A virtuous process will likely manifest in most organisations where efficiencies free up people and investments to reimagine and reinvent industries.

All good so far. But companies are starting to ask: how do you measure productivity, and is it important to do so?

Rather than talk about productivity in general terms, let’s focus on developer productivity. We know that an exorbitant amount of time is wasted in the average enterprise, as discussed in my blog post on developer productivity. Here is the challenge: If we could save twenty hours of a developer’s time with process changes and the use of Amazon Q Developer, would it be invested in meaningful areas, or would we just find creative new ways of wasting our precious resources?

To learn more about this topic, I spent time with Joe Cudby, a principal specialist in AWS’s Builder Experience team.

Joe, the term developer productivity is relatively new and not something I was familiar with as a former developer. AWS doesn’t typically use this term. Why is that?

My team regularly discusses software development journeys with AWS customers, including how they conceive of and measure developer productivity. But a clear shift is underway from a narrow focus on an individual developer’s productivity to a more expansive understanding of team development productivity at the organisational and SDLC levels. This shift, like embracing DevOps before it, necessitates real cultural change paired with increased collaboration that expands beyond engineering to intersect with business leadership, product management, and end users. We find that simply measuring a single individual’s productivity without these broader considerations is not typically useful.

What has prompted this focus on teams rather than individuals?

There is no single simple metric that can provide a full and nuanced picture of overall productivity or progress. Measures like lines of code developed were discredited many years ago, but overly simplistic measures such as story points—which are subjective and a measure of effort rather than value—are often still used. Using story points is a great way of planning and estimating work, but they are often misused as a measure of productivity. This problem becomes more pronounced when attempting to compare the productivity of individuals and teams even within the same company. Teams face very different challenges and opportunities depending on the heterogeneity or homogeneity of their technology stack and architecture. Within each of these applications and technology stacks, multiple languages, frameworks, and technical debt have accrued to differing levels over many years. Even the difference between maintaining a system and developing a new one creates significant productivity differences.

So how can an organisation start to think about development productivity rather than developer productivity?

AWS uses three core dimensions that extend beyond just productivity. The first is system health, which measures end-user-focused outcomes like product quality, performance, availability, and feature adoption. After all, being productive but producing the wrong outcomes is an oxymoron. Second, we look at the efficacy of the CI/CD processes, namely volume and speed metrics like deployment frequency, lead time between commits, and change failure rate. You might assume that this is just a velocity measurement, but implicit in this is the requirement to hit quality and security bars. Finally we look at team health across factors such as job satisfaction, turnover risk, and burnout indicators based on regular employee surveys and scoring. This might seem like a soft set of measures, but burnout is highly correlated to loss of productivity.

There are a number of other frameworks organisations could adopt, such as the SPACE framework, which looks across the five dimensions of employee satisfaction and well-being, outcomes, activity metrics such as code and task output metrics, communication and collaboration, and flow efficiency. The DORA framework is probably the best known; it connects technical capabilities to productivity by assessing test automation, architectural practices, and more. These drive productivity metrics like lead time and deployment frequency, which directly connect to overall organisational performance and employee well-being.

None of these are simple, single metrics. This reflects the fact that development productivity depends on the interplay and intersections of all of these various factors at the individual engineer, team, and company levels. The key to understanding development productivity is combining objective data on system-level outcomes, engineering output and activity, and regular surveys and scoring of subjective developer sentiment. This three-pronged insight informs a sustainable culture that retains top talent. To paraphrase an old saying, measuring the wrong things is often easy; measuring the right things might be hard but pays off.

How can these development productivity metrics be misused?

These metrics are often intended to indicate health rather than hire, fire, and reward. They should be used to help a team improve and self-monitor. One antipattern to avoid is Goodhart’s Law, where a measure of, say, development efficacy becomes a target. A common misuse is to use the metrics to compare teams despite the differences in technologies and challenges I outlined earlier. Use these metrics for what they are intended for: enabling teams to identify areas of improvement, monitoring the efficacy of their working practices, and ensuring this is all done sustainably.

What does the introduction of tools such as Amazon Q Developer mean to development productivity?

We see Amazon Q Developer being used by companies such as British Telecom to help in areas like code recommendations, identifying quality and security issues, automating testing, and providing personalised productivity insights. Amazon Q Code Transformation is another tool that helps reduce technical debt at scale, such as upgrading versions of.NET or Java. It gets development teams back to what they and their businesses want: delivering business outcomes. Be creative! AWS customers use these tools to create synthetic test data and unit test scripts, develop features, troubleshoot, remediate potential vulnerabilities, and more. An important side benefit of these tools is that emerging research shows the use of AI-based developer tools correlates to improved well-being.

What advice would you give on improving development productivity?

Improving development productivity should be viewed as a continuous, incremental journey of small gains compounded over time rather than seeking a one-time “silver bullet”. Start by using the data and metrics you have today to develop a baseline. Many of these will come from CI/CD tools that measure velocity, but don’t forget to look at your code quality and security posture. Adding a regular subjective survey is an easy additional step. Often the people who know how to improve productivity are the very people you are measuring, so ask for their insights! Pilot changes with your development teams and evaluate their impact. Use a combination of tools such as Amazon Q Developer with work changes that minimise distractions, reduce work in progress and dependencies, and focus the team on the highest value add work.

Many AWS customers outsource development. Any advice for them?

I have to believe that while, ultimately, both an outsourcer and an AWS customer might have different goals, they want to ensure their development teams are productive, working towards meaningful outcomes, and using the tools available to constantly improve team outcomes. Most of these more mature arrangements understand that the value is not in just charging for hours spent developing but in truly delivering value. I’ve seen AWS customers and their partners sit down and create transparency for the metrics I have discussed in this blog and then use these to look for improvements wherever they may lie. Ensuring the use of tools like Amazon Q Developer is encouraged in contracts is a step in this direction.

Conclusion

It is an exciting time to be a software developer. For those who have been at this for a while, this is one more evolution of the tooling we have grown up with, improving the ability to focus more on delivering valuable business outcomes and value. What I love about Joe’s focus on development productivity is the implicit acknowledgement that the unit of productivity for most organisations today is the team, not a solitary individual. Perhaps we technologists can help colleagues elsewhere in our organisations embrace this lesson too.

—Phil