The research paper titled “DevEx: What Actually Drives Productivity” from the DX team was released last month*. It explores a new developer-centric approach to measuring and improving productivity with a focus on developer experience (DevEx).
As emphasised by Abi Noda (Co-Founder & CEO of DX) in his introduction to the paper “with the proliferation of vendors and consultants promoting ineffective metrics, it can be difficult to know where to turn.”
In this post, we provide a summary of this paper.
Let’s start at the beginning, what is DevEx?
Developer Experience focuses on the lived experience of developers and identifies factors that affect their experience, such as:-
- interruptions,
- unrealistic deadlines, and
- friction within teams and with development tools.
The paper highlights that human factors, like having clear goals and feeling psychologically safe on a team, significantly impact developers’ performance and productivity. Traditional methods of measuring developer productivity, such as output or task completion time, fail to account for the complex and diverse activities that developers engage in. Thus they aren’t a reliable metric for measuring true DevEx.
Core Dimensions of DevEx
As a result, the research proposes a framework consisting of three core dimensions of DevEx: feedback loops, cognitive load, and flow state.
- Feedback loops refer to the speed and quality of responses to developers’ actions
- Cognitive load relates to the mental processing required to perform tasks
- Flow state represents a mental state where developers are fully immersed and focused on their work
These three core dimensions encompass the 25 sociotechnical factors affecting DevEx that were brought to attention in a previous research paper from Noda and colleagues**.
Three Core Dimensions of Developer Experience from: “DevEx: What Actually Drives Productivity”
The authors emphasise the importance of measuring DevEx to identify areas for improvement and track the impact of efforts. Measuring DevEx involves capturing both developers’ perceptions and objective data about engineering systems and processes.
The table below illustrates this measurement approach by assessing KPI metrics (i.e. “North Star metrics”) with specific examples of perceptual and workflow measures across the three core dimensions.
Example of DevEx Metrics from the Paper: “DevEx: What Actually Drives Productivity”
It really brings home the importance of understanding that one metric or dimension alone is not enough for measuring DevEx.
“For example, seemingly fast code review turnaround times may still feel disruptive to developers if code reviews regularly interrupt their work progress. As a counterexample, even if developers feel content with their build processes, using objective measures such as build time may reveal that feedback loops are slower and development workflows less streamlined than they could be. Developers often have concerns about the misuse or misinterpretation of metrics by leaders.”
By combining both the developer’s perceptions (their attitudes, feelings and opinions) and the objective data from the systems and processes, developer teams get the insights and recommendations for how to improve.
So how is this data best collected?
According to the authors, in order to measure Developer Experience (DevEx), surveys are crucial for capturing feedback and identifying friction points, providing fast and accurate measurements when designed correctly. Additionally, collecting data from systems can offer end-to-end insights, although it may require instrumentation and normalisation.
When conducting surveys, it is essential to design questions carefully, break down results by team and persona, compare against benchmarks, and incorporate transactional surveys. To maintain participation rates, leaders and teams must follow up on survey findings and take action, while considering the optimal survey cadence for their organisation.
Although this sounds like a thorough approach, it is unlikely smaller developer teams will have the resources to conduct these surveys. However, the researchers emphasise the importance for SMEs and startups to be aware of the factors involved in DevEx as they are integral for “innovating quickly and finding market fit.”
Benefits of DevEx Initiatives for Organisations
According to Gartner Inc, 78% of organisations surveyed have a formal DevEx initiative established or plan to do so in the near future. This shows that companies are aware and focussed on the importance of improving DevEx.
The paper provides real-world examples of larger companies, such as eBay and Pfizer, that have implemented DevEx initiatives to accelerate developer velocity.
eBay, for instance, has a dedicated DevEx team that conducts quarterly surveys and collaborates with feature teams to identify and address specific improvement areas. With these results they have fixed tooling and documentation gaps, automated or removed steps and processes for development and release, and streamlined team collaboration. They conducted regular retrospective sessions where developers could give their own feedback and make small improvements to the process to help create a feedback loop where they could to improve DevEx. As a result, just in the past year, updates are released twice as fast as before!
In Summary
The developer-centric approach proposed by the Noda et al., to measure and improve productivity, offers valuable insights and opportunities for organisations aiming to enhance developer productivity.
Shortening feedback loops, reducing cognitive load, and creating optimal conditions for flow state provide developer team leaders with a clearer idea of what to focus on to achieve developer velocity.
Do you agree? What are your thoughts on this approach to measuring DevEx and it’s impact on productivity? Please let us know your thoughts on this approach in the comments below!
Links to research papers:
*Noda, A., Storey, M., Forsgren, N., Greiler, M. 2023 DevEx: What Actually Drives Productivity; https://queue.acm.org/detail.cfm?id=3595878
**Greiler, M., Storey, M., Noda, A. 2022. An actionable framework for understanding and improving developer experience. IEEE Transactions on Software Engineering; https://ieeexplore.ieee.org/document/9785882.
