DevOps Metrics Matter: Why, Which Ones, and How

“When you can measure what you are speaking about, and express it in numbers, you know something about it.”

~ William Thomson Kelvin, physicist, mathematician and engineer

More simply put, you can’t manage what you can’t measure. To get better at DevOps (or anything) you need to reflect on what you’ve done and compare where you’re at to where you want to be. That’s where Value Stream Management comes in.

Agile provided the foundational basis for improving and managing the software development product and process through: active customer collaboration, dynamic change response, short development cycles, frequent deliveries, continuous face-to-face team communications (Scrum & Sprint), and ongoing learning. Lean’s main thrust was the identification and elimination of waste, which is any activity consuming resources, time, or space without maximizing customer value.

Value Stream Management (VSM) builds on these concepts to visualize and analyse end-to-end holistic workflow capabilities of product, value, and time from current state (idea or concept) to delivered state through the production stream. VSM focuses on improving flow and continuously optimizing business value, producing idea (code)-to-cash, and stakeholding in product/development management meetings while ensuring efficient operating environments committed to the highest quality.

Sounds pretty powerful, right? But a VSM tool is only as effective as the metrics it measures. Metrics help software development teams to understand, evaluate, control, make informed decisions, and predict flow of products/processes during software life-cycle stages. Alternatively stated,

“The goal of applied software measurement is to give software managers and professionals a set of useful, tangible data points for sizing, estimating, managing, and controlling software projects with rigor and precision”.

~ C. Jones, Applied Software Measurement: Global Analysis of Productivity and Quality

Need more convincing that metrics matter? Here are five reasons to take the time to measure the right things, and measure them right:

1. Fixing software process issues (identifying problems, improvement opportunities)
2. Understanding/improving quality (understanding quality level, increasing quality, ensuring thorough testing)
3. Problem avoidance/rectification (faster response, fixing defects, changing behaviour, preventing issue actualization)
4. Agile progress tracking (sprint progress, increasing visibility, achieving goals, balancing workflow)
5. Agile planning (prioritizing, scoping, resourcing)

Now that you’re sold on measuring your DevOps performance, let’s talk about what you should measure. HCL Accelerate has the ability to track many aspects of your software delivery pipeline, and which ones you focus on should depend on your organization’s unique key performance indicators. In most cases, the metrics that you’ll want to pay attention to are:

1. Lead time
2. Wait time
3. Throughput
4. Work item distribution
5. Change defect rate
6. Load

Lead time

Lead time is the time required for a work item to go from idea acceptance as unit of work to value realization from that work that includes both value added processing time and non-value added waiting time between sub-processes. The processing time for the work item is the time duration expended by a person or team on the activity. Ideally, the lead time value should trend to one day to shorten time-to-market and improve customer satisfaction.

Wait time

The wait (queueing or waste) time is an estimate of the time that the work item spends idle in a non-productive state during its processing by the value stream. Software product customizations (SPCs) are customer tailored solutions (typically high priority) requiring implementation in the current software release. Various sources of waiting time in SPCs that value stream managers are required to minimize and/or eliminate for maximizing both customer and product value include: code reviews, QA testing, security testing, and release cycles.

Throughput

The throughput of a value stream is the average rate of work unit items completed during a time period and quantifies the level of delivery value in the flow. The rate of work units processed by a given stage in a period of time may be expressed as its stage throughput. In order to improve responsiveness to customer requirements, goals identifying throughput improvements and then implemented will yield lead time reductions for value streams with fixed load capacities.

Work item distribution

The work item distribution is the percentage of work items completed relative to the total number of work items in a given set (same item types). Sets are user-specified and may include process-based (release or sprint) or time-based (completed items in given time period) items. This metric is used to prioritize specific work types aligned with projected business results.

Change defect rate

In the value stream, change defect rate is a quality measure expressed as a ratio of the number of defects integrated within the value stream over the completed items during a given time interval. Change defect rate aids with understanding the predictability of our value stream operation and provides insights for these questions:

• Are defects being detected early in our development processes?
• Is there consistent and reliable delivery in production environments without impacting our customers or business?

Teams operating with high change defect rates have rapid advance potential at the expense of having inadequately established quality standards allowing them to effectively move quickly and efficiently. Fixing team effort on defect identification in the value stream prior to entering production provides the basis for lowering change defect rate percentage rates.

Load

The load is the number of work items (active or waiting) active in a value stream at a given time. Load measures utilization capabilities of value streams related to productivity in the process flow.

Summary

It can seem daunting to track these metrics in your software delivery pipeline – manually gathering and measuring data from disjointed tools would be a pain. But Value Stream Management makes measurement too easy to dismiss, and these metrics are too important to ignore. HCL Accelerate, our Value Stream Management tool, automatically connects everything in your existing DevOps toolchain and gathers the data in visual reports, maximizing your entire measurement spectrum for an exceptionally tuned software performance experience. Try it for yourself with the free Community Edition of HCL Accelerate.