Getting managers curious is one of the starting points of Systems Thinking. In our organisation I have presented internally many times on the subject (either informally, during pre-arranged lunch time sessions or at senior level briefings), sent round countless articles, organised discussions with organisations who have implemented and benefited from Systems Thinking, and organised external speakers to present on the subject; Barry Wrighton from Vanguard and Richard Durnell from Thoughtworks Australia.
Following on from these activities one of our senior managers recently proposed:
Although we have a number of metrics currently in place we should look to start again, with a greater focus on measuring improvements as experienced by people within the organisation or even by its customers. I suggest that managers and teams work together to consider what measures would help them understand their actions, understand the results, help them identify opportunities for improvement, and measure the impact of their actions. This is a different approach to our metrics thus far, which had been determined, analysed and owned by a fairly distant management group.
From curious to a normative loop experience*
Following this we piloted the creation of new useful measures within one of our product delivery teams, who had been working together on a product for just under 2 years.
To start with I did some digging around the current metrics in place. Most were found to be either project or code centric:
- Within program budget (Red, Amber, Green)
- Within project budget (Red, Amber, Green)
- On time (Red, Amber, Green)
- % unit test coverage (Target: >50)
- Relational cohesion (Target: >1 or <5)
- Afferent coupling at assembly level (Target: <15)
- Afferent coupling at type level (Target: <5)
- % methods will less than 10 lines of code (Target: <10)
- LCOM Henderson-Sellers (LCOMHS) (Target: <0.8)
Although valuable these measures have no real connection with operational improvement. What was needed was a completely new set of measures created from a quite different perspective; useful measures relating to what matters to our customers. These new metrics should tell everybody how well we are doing from a customer perspective.
Step 1 – Define the Purpose
To start we need to understand and define the purpose from the customer perspective, clarity of purpose and measures that relate to purpose are essential to improvement.
To help us in defining purpose we:
- Talked to our customers!
- Studied demand
- Looked at the company values
- Talked to the product owner and stakeholders
- Looked at previous business cases and presentations
- Reviewed organisational business units goals
How do you study purpose? Go out into the system and study demand, what matters to your customer. Look at Value Demand vs Failure Demand, go and talk to your customers!
Jeremy Cox – Vanguard
With this information we then held a session with the team to define:
- Who are our customers?
- What is the purpose of our product or service from the customer perspective
This particular team are regarded as highly successful, have worked together for quite some time, and have produced high quality software, but interestingly not many within the team had even thought about either of these questions. There were a number of business cases in place, a number of metrics purporting that they were successful, but each were from an internal perspective and not from a customer perspective.
If you find that despite lots of measures you don’t really know much about that matters to customers, then it can be a powerful starting place for change. Clarity of purpose and measures that relate to purpose are prerequisites to learning and improvement.
To define the purpose the team brainstormed, wrote keywords on PostIts, grouped these into common themes, and from there formulated the purpose.
Once the purpose had been defined we gained feedback from various stakeholders and customers to ensure it matched their impression.
Step 2 – Define Measures
As our senior manager had observed, measures are usually set by management who are often distant from the work. Our new measures should be defined by the whole team and by the workers doing the work.
Measures should be in the hands of those doing the work that are useful in respect to purpose. When people have clarity of purpose, measures in their hands that relate to that purpose, then they help people understand what is happening where they work, and they are able to contribute more in improving the work. This results in greater control and flexibility.
We held a second session where the team:
- Reviewed the current measures
- Defined new measures relating to the new purpose, for example
- Number of D****** orders
- Number of T****** orders
- Number of orders per cost
- End to end time to fulfill customer demand
- Exploitation of i****** c******
- Defined useful internal improvement measures
- Money spent vs expected ROI
- Lead Time (from idea to live)
- Cycle Time (from the time team starts work, to ready for live)
- # Live releases
- # Failed releases
- # Orders per month (throughput)
- # Live defects per week (Failure demand)
- # Support calls per week (Failure demand)
- # Change requests (Failure demand)
Note, some of the above has been obfuscated due to commercially sensitive information.
It was agreed that these measures would be plotted in a control chart over time. Measures taken over time become more reliable, more predictable and the impact of changes can be seen more clearly.
You can see an example of a control chart below. The chart is showing end to end times (from a customer perspective) for a service. The data has been split after a process has been changed, and again after an IT system has been introduced. This is using real operational performance data which is quite different from the normal IT measurement (on time and on budget). We can observe the real impact of change over time.
Step 3 – Implement
We are now on a real journey of continual improvement. The team have created measures relating to customer purpose and these measures are in their hands. Any future work will be analysed against these measures and progress regularly studied. It has also been decided that following the success of this pilot that all other teams should adopt the same approach.
Its also worth noting that our new measures will, for the time being, co-exist with existing measures. Over time we will reduce the old measures as people realise they are becoming less and less useful. Its important to note that there are various levels of managers who are likely to cling to old targets and become perturbed by the thought of loosing cherished measures. They need to experience the normative loop* and discover themselves why the new measures are better for the customer and the organisation’s service.
Step 4 – Study
When setting measures you may not get it right first time, review your data regularly; is Failure Demand dropping? are end to end times reducing? are your customers happier? Measures will evolve as you do.
A useful checklist from Vanguard:
- What is the purpose? (from the customer’s point of view)
- What are the current measures in use?
- Can you point to current measures of improvement?
- Do these current measures relate to purpose or do they encourage an internal perspective?
- How do current work measures impede workflow?
- What measures would be useful to access achievement of a customer focused purpose?
- How could these be used by the people who do the work?
Jeremy Cox of Vanguard ran a “making measures work for you” workshop at a recent Vanguard network day, I blogged about this here.
Mary Poppendieck and Tom Poppendieck also have an excerpt from their Leading Lean Software book on defining purpose.
I have also produced some results of internal improvement measures (plotted in control charts).