Creating Useful Measures

Getting managers curious is one of the starting points of Systems Thinking. In our organisation I have presented internally many times on the subject (either informally, during pre-arranged lunch time sessions or at senior level briefings), sent round countless articles, organised discussions with organisations who have implemented and benefited from Systems Thinking, and organised external speakers to present on the subject; Barry Wrighton from Vanguard and Richard Durnell from Thoughtworks Australia.

Following on from these activities one of our senior managers recently proposed:

Although we have a number of metrics currently in place we should look to start again, with a greater focus on measuring improvements as experienced by people within the organisation or even by its customers. I suggest that managers and teams work together to consider what measures would help them understand their actions, understand the results, help them identify opportunities for improvement, and measure the impact of their actions. This is a different approach to our metrics thus far, which had been determined, analysed and owned by a fairly distant management group.

From curious to a normative loop experience*

Following this we piloted the creation of new useful measures within one of our product delivery teams, who had been working together on a product for just under 2 years.

To start with I did some digging around the current metrics in place. Most were found to be either project or code centric:

• Within program budget (Red, Amber, Green)
• Within project budget (Red, Amber, Green)
• On time (Red, Amber, Green)
• Velocity
• % unit test coverage (Target: >50)
• Relational cohesion (Target: >1 or <5)
• Afferent coupling at assembly level (Target: <15)
• Afferent coupling at type level (Target: <5)
• % methods will less than 10 lines of code (Target: <10)
• LCOM Henderson-Sellers (LCOMHS) (Target: <0.8)

Although valuable these measures have no real connection with operational improvement. What was needed was a completely new set of measures created from a quite different perspective; useful measures relating to what matters to our customers. These new metrics should tell everybody how well we are doing from a customer perspective.

Step 1 – Define the Purpose

To start we need to understand and define the purpose from the customer perspective, clarity of purpose and measures that relate to purpose are essential to improvement.

To help us in defining purpose we:

1. Talked to our customers!
2. Studied demand
3. Looked at the company values
4. Talked to the product owner and stakeholders
5. Looked at previous business cases and presentations
6. Reviewed organisational business units goals

How do you study purpose? Go out into the system and study demand, what matters to your customer. Look at Value Demand vs Failure Demand, go and talk to your customers!

Jeremy Cox – Vanguard

With this information we then held a session with the team to define:

1. Who are our customers?
2. What is the purpose of our product or service from the customer perspective

This particular team are regarded as highly successful, have worked together for quite some time, and have produced high quality software, but interestingly not many within the team had even thought about either of these questions. There were a number of business cases in place, a number of metrics purporting that they were successful, but each were from an internal perspective and not from a customer perspective.

If you find that despite lots of measures you don’t really know much about that matters to customers, then it can be a powerful starting place for change. Clarity of purpose and measures that relate to purpose are prerequisites to learning and improvement.
John Seddon

To define the purpose the team brainstormed, wrote keywords on PostIts, grouped these into common themes, and from there formulated the purpose.

Once the purpose had been defined we gained feedback from various stakeholders and customers to ensure it matched their impression.

Step 2 – Define Measures

As our senior manager had observed, measures are usually set by management who are often distant from the work. Our new measures should be defined by the whole team and by the workers doing the work.

Measures should be in the hands of those doing the work that are useful in respect to purpose. When people have clarity of purpose, measures in their hands that relate to that purpose, then they help people understand what is happening where they work, and they are able to contribute more in improving the work. This results in greater control and flexibility.

John Seddon

We held a second session where the team:

1. Reviewed the current measures
2. Defined new measures relating to the new purpose, for example
   • Number of D****** orders
   • Number of T****** orders
   • Number of orders per cost
   • End to end time to fulfill customer demand
   • Exploitation of i****** c******

Note, some of the above has been obfuscated due to commercially sensitive information.

3. Defined useful internal improvement measures
   • Money spent vs expected ROI
   • Lead Time (from idea to live)
   • Cycle Time (from the time team starts work, to ready for live)
   • # Live releases
   • # Failed releases
   • # Orders per month (throughput)
   • # Live defects per week (Failure demand)
   • # Support calls per week (Failure demand)
   • # Change requests (Failure demand)

It was agreed that these measures would be plotted in a control chart over time. Measures taken over time become more reliable, more predictable and the impact of changes can be seen more clearly.

You can see an example of a control chart below. The chart is showing end to end times (from a customer perspective) for a service. The data has been split after a process has been changed, and again after an IT system has been introduced. This is using real operational performance data which is quite different from the normal IT measurement (on time and on budget). We can observe the real impact of change over time.

Step 3 – Implement

We are now on a real journey of continual improvement. The team have created measures relating to customer purpose and these measures are in their hands. Any future work will be analysed against these measures and progress regularly studied. It has also been decided that following the success of this pilot that all other teams should adopt the same approach.

Its also worth noting that our new measures will, for the time being, co-exist with existing measures. Over time we will reduce the old measures as people realise they are becoming less and less useful. Its important to note that there are various levels of managers who are likely to cling to old targets and become perturbed by the thought of loosing cherished measures. They need to experience the normative loop* and discover themselves why the new measures are better for the customer and the organisation’s service.

Step 4 – Study

When setting measures you may not get it right first time, review your data regularly; is Failure Demand dropping? are end to end times reducing? are your customers happier?  Measures will evolve as you do.

Further information

A useful checklist from Vanguard:

1. What is the purpose? (from the customer’s point of view)
2. What are the current measures in use?
3. Can you point to current measures of improvement?
4. Do these current measures relate to purpose or do they encourage an internal perspective?
5. How do current work measures impede workflow?
6. What measures would be useful to access achievement of a customer focused purpose?
7. How could these be used by the people who do the work?

Jeremy Cox of Vanguard ran a “making measures work for you” workshop at a recent Vanguard network day, I blogged about this here.

Mary Poppendieck and Tom Poppendieck also have an excerpt from their Leading Lean Software book on defining purpose.

I have also produced some results of internal improvement measures (plotted in control charts).

* Normative loop – people have to be able to “see” and “experience” for themselves

Advertisement

Journey to Systemic Improvement – Lean eXchange video

The video from my Journey to Systemic Improvement talk at the UK Lean eXchange conference is now available.

I was originally told I had 50 mins to present plus questions, then was told just before going on I had 40 mins, so zipped through it!!!

A video of Benjamin Mitchell and I running the Red Bead Experiment in the openspace session towards the end of the day is also available. If you missed our previous “In the brain of session” where we ran the Red Bead Experiment, you can find this here (note the sound quality for this first experiment is much better).

Finally Benjamin and I interviewed each other after the Lean eXchange event, which can be viewed here.

Journey to Systemic Improvement – Lean eXchange presentation

Today I gave a talk at the UK Lean eXchange entitled Journey to Systemic Improvement.

My slides can be found here.

Note it is a media rich presentation so the PDF is almost 50MB!!!

A video recording of the presentation and our second running of the Red Bead Experiment will soon be available.

Red Bead Experiment Video and Materials

Benjamin Mitchell and I recently ran the Red Bead Experiment at Skills Matter.

We changed the experiment slightly to revolve around a software project rather than a factory:

• Product Owner and a Project Manager jointly played the role of Foreman
• A burndown of white beads was referred to
• Discussion afterwards focused in part on software based projects

A video of the session can be found here.

If you would like to run the session yourself then you may find our script of use, along with the F-Test, motivational posters and lessons. If you would like the spreadsheet that creates the control chart and burndown then please mail me as WordPress wont allow me to upload. You can buy the Red Bead Experiment kit here. Note we also supplied our own plastic box for decanting the beads, a tray to catch spillages, clipboards and pens for the inspectors.

The Transformation Forum run a Red Bead Experiment masterclass for those looking to learn how to run the experiment and meet with others who have done so over many years.

Systems Thinking, Cultural Change is Free

I have really been inspired by John Seddon’s Cultural Change is Free video (skip the first 5 minutes intro/waffle).

Here are the key points I took from his talk (which you may find useful if you dont have the time to watch the whole talk):

When asked how much work we have coming in, how many people we have, and how long it takes – the normal answer is to set targets. Morale goes down having to work to arbitrary targets.

If you are incentivised to sell only high value items then what happens to the majority of low value work? It gets ignored. When you incentivise behaviour you get less of the work you want, people get focused on the incentive not on doing the work.

You need to understand type and frequency of demand in customer terms; What is the customer trying to pull from your system?

• Value demand – thats why we are here
• Failure demand – failure to do something or do something right for the customer

Failure demand often runs over 50% of the system e.g. “you sent me a blue one and I wanted a green one” “it didnt fit” “it doesnt work”

Study demand in customer terms until you predict demand going forward. A lot of demand is predictable. Its only the predictable failure demand that you can turn off. Turning off failure demand is a massive economic lever. Deming says things will always go wrong, you need to find out what goes wrong predictably. Only the predictable is preventable. The only way to turn off predictable failure demand is to redesign the service.

Obsession with costs is the wrong obsession. If you manage costs your costs go up.

Variation is in the work, not in the workforce.

Working on people only tackles 5% working on the system tackles the other 95%

Study the work, study the causes of variation, what are the causes; the system is 95% of the answer. Take the big ones and then go to work on changing them, you will then see dramatic improvements. For example

• Is the worker trained on what the customer wants?
• Has the management studied demand?
• Are procedures working?
• Is the IT system working?
• Have the people in marketing sent something to the customer that others arent aware of?

Train your workers on high frequency value demand; stuff we get a lot of. They will get demand they are not trained for, they need to recognise this, when it happens they pull support from a specialist or their manager, the work stays with them and they work with the manager or specialist to complete it. The rate of learning for the worker becomes very fast.

Inevitably some of the work has to go elsewhere, so we need to concentrate on flow. Does the person about to hand off work understand what “clean” looks like to the person receiving it in the flow? They visit the person who will receive it and ask “what do you do with it” “what do you need from me for it to be clean when it arrives?” If everyone does this through the value chain you see vast improvements and far less rework.

You design the system to see the waste, when it is removed the system improves. When you improve the design to handle demand that hits the front of the system and the flow of work that gets handed off – costs go down by a lot.

If you want someone to do a good job then design a good job to do. The worker has to be responsible and have the means to control their own work. The job of management changes to a co-operative role, working on the system, fixing things outside of the workers control.

Counter intuitive truths

• Cost is not in activity, cost is in flow
• Standardisation does not give control, it creates waste
• Targets and all other abitery measures make your system worse

Systemic relationship between Purpose -> Measures -> Method

When you impose arbitrary measures such as targets into your system you create a defacto purpose which is to meet the targets, and you constrain method.

When you derive your measures from the purpose of the service from the customers point of view, and put those measures in the hands of the people doing the work, you liberate method. Innovation occurs, we are bringing the brain to work, we are using ingenuity; not to survive in the system, but to be engaged and understanding how to improve their work. When you design systems this way people actually work harder and they are less stressed!

There are 3 things every manager needs to know about targets

1. Targets and all other arbitrary measures always make performance worse
2. There is no reliable method for setting a target
3. When you use real measures derived from the purpose of the work from the customers point of view, in the hands of the workers, you achieve a level of improvement you would never have dreamed of setting as a target.

Culture change is free. Change is emergent. When you change the system, the behaviour changes.

The nature of change; we are taught if you want to change something you have to have a plan, set some milestones with some deliverables, and perform cost benefit analysis. Systems thinkers say there is no requirement for a plan, the only plan is get knowledge, change starts by studying the current system.

Follow the value work through the flow, see where they go. There is only 2 types of work going on; the work the customer wants to pull and everything else is waste. Forget the 7 wastes in manufacturing.
You cant get rid of waste until you understand its causes, so you need to understand what is causing the waste; design of you system, targets, IT systems etc

As you start a change by studying the work, you start to learn this mean me too! If you stop doing the wrong thing you stop getting worse. If you want people to innovate the responsibility has to be with them. Ask what measures are you using to understand and improve the work?

Economy comes through flow and not scale, the only way to absorb variety is to use people; to design them into proper jobs in these systems, we need human solutions to these human problems.

Failure Demand

There are 2 types of customer demand in a transactional service system.

Value Demand:

Demand we want. Why we are here from the customer’s point of view

Failure Demand:

A failure to do something or do something right for the customer

If we can switch off failure demand, we can increase the capacity of the system. To do that, we need to learn how to design our services to meet our customers value demands.

Learn more about Failure Demand

Lead Time vs Cycle Time

Lead time clock starts when the request is made and ends at delivery. Cycle time clock starts when work begins on the request and ends when the item is ready for delivery. Cycle time is a more mechanical measure of process capability. Lead time is what the customer sees.

Lead time depends on cycle time, but also depends on your willingness to keep a backlog, the customer’s patience, and the customer’s readiness for delivery.

Another way to think about it is: cycle time measures the completion rate, lead time measures the arrival rate. A producer has limited strategies to influence lead time. One is pricing (managing the arrival rate), another is managing cycle time (completing work faster/slower than the arrival rate).

Corey Ladas

If you want to learn more about where our fixation on Lead Times and Cycle Times comes from, then take a look at my new free book entitled Theories of Work: How We Design and Manage Work or go to the specific chapter in the book here.