We Don’t Need No Frickin Architects

“We Don’t Need No Frickin Architects”

I’ve lost count the number of times that I have heard this statement (or some kind of derivative) during my career when mixing with Agilistas. The reasoning behind these statements is that architects are either defunct in the agile world, or that they need to roll their sleeves up and relearn to code alongside others in a delivery team. It makes no sense to over-egg the pudding.

I understand the reasoning, but as a Systems Thinker it is my belief that there is a need for architects.

Lets listen to what Dr Russell Ackoff, a Systems Thinking pioneer and organisational theorist, has to say on the subject.

A system has 3 properties:

  1. Each part of the system can affect the behaviour or the properties of the whole. They don’t necessarily do it all the time but they can.
  2. The way each part affects the whole, depends on at least what one other part is doing. The parts aren’t independent, they interact, they are connected. No part of the system has an independent effect.
  3. If you group the parts of a system into sub-groups i.e. sub-systems, they have the same properties the parts do. Every subsystem, or collections of parts, can affect the whole, and none has an interdependent effect.

The performance of a system is never a sum of the performance of its parts taken separately, its the product of its interactions. The performance of a system is based upon how the parts interact and fit, never on how they act separately.

There are many places where improving the performance of the part, will make the performance of the whole worse.

Architects knows this intimately:

  • An Architect draws the house first; the whole, then he adds the rooms; the parts.
  • He only improves a room, in a way that improves the house (the whole).
  • Sometimes a part needs to be made worse to make the whole better. If he can make the room worse, but make the house better, an Architect will do it.
  • The objective is to build the best house, not the best rooms.

If we have a system of improvement that’s directed at improving the parts taken separately, you can be absolutely sure the performance of the whole will not be improved.

The doctrine of the Western world is to think “divide and conquer”, if every separate part of the organisation is managed well, then the whole system will improve. This is absolutely false!

This is completely counter intuitive, we are committed to managing the actions of the parts, not their interactions. Most improvement programmes look to improve the parts separately not the whole. The whole is what should be the focus for continuous improvement.

In the above quote Ackoff is referring to the Organisation as a System, he is not referring to technology based systems. However the same thinking can be applied to technology based systems too.

We are encouraged in Agile to break work down into the smallest possible value adding chunks (often called either Epics, or Stories, or Features, or MMFs, or MVPs etc). Breaking work down makes sense. With the least amount of effort we can not only validate learning (by answering questions and testing hypothesis early, for example whether customers will use or buy our output), but also to minimise risk and possibly realise value earlier.

There are some thought leaders that even say that projects will soon become defunct, to be replaced instead by a constant flow of smaller chunks of work flowing through the delivery pipeline, with projects just becoming containers for these chunks of value adding increments.

As an Agile practitioner this is all good by my book, and makes sense to me.

But the Systems Thinker in me thinks differently. Lets listen again to what Ackoff is saying

“There are many places where improving the performance of the part, will make the performance of the whole worse

and that

“Architects knows this intimately”

So with a constant flow of small chunks of work, who is thinking about the whole? Who is concerned about the interactions between the parts? Who has the end game in mind?

I have spoken before about the difference between efficiency and effectiveness. Where efficiency is doing things right, but effectiveness is doing the right thing. My observation around Agile, Kanban and the like, is its all about efficiency, whereas we want to be more effective. I like the way a colleague of mine once put it

“We run the risk of rowing really fast as a high performing agile team, but in completely the wrong direction”.

As Peter Drucker says

“There is nothing so useless as doing efficiently, that which should not be done at all”.

So, we are at risk of building a whole load of well engineered parts that do nothing as a whole, or could in fact make things worse. Here is Ackoff again

When you try and put the best parts together you think you will get the best whole. This is wrong!

As an example, if you took all of the different cars on the market today, and asked a group of engineers which had the best engine, which had the best transmission, which had the best alternator etc and they took these best parts and tried to put them together, you would not get an automobile; the parts wouldn’t fit, let alone give you the best car in the world. It’s the working together that is the main contribution to systemic thinking.

Our entire culture is built on analytical thinking. The product of analysis is how things work, never why they work the way they do.
Analysis takes you inside the system, and how it works. It provides knowledge but not understanding.

By using a process of analysis you can reveal structure and say how something works, but not why something works in the way that it does.

Analysis has three steps:

  1. You take the thing you want to understand apart.
  2. You explain the behaviour of each part taken separately.
  3. Lastly, you reaggregate your explanation of the parts into an understanding of the whole.

Analysis became the dominant mode of thought in the Western world for hundreds of years. Today we still use analysis and thinking as synonymous terms. This thinking permeates all of our institutions, corporations, etc.

To reiterate, the process of analysis it to separate the whole into parts and study each part individually. Over to Ackoff

You will see this happening at an early age, if you give a child something they have never seen before, for example a puzzle, and they are confronted with a need to understand it, the first thing they will do is take it apart, the second thing they will do is to try to understand each part taken separately, then they will reassemble it to try and get an understanding of the whole.

This explains why inherently we feel that breaking work into parts to put them together again seems like a good thing.

To understand “why” questions, you need to use a process of synthesis. Synthesis is the opposite process to analysis and consists of three steps:

  1. You take the thing you want to understand as a part of a larger whole.
  2. You explain the behaviour of the containing whole.
  3. Lastly you disaggregate the understanding of the containing whole, by identifying the role or function of the parts.

Synthesis is an another way of thinking, it’s about making sense of the whole, which provides explanations of the behaviour of a system. You take the thing you want to understand and study it as a part of the larger whole.

If you want to understand how something works you use analysis, if you want to understand why it works you use synthesis. To be effective we should be using a combination of both.

The essential property of a system (whole) is that it cannot be divided into independent parts. If you apply analysis to a system, you take it apart and it loses all its essential properties, and so do its parts. This is easily demonstrated by Ackoff:

An example is a car, its a product of its parts. When a car is taken apart the system looses its essential property; moving from A to B. When we take a car apart we have cut its interactions. Not only that, we remove the essential properties of the parts. The motor cannot move from A to B on its own.

Here is another example, you can write, your hand can’t write, its easy to demonstrate by cutting your hand off and asking it to write, it wont do it!

As a sub note, I like the notion of Consumer Driven Contracts as an example of concern regarding interaction between parts:

A consumer-driven contract is a representation of a service provider’s obligations to all its current consumers. Consumer-driven contracts are created when each consumer communicates to the provider its specific expectations. The obligations created on the provider’s side establish a consumer-driven contract. As part of adopting a consumer-driven contract, a provider is free to evolve and innovate its service offering just so long as it continues to satisfy its existing obligations.
Source: http://www.infoq.com/news/2008/07/consumer-driven-contracts

However I do wonder if this is merely concerned on how two parts interact with each other, rather than understanding the whole.

What we do need is architects, but….

What I am not advocating is the style of architect that contemplates their navel for months on end, or sits atop an ivory tower dictating policy, standards, monolithic specifications, and the “one right way” to all. I’m also not suggesting that we introduce inspection regimes, or to build gold plated enterprise solutions that we realise are either out of date by the time they are released, or that more nimble competition have beaten us to it.

What I am suggesting however, is the need for someone who understands the whole, with the ability to use synthesis in addition to analysis. Someone whom is viewed as an additional stakeholder by all of those involved in the creation of the parts.

The modern architect gets visibility of, and has interactions with, the parts, but also has an appreciation of the whole. Agile brings transparency to the development process that can enable an architect to see what’s really going on within the parts, that allows them to react based on their objective of building the best whole, one that meets the customer’s needs.

There is a balance. The objective is to build the best house, not the best rooms.

What Did Deming Really Say?

There is a really great Quality Digest Article by Davis Balestracci entitled “What Did Deming Really Say?”

An extract is below.

The 1980 NBC television show, “If Japan Can, Why Can’t We?” introduced the teachings of W. Edwards Deming to U.S. viewers and caused a quantum leap in awareness of the potential for quality improvement in industry.

Those of you familiar with Deming’s funnel rules (which shows that a process in control delivers the best results if left alone) will smile to realize that his rule No. 4—making, doing, or basing your next iteration based on the previous one—also known as a “random walk,” has been in operation for the last 30 years.

Jeff Liker, professor of industrial and operations engineering at the University of Michigan, beautifully describes the random walks that have taken place within the time spans of Six Sigma and Lean. In a private correspondence with leadership expert Jim Clemmer, Liker writes:

“Originally Six Sigma was derived from Toyota Quality Management (TQM) by Motorola to achieve six sigma levels of quality, and then through Allied Signal and GE it morphed to projects by Black Belts based on statistics to become a cost-reduction program – every project needs a clear ROI. In other words, we denigrated the program from a leadership philosophy to a bunch of one-off projects to cut costs. It was a complete bastardization of the original, and it rarely led to lasting, sustainable change because the leadership and culture were missing.”

“A similar thing happened to Lean when it got reduced to a toolkit (e.g. value-stream mapping, KPI boards, cells, kanban).”

“Lean and Six Sigma in no way reflect the original thinking of excellent Japanese companies or their teachers like Deming.”

Clemmer also cites multiple studies from 1996–2007 concluding that about 18 to 24 months after these various quality systems are launched, 50–70 percent of them fail. Liker concurs and feels that the four key failure factors, in this order, are:

  1. Leadership lacking deep understanding and commitment
  2. Focus on tools and techniques without understanding the underlying cultural transformation required
  3. Superficial program instead of deep development of processes that surface problems solved by thinking people
  4. Isolated process improvements instead of creating integrated systems for exceptional customer value

Virtually everyone agrees that the No. 1 barrier to improvement is still top management’s inability to be visibly committed to quality. Is this the “elephant in the living room” or as Clemmer calls it, “the moose on the table”?

The Kanban Dance LSSC11

Here is the Kanban dance from LSSC11*

The moves are

  • Double Loop Learning
  • Columns
  • Swimlanes
  • Cards

* Not in any way endorsed by David J Anderson and Associates.

Learning at Conferences

  • An ounce of information is worth a pound of data.
  • An ounce of knowledge is worth a pound of information.
  • An ounce of understanding is worth a pound of knowledge.
  • An ounce of wisdom is worth a pound of understanding.
  • That makes an ounce of wisdom about 42,000 ounces of data.

Despite this, most of the time spent in school is devoted to the transmission of information and ways of obtaining it. Less time is devoted to the transmission of knowledge and ways of obtaining it (analytical thinking). Virtually no time is spent in transmitting understanding or ways of obtaining it (synthetic thinking). Further more, the distinctions between data, information, and so on up to wisdom are seldom made in the educational process, leaving students unaware of their ignorance. They not only don’t know, they don’t know what they don’t know.

The reason so little understanding is transmitted by teachers is that they have so little to transmit. They are more likely to know what is right than why it is right. Explanations require discussion if they are to produce understanding. The ability to lead fruitful discussions is not an attribute of most teachers.

Russell Ackoff, 1999

This is why I feel I have greater learning from corridor conversations at events such as LSSC11 than actually in the sessions themselves. Some feel the conference format is outdated, but I’m unsure on the alternative.

We Are Beyond Your Mindset

Your firms are built on the Taylor model. Even worse so are your heads. With your bosses doing the thinking while workers wield the screwdrivers, you’re convinced deep down that is the right way to run a business. For the essence of management is getting the ideas out of the heads of the bosses and into the heads of labour.

We are beyond your mindset. Business, we know, is now so complex and difficult, the survival of firms so hazardous in an environment increasingly unpredictable, competitive and fraught with danger, that their continued existence depends on the day-to-day mobilisation of every ounce of intelligence.

Konosuke Matsushita founder of Panasonic, Technics

Lean Software Management BBC Worldwide Case Study

Dr Peter Middleton and I have had our “Lean Software Management BBC Worldwide Case Study” paper accepted by the IEEE Transactions on Engineering Management. It will be published in the February 2012 issue. The paper was edited by Dr Jeffrey K Liker, author of the Toyota Way.

You can download a copy of the paper prior to its publication here.

I believe it will be one of the most significant papers in Software Engineering this decade.

David Anderson

Abstract

This case study examines how the lean ideas behind the Toyota production system can be applied to software project management. It is a detailed investigation of the performance of a nine person software development team employed by BBC Worldwide based in London. The data collected in 2009 involved direct observations of the development team, the kanban boards, the daily stand-up meetings, semistructured interviews with a wide variety of staff, and statistical analysis.

The evidence shows that over the 12-month period, lead time to deliver software improved by 37%, consistency of delivery rose by 47%, and defects reported by customers fell 24%.

The significance of this work is showing that the use of lean methods including visual management, team-based problem solving, smaller batch sizes, and statistical process control can improve software development. It also summarizes key differences between agile and lean approaches to software development. The conclusion is that the performance of the software development team was improved by adopting a lean approach. The faster delivery with a focus on creating the highest value to the customer also reduced both technical and market risks. The drawbacks are that it may not fit well with existing corporate standards.

Value Delivered

The paper doesn’t include the increase in business value delivered over the period of study. This was due to confidentiality agreements. What I can say is that during the period of study, the digital assets produced rose by hundred of thousands of hours of content, a 610% increase in valuable assets output by software products written by the team.

Authors

Peter Middleton received the M.B.A. degree from the University of Ulster, Northern Ireland, in 1987, and the Ph.D. degree in software engineering from Imperial College, London, U.K., in 1998.

He is currently a Senior Lecturer in computer science at Queen’s University Belfast, Northern Ireland. He is the coauthor of the book Lean Software Strategies published in 2005, and the Editor of a book of case studies on applied systems thinking: the Delivering Public Services that Work published in 2010. His research interests include combining systems thinking with lean software development to help organizations significantly improve their performance.

David Joyce is a Systems Thinker and Agile practitioner with 20 years software development experience of which 12 years is technical team management and coaching experience. In recent years, David has led both onshore and offshore teams and successfully led an internet video startup from inception to launch. More recently David has coached teams on Lean, Kanban and Systems Thinking at BBC Worldwide in the U.K. He is a Principal Consultant at ThoughtWorks.

Mr. Joyce was awarded the Lean SSC Brickell Key award for outstanding achievement and leadership