Dave Snowden

The banality of measurement

RSS Feed

One of the insights which can come with age/experience is that the vast majority of people have good intentions or at least do not intend evil. Most are caught up in a flow of events in which the needs and expectations of the present dominate actions. Evil often results and may then be compounded by personal choice, but how far that choice is deliberative or personal remains open to question. If you look at the pictures of any acknowledge tyrant of modern times when they were a child do you see the evil that came after in their eyes? The corollary of that is that Good and greatness are similarly circumstantial in nature; personal choice can amplify, direct or dampen what is an emergent property of multiple interactions over time. For all of us there are bifurcation points at which we can make choices but few are saintly enough, or evil enough to break a wider pattern of social interaction and the dominant tropes of their society. I’m going to look at this issue more specifically in the context of leadership tomorrow when I have finished reflecting on yesterday’s speech by Obama. Today I want to return to something I have addressed in many a speech and post over time, namely the nature of measurement.

One reason this is on my mind is that it came up in a conversation I recorded yesterday at Bangor University. I’m up all this week (bar a side trip to London today) working on the set up of the new Centre for Applied Complexity (CfAC). John (Dean of my host department Psychology) and Andy (Design guru, Pontio, innovation etc) got to gather in the newly opened Pontio to chat about the Centre. We were meant to be discussing the agenda but got into the content immediately and luckily the technician has the sense to start recording so the whole thing will be more spontaneous when we release it next week. One of the issues that came up several times, and which will be a significant area of CfAC’s work is that of measurement. As the Hugh’s Gaping Void cartoon implies, measurement too often becomes and end in itself and in the modern organisation it can be an obsession. This despite the evidence showing that there are significant problems. In presentations I often use two quotes:

When a measure becomes a target, it ceases to be a good measure
Strathern (variation on Goodhart’s Law)
There is no question that in virtually all circumstances in which people are doing things in order to get rewards, tangible rewards undermine intrinsic motivation
New Scientist. 9th April 2011 pp 40-43

Now a mistake a lot of people make when they see those quotes is to assume that measurement of itself is wrong. Therein lies foolishness and the perpetuation of foolishness. Goodhart originally argument was that statistical instruments should not be used for policy purposes, he was not opposed to measurement, but to making a measurement system a goal or target. Equally if you don’t need intrinsic motivation then rewards linked by implication to explicit measurement systems may be a sensible intervention. They may also be an intervention of last resort when all else has failed. A good example of that is imposing quotas to overcome discrimination. Ideally it should not be necessary but sometimes it is the only instrument left to policy makers. Used it will have unintended consequences as do all interventions in the complex domain.

We also need to think about why measurement has become a modern obsession. I would like to posit three reasons:

  1. Most decision makers are distanced from the work of those they measure so the targets become a proxy for intimacy.
  2. Fear of blame (it’s not fear of failure guys) encourages measurement as it allows blame to be deflected.
  3. Excessive transparency means there is no space for failure and therefore no space for learning.

You can see all of these in managers and politicians alike. The trouble is that when things go wrong we increase the measures, increase the punishments and demand more transparency so we get a downwards spiral and trust is simply the first of many virtues to be thrown out with the bathwater. The other major issue is that targets become an entrained pattern of expectation and people cease to question. I’ve railed at the way in which teachers who inspire kids to learn are punished, while teachers who complete and execute learning plans are rewarded. But the measurement system has become the norm over the decades and it is now what people expect so they no longer challenge it. I could give other examples but you get the point (or maybe not!).

So what to do? Well, a key lesson (and it took me years to learn it) is that in human systems you start from where people are, rather than where you want them to be. I’ve also realised over the years that this is more ethical as well as being practical so that is no bad thing. I should quickly qualify that by the way to say that in extremis you have to take another approach. Given that measurement is the current orthodoxy then the most effective way to change is to create measurement systems that are authentic to the thing being measured. I’ve been playing with this for some time since we started the quantitative approach to understanding the dispositional state of the present which became possible with SenseMaker®. For several years I played around with ideas of impact not outcome and so on but a lot of those words had already been taken and coopted (with good intent) to linear causal models of reality. Last year I realised that we needed new language and in one presentation a memory returned from the sixth form when those of us doing pure and applied mathematics as separate subjects realised that our Physics teacher had no knowledge of vector mathematics and thought the parallelogram of forces was the bee’s knees. We were as cruel to him as only 17 year old’s can be and I sort of regret that now. But vectors seemed right so I tried it out and it worked in getting people to think differently. Vector measures allow us to determine the direction and speed of travel from the present and with SenseMaker® and fitness landscapes we cannot only measure that we can create real time feedback loops and targets that adjust to context.

So the moral of this story? Outcome targets for the complicated domain of Cynefin; vector targets for the complex. We are also going to link this to one of our research programmes that will rethink nudge economics in terms of vectors and that I really excites me.