Dave Snowden

The banality of measurement

RSS Feed

One of the insights which can come with age/experience is that the vast majority of people have good intentions or at least do not intend evil. Most are caught up in a flow of events in which the needs and expectations of the present dominate actions. Evil often results and may then be compounded by personal choice, but how far that choice is deliberative or personal remains open to question. If you look at the pictures of any acknowledge tyrant of modern times when they were a child do you see the evil that came after in their eyes? The corollary of that is that Good and greatness are similarly circumstantial in nature; personal choice can amplify, direct or dampen what is an emergent property of multiple interactions over time. For all of us there are bifurcation points at which we can make choices but few are saintly enough, or evil enough to break a wider pattern of social interaction and the dominant tropes of their society. I’m going to look at this issue more specifically in the context of leadership tomorrow when I have finished reflecting on yesterday’s speech by Obama. Today I want to return to something I have addressed in many a speech and post over time, namely the nature of measurement.

One reason this is on my mind is that it came up in a conversation I recorded yesterday at Bangor University. I’m up all this week (bar a side trip to London today) working on the set up of the new Centre for Applied Complexity (CfAC). John (Dean of my host department Psychology) and Andy (Design guru, Pontio, innovation etc) got to gather in the newly opened Pontio to chat about the Centre. We were meant to be discussing the agenda but got into the content immediately and luckily the technician has the sense to start recording so the whole thing will be more spontaneous when we release it next week. One of the issues that came up several times, and which will be a significant area of CfAC’s work is that of measurement. As the Hugh’s Gaping Void cartoon implies, measurement too often becomes and end in itself and in the modern organisation it can be an obsession. This despite the evidence showing that there are significant problems. In presentations I often use two quotes:

When a measure becomes a target, it ceases to be a good measure
Strathern (variation on Goodhart’s Law)
There is no question that in virtually all circumstances in which people are doing things in order to get rewards, tangible rewards undermine intrinsic motivation
New Scientist. 9th April 2011 pp 40-43

Now a mistake a lot of people make when they see those quotes is to assume that measurement of itself is wrong. Therein lies foolishness and the perpetuation of foolishness. Goodhart originally argument was that statistical instruments should not be used for policy purposes, he was not opposed to measurement, but to making a measurement system a goal or target. Equally if you don’t need intrinsic motivation then rewards linked by implication to explicit measurement systems may be a sensible intervention. They may also be an intervention of last resort when all else has failed. A good example of that is imposing quotas to overcome discrimination. Ideally it should not be necessary but sometimes it is the only instrument left to policy makers. Used it will have unintended consequences as do all interventions in the complex domain.

We also need to think about why measurement has become a modern obsession. I would like to posit three reasons:

  1. Most decision makers are distanced from the work of those they measure so the targets become a proxy for intimacy.
  2. Fear of blame (it’s not fear of failure guys) encourages measurement as it allows blame to be deflected.
  3. Excessive transparency means there is no space for failure and therefore no space for learning.

You can see all of these in managers and politicians alike. The trouble is that when things go wrong we increase the measures, increase the punishments and demand more transparency so we get a downwards spiral and trust is simply the first of many virtues to be thrown out with the bathwater. The other major issue is that targets become an entrained pattern of expectation and people cease to question. I’ve railed at the way in which teachers who inspire kids to learn are punished, while teachers who complete and execute learning plans are rewarded. But the measurement system has become the norm over the decades and it is now what people expect so they no longer challenge it. I could give other examples but you get the point (or maybe not!).

So what to do? Well, a key lesson (and it took me years to learn it) is that in human systems you start from where people are, rather than where you want them to be. I’ve also realised over the years that this is more ethical as well as being practical so that is no bad thing. I should quickly qualify that by the way to say that in extremis you have to take another approach. Given that measurement is the current orthodoxy then the most effective way to change is to create measurement systems that are authentic to the thing being measured. I’ve been playing with this for some time since we started the quantitative approach to understanding the dispositional state of the present which became possible with SenseMaker®. For several years I played around with ideas of impact not outcome and so on but a lot of those words had already been taken and coopted (with good intent) to linear causal models of reality. Last year I realised that we needed new language and in one presentation a memory returned from the sixth form when those of us doing pure and applied mathematics as separate subjects realised that our Physics teacher had no knowledge of vector mathematics and thought the parallelogram of forces was the bee’s knees. We were as cruel to him as only 17 year old’s can be and I sort of regret that now. But vectors seemed right so I tried it out and it worked in getting people to think differently. Vector measures allow us to determine the direction and speed of travel from the present and with SenseMaker® and fitness landscapes we cannot only measure that we can create real time feedback loops and targets that adjust to context.

So the moral of this story? Outcome targets for the complicated domain of Cynefin; vector targets for the complex. We are also going to link this to one of our research programmes that will rethink nudge economics in terms of vectors and that I really excites me.

  • Anthony Green

    Hi Dave
    Regarding “Fear of blame” do you see any corollary with Chris Matts’ thinking on “Risk Aversion” vs “Risk Management” cultures? https://theitriskmanager.wordpress.com/2015/02/21/risk-averse-risk-managed/

    • Dave Snowden

      I’m less sure. Risk is always there so Chris’s distinction is useful. Blame is something that is legitimate or not and may not be needed. But I see what you are saying

    • http://www.Broadleaf.com.au Stephen Grey

      Two points

      [1] Just as taking on board the implications of complexity requires us to turn our approach to the world inside out, looking at it as a whole rather than from within the cogs and levers we used to assume were there to be controlled, there is a useful shift to be made from the “risk is bad” standpoint to an appreciation of uncertainty as inevitable, may work in our favour or against us, and looking at how we want to proceed in this setting. Matts’ article seems to be lodged in the “risk is bad” view of the world, which has limited utility.

      [2] Fear of blame and other motivations around monitoring and reporting are discussed in a useful paper that I mentioned and summarised in a recent LinkedIn Cognitive Edge Group post. “The Pitfalls of Project Status Reporting” by Mark Keil, H. Jeff Smith, Charalambos L. Iacovou and Ronald L. Thompson http://sloanreview.mit.edu/article/the-pitfalls-of-project-status-reporting/ or http://www.amazon.com.au/The-Pitfalls-Project-Status-Reporting-ebook/dp/B00JI8QE4G Nothing new but this is a beast that won’t die.

      • Dave Snowden

        One of the new CfAC initiatives is to create an approach to Project Management that will include automatic reporting from field records – early triggers rather than requirement to report deviation from the plan

        • http://www.Broadleaf.com.au Stephen Grey

          I am very interested in this and have given the matter a significant amount of thought based on over 20 years working with very large projects in a variety of sectors and less time spent on SenseMaker projects. If I can contribute in any way, please let me know.

          There is clear evidence that about half the problems that are perceived as having affected project success were known about but not treated. That is an obvious low hanging fruit target.

          There are structural issues that SenseMaker could circumvent as well, concerning motivations to speak out and confidence that adverse reports are valid.

          There are case studies indicating that confidence in the quality of reports is a major issue delaying the collation and transmission of reports. See also http://sloanreview.mit.edu/article/the-pitfalls-of-project-status-reporting/ or http://www.amazon.com.au/The-Pitfalls-Project-Status-Reporting-ebook/dp/B00JI8QE4G for a dispasionate assessment of the subject.

          • Dave Snowden

            Watch this blog. I will announce the link to the CfAC site as soon as it is up

          • http://www.Broadleaf.com.au Stephen Grey

            I meant to refer to the source of my assertion about 50% of the sources of disruption of projects being missed. It is:

            Kutsch, Elmar. “Bridging the Risk Gap: The Failure of Risk Management in Information Systems Projects.” Research Technology Management 57.2 (2014).

            The paper used to be available online but seems to have disappeared, which is a pity as it provides a lot of interesting insights. I think it can be requested from the lead author via ResearchGate.

            My summary of the stats in the paper is that, of the problems that affected major technology development projects

            2-3% were deemed unknowable, a designation worth treating with caution
            6% were overlooked because no one involved had the expertise required to spot them
            17% were identified but not addressed because the team lacked confidence in their ability to analyse them
            21% were analysed and deemed worthy of attention but still not addressed because (a) to mention them would undermine confidence in the project, (b) the management were unwilling to commit resources at the time to something that would not have an effect until many months or years in the future, or (c) the project manager lacked authority to act (and presumably an effective escalation mechanism)

            That leaves only about half the issues that would eventually cause problems actually being considered within whatever project systems were at work.

            It is hard to believe that signs of all these difficulties were not visible in advance, whether formal risk management systems had identified them or not. That is leaving aside questions of timeliness and delay in realising that something interesting was developing.

          • Dave Snowden

            Thanks that is useful …

  • Cjm

    Not all that can be counted counts, and not all that counts can be counted!

  • Gaetano Mazzanti

    “in human systems you start from where people are, rather than where you want them to be”
    uh, this is so unSAFe :) and it even sounds Kanbanish!

    • Dave Snowden

      unSAFe good, Kanbanish, not bad but …

  • Chris Corrigan

    Great. I’ve used the concept of vectors for a decade or more talking about directionality and targets in endeavours such as “community building.” I remember one time watching one of the Lord of the Rings movies with my wife Caitlin and seeing orcs win a small battle against the humans. The orc captain stood up on a rock and declared “The time of humans is over!! The time of the orcs has begun!!”

    Caitlin turned to me and said “What are the orcs going to do with all that time? Open cappuccino bars?”

    I use that story a lot to question people who are working for complex outcomes. So you believe that there is a finish line to your work? If no, then you are in something complex and you need a vector and not a target.

    Here are a couple more posts about that relating to facilitating:



    • Dave Snowden

      Thanks Chris – I’m planning to work on a series of posts on facilitation later in the year so those links are useful

    • http://www.Broadleaf.com.au Stephen Grey

      I really like the “vector, not a point” observation. It immediately breaks us out of several unhelpful modes of thought.

  • Andrew Wagner

    I suspect we can extend this thought to simple domains: in manufacturing we use a go/no-go gage to indicate if a hole is within tolerance. Binary, or discrete attribute data is totally appropriate in that context. Yet, red-yellow-green charts rarely tell us anything useful in complicated or complex domains.

    • Dave Snowden

      and may even be misleading?

      • Andrew Wagner

        My comment was a poor attempt at understatement. 😉
        Yes, I’ve heard them called “watermelon” metrics. Green on the outside. Red on the inside. More so in a culture with a lack or trust or a focus on blaming the messenger.

  • Barrett W Horne

    This is a quite helpful post. I really appreciate the concept of ‘vector targets’ in the complex space. (Thank you also to Chris for the links.)

    • Dave Snowden

      I’ll do my best to be really helpful next time :-)

      • Barrett W Horne