1445682855-featured.jpeg
Dave Snowden

Evidence?

RSS Feed

Somehow an image from the Salem Witch trials with a powerful image of patriarchy seems appropriate to continuing by discussion of what constitutes evidence in a complex system. Miller’s The Crucible was one of the first plays I ever saw as a teenage in the old Chester Gateway and it made a lasting impression on me. The evidence of of a corrupt and corrupting testimony overlaid onto existing prejudices and needs constituted proof enough to allow ‘strange work for a Christian girl to hang old women’ to become a terrible reality; but with the best of intentions.

Now I have no problem whatsoever with the concept of evidence based policy, in either the public or the private spheres. My issues are (i) that people too often confuse evidence with proof, and (ii) that what constitutes evidence is very different in complex systems to complicated ones. Given that all social systems have high degrees of complexity we therefore have a problem with any notion of evidence that assumes repeatability of cause and effect even within similar contexts let alone different ones. Human systems have levels of complexity that shift us beyond modelled patterns of behaviour. We learn from and interact with systems in multiple unmanageable ways. Check in on my post about cobra, butterfly and hawthorn effects if you want some very basic illustrations.

One we both understand and internalise the concept that most human systems are disposition not causal then it logically follows (as I suggested yesterday) that much of what is evidence will be an emergent property of interactions with and within the system over time; not some innate feature of the system that is constrained sufficiently to allow repetition. A randomised control trial is cool for testing drugs in some cases. The physiological reaction of our bodies to those drugs and to any invading agent is a level where there is little conscious control of social interaction. But when we try and move it to the social domain we trespass on both scientific and ethical issues. For system level impact we can dismiss RCT a priori while recognising that it may have utility for aspects of that system. In the development sector this is an increasingly problem given the demand for defined outcomes and the RCT obsession (which has many aspects of the witch trial) and the lack of effective critical opposition. It has become a norm through which funding is given as it gives the comfortable appearance of predictability and creates a lot of research and consultancy revenue. Sorry to be cynical there, but it needs to be said.

So what constitutes evidence within a complex system? Well one of the Centre for Applied Complexity research programmes which we identified yesterday in a great meeting at Bangor University will look specifically at this, and also aim to create some (sic) evidence. But for the moment there are some clear steers on what we should look for, or the process we should adopt to create sustainable practice with the right evidence base:

  1. We need to start with a dispositional map of the current situation: what aspects are stable;  where is there potential for phase shift or incremental change?   What can we change where we can monitor the impact of change over time?  We always start with how things are in complexity; yes a sense of direction for future travel but not a precise goal.
  2. What does natural science say is possible and impossible?  Within those boundaries what are the lowest energy costs of replication?  Can we change that by changing the fitness landscapes?
  3. What coherent actions can we take that might shift the system in the right direction (what is right is for another day).  Can we create safe-to-fail experiments to achieve that shift?
  4. Have we got an assemblage/strange attractor aspect of this system which indicates that a social construct now exists independently of its contractors?  How stable is it?
  5. What aspects of success (again a question for another day) are repeatable at what level of granularity?  What type of information exchange is necessary or possible between the grainsI?  How would we monitor the properties that emerge from those interactions?

And that is just for starters. At the moment this is a little academic so the next stage is to shift it to practice. But that will be practice informed by theory, not theory determined by conventional current practice or a dominant trope of belief.

  • Pingback: Evidence in the Complex domain – Some observations. | The Three Letter Moment()

  • John

    What constitutes evidence within a complex system? Being in Software Development I use this analogy; Gold! To produce pure gold, you have to take it through a process. You challenge your objects immediate state – by applying Heat to it! You can say it looks like gold and feels like gold (verify) – but until you apply heat in investigative and experimental actions to a certain limit (approx. 1000 deg.) you don’t know you have pure gold. Once you have explored to a relative limit that exposes assumptions and eliminated aspects of the unknown (validation) – you can state that this object is pure gold. Could this be a golden analogy complexity? Only time will tell!

    • Dave Snowden

      Its a novel idea, but it is testable in a repeatable way. The point in a CAS is that you don’t get repeatability

  • Ray MacNeil

    This piece struck me in the day more than most Dave. Must be my particular ranting today and what’s going on here in the moment.

    But you said it twice and so I would like to ‘challenge’ you to speak more about what the word ‘beneficial’ (a.k.a ‘right’ and ‘success’) means in the statement, ‘managing the emergence of beneficial coherence within attractors, within boundaries”.

    I am actually asked that question more and more and have my own thoughts, such as they are, but am eager to hear you explore that a little more. I think this is a spot where people indoctrinated into traditional strategic planning methods struggle (I need a goal!!). I think its also part of the conversation with the mass-self-organizing-as-a-response-to-complexity folks. Of course, what I mean is that work in human systems is, and should be, very much goal directed. And (as you’ve pointed out before) that’s commonly misunderstood in my experience.

    As for the gold analogy sent in by John, a little more explanation might be required for this non-mineralogical mind.:). While its not unusual, I’m missing something in the analogy.

    • Dave Snowden

      Vector based not goal based is key. Beneficial then means moving to the ‘adjacent possible’ on a fitness landscape – that can be determined by the community or by authority depending on context.

Top