Dave Snowden

Scaling in complex systems

RSS Feed

I wrote a series of posts about scaling over three year ago which in part addressed some of the perversities of methods focused on accreditation revenue to which I referred yesterday. They were important posts and sections will be dumped in Scrivener over the next months as I get ready for an intense booking writing week in a remote cottage on Ynys Môn this January. Naturally I have done a lot more thinking since then but the fundamentals remain: you don’t scale a complex system by aggregation or imitation but by decomposition to an optimal level of granularity followed by recombination (possibly exadaptive) in context: complexity is about how things connect far more than what the things are. I remember an argument about this in IBM days when I had finished working through a radical new organisational model in which the role of IBM would be to centre a network rather than own resource. It was rejected on the basis that scale was perceived in terms of size and number of reports, in turn linked to hierarchical and aggregative/reductionist ideology.

The so-called scaled frameworks in Agile made, and continue to make similar mistakes with varying degrees of absurdity. The worst simply aggregates everything that has ever been called Agile into a complicated flow diagram, reducing complexity to an over-constrained structure. This is scaling by dumbing down to a lowest common denominator approach; it gives an impression of order by constraining practice to the explicit. As such it mimics the errors made in BPR which were magnified in Sick Stigma to the point of doctrine. The net result of all reductionist approaches to process management was the emergence of high energy wastage and stress as humans worked around the system to make it work despite itself; effectiveness was sacrificed on the altar of superficial efficiency. Interestingly in twitter encounters with some advocates of the SAFe Borg yesterday this (the idea that you work around something to make it work) was put forward as a defence – good consultants/coaches cherry pick what they think will work and modify it in practice while maintaining the pretence of the over-simplification they think is necessary to sell to senior IT Management. I think the phrase here is trying to force a virtue out of a vice.

To illustrate the point let’s take organisational forms. We know that the behavioural characteristics of teams of five, fiveteen and one hundred and fifty are very different. You can’t take the practices and interdependencies of a small team and replicate them in a large team. In a small team a level of intimacy and knowledge of the characteristics of other team members allows smooth coupling and pre-emptive actions to mitigate faults and weakness. In a large team we become increasingly dependent on the explicit statements of qualifications and competences and less on the tacit understanding of capability in different contexts; compliance and conformity are privileged over anticipation and adaption. Taking a complex perspective we would create variable expectations of different identity clusters (people can belong to different identities) with management and monitoring of the nature of interactions and emergent stabilities that arise from those interactions. This allows a large system to grow very quickly and at times in novel and unexpected ways as a series of situated networks. Think of a Portuguese Man of War which is a symbiosis of different creatures which assemble in a specific context, or of Slime Moulds which again change their nature under conditions of starvation; in software Object Orientation and Micro-services allow novel combinations to emerge if properly architected. It’s not that we don’t have the models to use here, its just that we are stuck in engineering diagrams.

Alicia Juarrrero famously coined the phrase like bramble bushes in a thicket to describe a complex adaptive system and it is a title Kurtz and I used in this paper which represents early thinking on organisational design and networks. In a complex system you need to define your identity focal points and manage your connections, rather then get out the flow charting template and create excessive structure. Bramble bushes bear fruit, but also prick the unwary; structured flow diagrams are safe in the sense of the absence of novelty, the absence of risk and the absence of imagination.

  • David J C Morris

    Provocative article, as always Dave.

    Firstly, let me say that I am a practitioner not a consultant. I do not get paid to train people nor to configure processes and systems.

    I am a pragilist. I like what works, and what works differs for different contexts. While these days that is mostly some flavour of lean or agile; as you would expect from someone trained by you in Cynefin, there are other models that work better for other contexts.

    Sometimes what is appropriate is pockets of individual teams doing their own thing on products that don’t need to integrate or that manage integration through a series of micro-services and contracts. Sometimes it is a network of disparate teams who interface on as as-needed basis and can be trusted to coordinate and synchronize accordingly. Sometimes it is a multi-tiered model of teams within teams within teams.

    I like Nexus, I like LeSS, and I like SAFe. I have never succumbed to the temptation of saying that I had a favourite uncle or that I loved one child more than another. That’s the land of disharmony and false dychotemy. I choose not to live there.

    Underlying many such arguments against SAFe seems to be a thread that says; because SAFe is a framework of disparate practices, there is no guarantee that it will work. That is true. That is accepted by most SAFe practitioners. That is also built in. It is a framework. It is a starting point.

    While SAFe offers some metaphors to help management understand, it is based on sound agile principles and practices. Chief amongst them being transparency, inspection, and adaptation. Of the framework as well as the product.

    Even Scrum, that holy of holies to anti-SAFers, calls out that we can adapt Scrum once we’ve run with it for a while.

    And here’s what it boils down to. Most times I hear this argument, it is from people with competing frameworks. It is negative marketing.

    Ironically, it is what we used to do to distinguish agile from waterfall practices. Me included. Guilty as charged. Mia Culpa. Mia Maxima Culpa. But then I learned. From deeper understanding of the agile manifesto. From reading more broadly. From other practitioners. From meditation. And, from yourself too, Dave.

    Critically, it is against the very first value pair in the agile manifesto. It places adherence for or against a process above the respect and willingness to listen and collaborate with other practitioners.

    I expect that from Ken. I have said the same to him too. He has been negative about Dean and SAFe for years. Mostly because he was pushing what grew into Nexus. It’s not forgiveable, but it is expected.

    So it was not surprising to see that you now also have products in development. Disappointing, but not surprising.

    Perhaps, these days, we just do whatever it takes to generate an income. Perhaps I have misunderstood the intent and the action? Either way, as a pragilist, I await them with interest, of course.

    However, I don’t like the way discourse is going. It’s creating Trump-isms in the agile world, and it’s ugly. Could you please stop killing my new-age fluffy bunnies?

    From a fan (still).

    • Dave Snowden

      Sorry, but the idea that anything is good and everything should be accepted is not something i agree with. SAFe basically is trying to sweep anything remotely Agile into a training and certification framework. It started selling by using a pyramid selling scheme with no requirement for experience. Now it is established it is cynically trying to raise the cost of entry (and thereby its revenue). I constantly tell people that I am sure that good consultants can make the monstrosity work despite itself, but the overall construct is ontologically incompatible with Complexity.

      Now the classic response (and you use it to a degree) is to say that Scrum is just as bad, or that Ken just wants to make money. This is deeply cynical and wrong. Scrum is a technique not an over arching framework. Nexus looks (but I am still looking at it) as consistent with CAS and LeSS at least works incrementally. I think that Ken, like me, thinks SAFe is a betrayal of Agile and neither of us would seek forgiveness, or accept your attempt to explain that away as a rival commercial interest.

      When SAFe added in DevOps the cynicism was clear

      As to our offerings – we run training in a variety of things, we offer no certificates, no accreditation. We are under pressure to provide some type of quality control and I am looking at ways to do that. Anything we do will include an apprentice period under supervision and will not be linked to a training course. If we do anything it will relate to specific methods and tools not a whole field. You have very much mistaken the intent and action.

      SAFe is not a fluffy bunny, it is a highly structured near waterfall fox hiding in a rabbit skin

      • David J C Morris

        I suspected I had succumbed to a degree of mistakery regarding the products you were developing, so thank you for taking the time to respond.

        A couple of points:

        While I love the analogy of a fox in a rabbit skin, the fluffy bunnies to which I referred were not the frameworks, but rather the “respect and willingness to listen and collaborate with other practitioners”. I find that discourse around SAFe has become increasingly discourteous, involving people taking up positions. Hence my allusion to Trump-isms. It doesn’t feel right for people to be fighting one another like this in the a space that is intended to be collaborative. That is ultimately what prompted me to write.

        If I have conflated strength of feeling with commercial interest and if that truly has nothing to do with it, then that is my bad. However, it that is the case, then I struggle to comprehend the anger. No-one seems to get that emotional when they talk about what is wrong with waterfall, and that in many ways is the antipathy of the agile mindset.

        You talked about SAFe being a betrayal; perhaps that word is a hint to the source of the anger in some? If so, a betrayal of what exactly? Most people who are not taking sides see SAFe as part of the agile spectrum and are bemused by the strength of feeling and language.

        Trying to be mindful, I hope that I am not taking up a position myself. I don’t think SAFe is perfect. We don’t use it where I work (although there are shades of it in places). I like, use, and plunder a wide range of tools, frameworks, and techniques. SAFe has its faults, but then every framework does. If we’re honest no framework is perfect. All frameworks are models. “All models are wrong, but some are useful”. Right?

        I encourage and applaud disagreement. A good friend of mine has a motto “strong opinions, gently held”. It is only through the cut and thrust of debate and the application of intellect can we learn, improve, and change. This should be possible without taking up opposing extreme positions, without denigrating someone’s work. The casting of a group or a person as a betrayer starts to sound more like a religious position that is absent of intellect.

        For someone who has been an agile practitioner over 20 years, my recollection of the history of SAFe is that it started as a community effort. Dean shared ideas around his big picture on his blog. Lots of us gave feedback and suggestions. The big picture progressively evolved into SAF, and then SAFe. More and more people were interested in it, and the book, the training, and the consultancy grew from there. Many people have become successful following these steps without the accusations that Dean has to put up with.

        One the one hand, as a colleague of mine once said, “I don’t have a dog in this fight”. On the other hand, “the only thing necessary for the triumph of evil is for good men to do nothing”. It’s a quote. Of course, I am not saying that what Ken, you, or others is saying is evil, but then neither is what Dean is saying.

        • Dave Snowden

          I must admit I thought my post of the previous day was pretty clear on what I thought were principles of creating training and quality control. However happy to repeat and happy we have cleared that up.

          Also, to be clear, this post is about how you SCALE a complex system. I used SAFe as an illustration of how not to do it. When I post specifically on the commonalities of SAFe with an established pseudo-science (NLP) I may move this exchange there with a note.

          Otherwise I think I may not have emphasised enough that SAFe is not another method like SRUM but an attempt to define the whole field. So being open to multiple methods and frameworks in one thing, to have accept that one of those is attempting Borg like to absorb everything else into its compass is something completely different. Not only that SAFe was cynical (in the wrong sense of the word) in renaming things SCRUM-X etc. The latest dumping of DevOps into the sprawling engineering flow model just confirms it.

          No one should get wound up about waterfall, there is nothing wrong with it in many contexts. If SAFe was just some methods within the Agile family then it would be fine, but it is not one part of the spectrum it is trying to define the spectrum. Scrum is a method within the overall Agile compass, there are others such as Kanban, mystuff on unarticulated needs and many others. Not even LeSS tries to assimilate anything remotely connected with Agile. Neither did it start with a pyramid selling scheme. Yes there were a few people involved, but to encompass a show field you have to collaborate far more extensively than Dean did. SAFe was a brilliant marketing campaign, based on limited cases, playing to the two weak points in Agile: acceptance by IT management and the desire for training revenue. Now it has got market leadership it is spawning multiple releases, courses, letters after people’s names and is finally trying to introduce quality into its certifications – classic market tactics only raising the barrier to entry after you own a lot of the space. A betrayal is mild language.

          As to Trumpisms – nice try but while I have several GOP friends (including some very senior ones) that does not require me to be nice about Trump or (to use your words) be courteous to strong advocates or to avoid anger about many of the things he does.. I’m happy to respect that some people (but not all) simply want to get along with everyone and that well intentioned people can make anything works. Condemning the Borg does not mean condemning the original humanity that is attempting to avoid assimilation without realising the nature of the beast.

          • David J C Morris

            Is there anyone claiming that SAFe is the only framework and that all others have no place? If that was the case, I think a sense of betrayal and anger would be an understandable response. Even ridicule. I don’t see that, however. Such an argument feels a bit like a false strawman.

            The packaging of so many techniques and methods into one open-source framework could be viewed as helpful, but I can also see that it could look avaricious and borg-like. Not sure that was ever the intention, but this has given me a glimpse into that point-of-view.

            Thank you for entertaining the discussion. While I am prepared to defend someone I think is being unfairly maligned; I can also recognise when I have fallen into a maelstrom that is disappearing into itself. Thank you for seeking to protect those you see as unknowingly being assimilated. A Luta Continua. Kia Kaha.

          • Dave Snowden

            Really? SAFe encompasses all other methods and is positioning itself as such by its nature. A recent trend is to attempt the ‘let many flowers flourish’ approach, but that was only adopted after it had achieved apex predator status in the field, not while it was starting up. My take is that having it got it wrong once, Dean learn well and marketing brilliantly. So betrayal, a moderated and resigned anger and ridicule are long over due. The commonalities with Six Sigma and, in a different field and different age, NLP are scary.

          • David J C Morris

            No idea what you could have against natural language programming. Only kidding. It’s been fun. Thanks again.

          • http://agilepainrelief.com/notesfromatooluser/ Mark Levison

            @davidjcmorris:disqus I thin the other Dave’s key point is beware of anything that is prescriptive, in the context of Agile. SAFe is not marketed as a grab bag or open source toolset. Its marketed as a whole thing, people buy the picture not the parts. In addition I’m not convinced the parts are practiced correctly when done the SAFe way.

            I’ve seen enough adoptions of SAFe to say that they’re not practicing Scrum and I have deep experience with Scrum.

            Some SAFe adoptions may transform themselves to a better place but doesn’t happen by default.

          • David J C Morris

            Agreed. SAFe itself does invert Scrum values. For example, the model deliberately reduces the degree of autonomy of delivery teams. It does this for a reason (although many disagree with the reasoning). Any SAFe protagonist denying that is in denial.

            This is why I try to maintain a degree of agnosticism. Just as in some circumstances a plan-driven waterfall approach is still appropriate, so too in some organisation the SAFe scaled model is appropriate.

            Any form of agile practice is a phase-shift from the analytic to the synergistic mindset (ref. the Marshall model). SAFe is definitely a blend, retaining more of the analytic mindset at the senior level.

            Organisations that still rely on a high degree of control, feel safer with tools like kanban and a tiered / staged framework; although this is antithetical to the collaborative and cuitivating approaches many agile practitioners have (ref. the Schneider model). More coaching and change is needed at the senior level on those organisations, if they want to right-shift to more synergistic modes of working. Not all will be ready for that. SAFe suits them fine.

            Organisations that are ready to move further along that continuum, are ready to give more autonomy to their delivery teams, and enjoy the benefits of bottom-up innovation. That’s awesome. We should definitely have more of that.

            That’s where I want to take the organisation I work in; so I consider an enterprise agile framework as a key enabler for that. A mechanism that we can use to continually evolve towards that end. As I said in an earlier response, while we don’t use SAFe, if you squint it does look a little like it in places. That’s in part to deal with where we are right now. There’s nothing inherently wrong with this. I enjoy working in an organisation this size that has the support and drive from senior leadership to be taking them on this journey, as well as from practitioners.

            Working in similar organisations, I have led bottom-up agile adoption, but this risks achieving little more than putting an F1 engine in the tractor.

            Smaller / younger organisations who are already in the right mindset, don’t take much nudging to adopt the right practices and behaviours. I’ve helped a few of them get further along the continuum and enjoyed that.

            As to what I am doing now; sometimes I liken it to shifting not just one container ship but a whole fleet of them. Should we deliberately scuttle them all so that we have to take to smaller / faster powered boats. We could, but we’d lose all the value we were carrying as well and I’m pretty sure the organisation would not survive that. I came here because I believe in our mission, the way we treat our associates, and the hard-yards we are doing to change.

            Something of a long response, which reflects the thought I have given this over the past week. As I also said in an earlier response, mixing my Portuguese and Maori: a luta continua, kia kaha. The struggle continues, stay strong.

          • davesnowden

            I think if SAFe had said they were producing a set of techniques that allowed a more agile approach to Waterfall and structured management no one would have had a problem. But the claims were to the field as a whole and that is problematic. I think full adoption of SAFe is retrogressive not progressive – that is to say it is in no way a Trojan Horse.
            The approach you seem to be adopting which is to gradually shift people from where they are is something anyone with a. Background in complexity would agree with. But presenting an all singing, all dancing, high cost construct is a very different matter.
            Prof Dave Snowden
            Cynefin Centre & Cognitive Edge
            Sent from my iPad Pro

          • Sjoerd Kessels

            I get a bid sad whenever I hear or read these hostile comments on SAFe. I also must admit I am slowly loosing my patience with these people at the same time. In my experience, whenever I get a chance to really discuss the topic with them, it turns out they actually don’t know much about SAFe. They haven’t attended a course, never were responsible for using it in a real-world environment. They don’t seem to be very interested in learning from people that have actually used it (like I see in this thread). People just don’t seem to get past the big picture…and then judge. They interpret things without taking much effort to check their interpretations. We get into flame wars between frameworks, ideas, and theories which doesn’t help anyone. We are supposed to help organizations improve, not having contests who has the biggest…

            I attended a LeSS training some time ago. When the trainer started to bash SAFe (I have a strong opinion on what it means if you need to sell your framework by bashing others) he quickly found out there were plenty of SAFe practitioners in the room who corrected him on almost any interpretation he had on SAFe practices. Luckily he was pretty open minded. Although I don’t think his opinion was completely changed, he did admit he needed to study more because he was obviously wrongly interpreting things. In the end we did agree that both SAFe and LeSS try to accomplish the same things, using the same underlying bodies of knowledge, in a manner that really is not that different at all. As a matter of fact I regularly mix practices from both.

            A comparable experience was even more recent when I attended the most recent Cynefin training in Amsterdam and at some point the term ‘popular Systems Thinking’ came up which was described in a pretty demeaning way on Systems Thinking. There were several Systems Thinkers in the group that attempted to get the trainers to be more specific. Unfortunately the trainers did not get much further then ‘It is Dave’s opinion’; ‘I must admit I am not a expert in this field, but Dave feels…’ (why do you teach something if you can’t explain it for yourself); and ‘well, we actually mean anything that is over-popularized’. The latter got the comment ‘Then why don’t you write “popular anything” in the slide and not “popular systems thinking”? While the group clearly was looking for ways to combine things, I got the impression Cognitive Edge (not the trainers personal opinion) officially wanted to favor their own thinking over that of others.

            Another example: I overheard a discussion on a session that was presented by Jeff Sutherland during a recent Agile leadership conference in Amsterdam. Jeff was explaining Nexus and, yes, bashing SAFe at the same time. He explained the concept of chaining Scrum of Scrums to scale. The attendees that discussed the session were asking ‘why do you think SAFe is hierarchical and chaining Scrum of Scrums is not?’

            It is funny and sad that I see a lot of SAFe practitioners with open minds towards other ideas, attending courses, participating in discussion, etc…but I miss the same open mind the other way around. It is the same in this thread: Dave J C Morris is very subtle and nuanced in saying ‘Aren’t we a little black and white in this rage against SAFe?’ He doesn’t seem to be trying to convince you, he just seems (my interpretation) to try to make a case for a little open mind.

            What worries me is the ease at which you call SAFe non-agile without naming any specific practice. So let me take the liberty to name two I regularly hear from people standing high on mount stupid:

            “SAFe is actually waterfall”. When I ask to be more specific they usually mean the concept of the Program Increment which entails 4-6 sprints, and starts with a Big Room Planning event. And yes, you can make that into a waterfall process. I have actually seen that happen, but I have seen the same happening with plain vanilla Scrum as well. It is not waterfall in my opinion: There is no big upfront plan that states in detail what is supposed to be delivered at the end of the PI. To the contrary, we try to deliver as fast and often as possible and responsible, maybe even every day. Yes, you plan at the PI Planning, but the idea is: Instead of having people who want the work done make the plans for the people who actually do the work, let’s bring them all together in one room. And let’s try to put a stake in the ground of where we like to be 2-3 months from now and what seems feasible. It is not detailed, it is more in terms of intent, at a much higher lever of granularity then the User Stories teams work with in sprints. Let the Scrum teams decide on the ‘how’ themselves during the actual Scrum sprints. So the goals are incrementally implemented during regular two week sprints. It is ok if the plan changes along the way. It is ok if teams come up with better ways to reach the desired goals. It is ok to not implement everything anticipated because we get feedback along the way. It is nok to replan during the PI. It is ok to change priorities if that really makes sense. There is no detailed big plan upfront, there is no big delivery at the end. There just isn’t, even if that is what you want to see in the picture.
            One comment we got was “we created a plan in two days that until now costs us months of arguing and back and forth discussions”. And yes there is a plan then, but we don’t expect that everything goes according plan. It is the same as in any agile approach: it is not so much the plan itself as the activity of (continuous) planning that matters. How is that not agile?

            Portfolio management and budgetting is another topic I hear a lot of comments about. But what SAFe is trying to do here is: Let’s try to get out of the portfolio of projects where the person with the loudest voice gets priority. Let’s not let project managers claim resources for the sole benefit of his own project, even if it hurts the company as a whole. Let’s lose the concept of projects with conflicting stakeholders, replace them with business opportunities that we split up into smaller parts. We then put all these parts in one common backlog and prioritize the backlog in a way that tries to assess what would bring the most value to the company and do that first. How is that not agile?

            I am not saying there is nothing wrong with SAFe. I am actually quite critical on parts. I wish they had never added a fourth layer to SAFe 4.0, and we sometimes joke that hopefully SAFe 5.0 will not have… 5 layers…But I feel I am entitled to be critical because I actually work with it and know it in-depth.

            I agree with Mark Levison that in many cases SAFe is used in another way then is described by David J C Morris. But as Henry Kniberg commented on SAFe: “Tools are neither good nor evil. It depends on what you do with it”. Scrum is widely misused. I don’t see anybody saying that therefore Scrum is evil. And I am sure people misinterpret your work and use it in a way that hurts.

            Yes, the big picture is…big. And potentially confusing. And it has a danger of misinterpretation. There is a responsibility for Scaled Agile Inc there. But there is also a responsibility for people that read the big picture to suppress your first interpretations and talk to people who can help clarify things.

            You know what: We have actually drawn the SAFe picture sidewards sometimes (left to right), just to avoid people from immediately jumping to the conclusion that it is hierarchical, dismiss it, and stop listening. And the amazing thing is: that actually works…

          • http://agilepainrelief.com/notesfromatooluser/ Mark Levison

            David – I think your use of SAFe might well be effective. It is not how I see SAFe used on most occasions. I’ve seen it used as the end point/final stage as the goal. I don’t think there is a goal state, just an organization that evolves. Organizations that evolve seem interesting to me – whatever their model (SAFe, LeSS, Nexus, Mark Levison’s model: https://agilepainrelief.com/notesfromatooluser/2017/01/beyond-scrum-blog-series.html). Organizations that don’t evolve rapidly will die. We’re seeing this more and more (Blockbuster, Sears Canada, …). – Mark

  • Matt

    “In a complex system you need to define your identity”.
    I found especially this one a very interesting point. In my opinion, nowadays looking at successful companies we can observe a strong, genuine identity. These companies have their own culture (growth organically over time) not based on models or scaling frameworks. Times are changing but we aren’t avoiding the same mistakes of the past: I think about what American managers used to do when they were visiting Japanese factories when Lean and Toyota methods were popular. Toyota had a well defined identity and American managers during their “Kaizen tours” were obsessed by practices instead principles. Nowadays is pretty the same, some people are focused on scaling approaches (or other cultures’ models – i.e. Spotify or Netflix) acting by copying and following scaling dogmas. In these cases I can’t see place for identity or novelty. Identity has a fundamental role here.

    • Dave Snowden

      Well to be strictly accurate I said ‘identity focus points’ namely that an organisation has a fractal relationship over multiple contextual identities which are coherent but not conformed. But that supports what you say

      • Matt

        Thanks for the clarification, then considering this I was wondering about the rituals’ cognitive function. Are rituals themselves a good way to provide coherence between these multiple contextual identities? I’m saying this because what suddenly comes in mind to me are routines, dogmas and mere practices imposed by over-constrained structures that are pretty far from rituality and seems to lead to conformity. Or am I misunderstanding the two terms: coherence-conformity?

        • Dave Snowden

          Well ritual can trigger an identity shift – aligning with role if we invest enough in training. As you suggest (I think) we can also do that by disrupting established patterns of behaviour. The idea of creating liminal states is also important here – I’ve only just started on that with Cynefin. Coherence is as you suggest opposed to conformity, or at least conformity is an extreme state of coherence that is only sustainable in highly stable situations.

          • Matt

            Clear to me, thanks again :)

  • Luca Minudel

    So far I’ve found two interesting ideas on scaling complex systems are
    – the approach with patches from Kauffman
    – and this paper that contains interesting ideas on how to take into account the social context
    ‘Change in complex adaptive systems : a review of concepts, theory and
    approaches for tackling ‘wicked’ problems in achieving sustainable rural
    water services’

    What authors and previous works are you considering when you look at scaling complex socio-technical systems?


    • Dave Snowden

      I’ve used kauffman, but in general I’m working from first principles here

      • Luca Minudel

        looking forward to read more about it from you.