comment 0

A shift in perception: exploring ways to work with complexity in international development

I really enjoyed to go back this week to this paper from 2016 by ODI’s Anne Buffardi: When Theory Meets Reality.

Anne Buffardi, exploring ways to work with complexity

I went back to Anne’s paper because I am doing some work on defining some principle the can help design and set up monitoring, evaluation and learning (MEL) systems for development project/programmes that tackle complex social and governance problems.

I have been quite involved over the last few years with the discussions in various communities: PDIA, Doing Development Differently, adaptive programming, and to some extent thinking and working politically.

There are a lot of discussions and conversation about principles that can help shift the way development initiatives are designed, managed, and implemented (PDIA is a bit different as in addition to principles it proposes also concrete tools to manage uncertainty and complexity) . These principles speak to the shift of perception needed in the international development community to move from a mindset that sees problems and possible solutions in isolation from the context and systems that enable them to start dancing with the systems in which they operate.

This shift is important and necessary, but translating these principles into practical actions and ways of working is difficult. Anne mentions some of these challenges in her Methods Lab paper.

I have the same challenge in my work on problem-driven political economy and on MEL systems design. I am on board with the principles. I agree and believe in them but then struggle to apply them to, for example, coming up with a concrete outcome indicators that speak to the complexity and uncertainty that the project is trying to influence.

Anyhow, the journey continues and the paper Anne has written in 2016 has many insights that resonate with me. I am pasting below the key messages of Anne’s paper. Go online and download it. It is a great read.

  • Over the last 50 years ,repeated calls for adaptive learning in development suggests two things: many practitioners are working in complex situations that may benefit from flexible approaches, and such approaches can be difficult to apply in practice.
  • Complexity thinking can offer useful recommendations on how to take advantage of distributed capacities, joint interpretation of problems and learning through experimentation in complex development programmes.
  • However, these recommendations rely on underlying assumptions about relationships, power and flexibility that may not hold true in practice, particularly for programmes operating in a risk averse, results-driven environment.
  • This paper poses guiding questions to assess the fit and feasibility of integrating complexity- informed practices into development programmes.

Here the extract from the conclusions of the paper:

“Complexity thinking can offer useful insights on how to approach situations and challenges faced by development practitioners. However, these recommendations rely upon underlying assumptions about relationships, power and flexibility that may not always hold true in practice, where organisations have established internal and external ways of working. By posing guiding questions to assess the fit and feasibility of integrating complexity-informed practices, this paper aims to help development programmes identify which dimensions of complexity are most appropriate and realistic to address; and, how they can work within their operating constraints to take advantage of distributed capacities, joint interpretation of problems and experiential learning.

Classifying which practices are more and less feasible enables decision-making and management styles to be explicit, rather than couched in aspirational rhetoric about sharing and collaboration, learning and adaptation, which is not realistic in practice. If undertaken before programme implementation, it can help identify decision-making and communication mechanisms that can facilitate interaction among dispersed actors, create incentive structures that reward learning as well as delivery, and allocate sufficient time and resources for data collection, analysis and interpretation to enable monitoring to inform decision-making.”

Photo credit:  Unsplash

comment 0

The 72.000.000 question

The US presidential election, our digital future, and a question for the friends at ODI’s Digital Societies programme

A week has passed since Joe Biden and Kamala Harris won the US presidential election. Donald Trump has not yet conceded and has launched a flurry of probably ineffectual court cases. Votes are being recounted in Georgia but they are not going to change the outcome.

I listened to many commentaries since the election: NYT’s The Daily, the Guardian’s Politics Weekly and Today in Focus, Talking Politics with David Runciman. They all agree that the support to Donald Tump has not changed since 2016, it has even increased. On the other hand, a huge number of Democrats have cast their vote for Biden and Harris which has tipped the election in their favour. That shows also in the popular vote where Biden has received ca. 77.000.000 votes vs. the 72.000.000 for Trump.

During the last few days I found myself thinking about those 72.000.000 people who voted Trump. The large part of those voters are older, white, non-college educated, and living smaller towns and rural areas. They are angry. They believe that the economic and political system has let them down. Their factories have closed. Their jobs have gone overseas or have been taken by immigrants. Trump listens to them and speaks to their anger and fears.

The economy of today (and tomorrow) is marked by an increase in productivity and a decrease in the demand for labour. This trend is pushed by digital technologies and automation and will only accelerate during this century. So, what will happen to those 72.0000.000? Early retirements for tens of millions? Not likely. Tens of millions of bullshit jobs, in the word of the late Prof. David Graeber? Probably.

If this is the trend, are we going to see more populist and right wing leaders being voted in office as hundreds of million of citizen are left stranded by the digital revolution around the world?

I realise this is quite a bleak view of the future, but that number, the 72.000.000, left me with questions about our democratic future which are not simple to answer.

So, I turn to the friends at ODI’s Digital Societies programme: what do you think?

Photo credit: Trump rally in Wildwood NJ by David Todd McCarty on Unsplash.

comment 0

From Cold Data to Warm Data

Warm Data are contextual and relational information about complex systems.In other words, warm data involve transcontextual information about the interrelationships that integrate a complex system, as well as interwoven complex systems.
Warm Data Lab

I wrote to a colleague yesterday. During a meeting he had asked about a research paper he had written with others. He was rightly worries about making general claims without making explicit the limitations of the work.

His question remained with me. Yesterday evening I spoke about it with my wife at the end of our dinner. About six months ago, she has participated in one of the Warm Data Lab trainings organised by Nora Bateson. The points she made during our conversation were inspired by the training she did with Nora.

Every piece of research has limitations. Every piece of research is like a sketch of a tiny part of the complex reality in which we live. As a sketch, it can only give a glimpse into the reality and systems that we observe and live in. These sketches help draw (or begin to draw) maps of reality, but, as Nora Bateson says, maps are not the territory.

Research findings are just data and information, they are not knowledge. As such they are cold data. To gain meaning and influence the systems they need to become warm data. To become warm data they need to become alive as part of the conversations with the people who are affected by the problems the research has identified. They have to become almost like the fuel of dialogues and of new links and relationships between individuals. Warm data help surface the hidden knowledge that people affected by the problems that we might research. Warm data can change systems because they become part for the web od co-developments of those systems and are part of our collective ability to perceive, discuss, and try to address complex issues. 

The journey continues.

Photo credit:  Sveta Fedarava on Unsplash

comment 0

Three questions about MEL for projects dealing with complexity

Last week I wrote to the Peregrine Discussion Group on Better Evaluation with a couple of questions on M&E or MEL of initiates that try ton address complex social and development problems. I have received many very interesting replies which i am posting below together with my questions.

My questions

1) What do you think are the key principles that we need to have in front of us when designing MEL systems for complex and adaptive (and uncertain) projects and programmes?

2) Do we need a different MEL system and tools?

3) What do you suggest I should read on this topic and question?

The answers

Russel Gasser

1) What do you think are the key principles that we need to have in front of us when designing MEL systems for complex and adaptive (and uncertain) projects and programmes?

Complexity is not cleanly separated from and unrelated to the causal and chaotic domains, the boundaries are not clear lines much of the time, and trying to put in clean separators between what is complex and what is not, is itself a causal/linear approach to a complex issue. Trying to define a fixed set of MEL principles for the complex domain is a causal/linear approach that is necessarily doomed to failure,  both in terms that (a) it assumes that it is possible to predict what will be needed and also (b) it is based on an assumption that a finite range of methods can cover every case.  Both of these assumptions are false in the complex domain – that failure of these assumptions is one way to define what complexity actually is, and separate it from “complicated”, “difficult to understand” and “nothing like what I have experienced before”.  Neither of the first two is complex (other than in everyday conversational use of “complex”) and the third may be complex but actually describes the previous experience of the person, not the complexity of the situation. Causality is a one-way street in the complex domain – it can only be seen with hindsight. If it is truly “complex” then measurement of predicted results can only really measure the degree of luck or skill at prediction in guessing the outcome, not the value or success of the intervention.  Perhaps the most important question to answer for the real world is: How can we create the link from funders who are locked in to the linear/causal model, to providing funding for and using an evaluation that is based on complexity ?  Until you get the complex domain evaluation (or MEL) funded, and until there is the possibility that organizations change what they are doing as a result of evaluation in the complex domain, then it is academic research that lacks clear application. Nothing wrong with academic research, but this forum isn’t likely to be the best place to address it.

2) Do we need a different MEL system and tools?

MEL is not a single process or system.  The toolkit is already huge. Some of the processes and tools are suitable for use in the complex domain, some are not. Answering this question has the same issues as mentioned above:  (a) it assumes that it is possible to predict what will be needed and also (b) it is based on an assumption that a finite range of methods can cover every case. A better approach might be to ask: “What do we need to know and understand in order to be able to decide which tools and methods are likely to be useful in the specific complex intervention we are looking at?”  The specific nature of the complex domain means that we can only really consider one intervention at a time.A really important feature of complexity is that solutions are NOT transferable.  The MEL that works on one complex intervention is unlikely to be useful without changes for other complex interventions (that is the nature of complexity) and we can be certain that it will not be useful for all possible interventions in the complex domain. Monitoring is about the measurement of what is happening. There may be no reason to significantly change this from previous practice, beyond being far more obsessive about knowing what is going on right now and being able to respond quickly to build on success and damp-down failure.  If monitoring is not providing useful information in the causal domain then that won’t change when moving to the complex domain.===Evaluation in the complex domain can’t ask “did we achieve what we expected to achieve?” as that is largely meaningless.  But evaluation can still be any one of a range of approaches, and parts of the OECD DAC are still useful (e.g. “unforeseen consequences” in impacts).  Evaluation in the complex domain can usefully ask questions about what changes resulted from what aspects of the intervention, what was cost-effective and what was a waste of resources, how the intervention approach could be improved, etc.  Techniques like Outcome Harvesting and various story-telling techniques that are capable of identifying what really made a difference, what changed as a result of any intervention, and for whom, may be able to capture outcomes and impacts if scope and focus are correctly set. Learning is still much the same, it involves many of the same core skills for people to appropriate new information, internalize analysis and consequences, reinforcement to embed the knowledge, and a willingness to change beliefs and practices in the light of new information and experience.  If a causal/linear system is not supporting learning and producing new insights and approaches within an organization then transferring it to the complex domain won’t change the lack of learning. Selecting who might learn what, and for what purpose, needs even more detailed attention as complexity increases. Politics and social interaction are examples of interactions that are mostly in the complex domain, insight into “learning” in political and social contexts may provide useful guidance. 

3) What do you suggest I should read on this topic and question?

Dave Snowden’s Cynefin Framework publications and videos, and the Cognitive Edge website.  If you are interested in the theory behind the Cynefin view of complexity then Max Boisot (a tough read…)- Jonny Morell’s blog- John Mayne specifically on Attribution by Contribution- Outcome Harvesting website and blogs – Better Evaluation website on “What is evaluation”. Any web search engine will turn up plenty of other content on complexity as you search for these, and probably lead you to other useful sources – if you find anything really useful then maybe you could share it on Peregrine ?


Silva Ferretti 

Current RBM are totally not fit for purposes. They allow very little space to understand the meaning of results within context and the processes through which they are achieved. Unfortunately, changing approach requires rewiring our way of thinking. From an idea of linear change (if i do A then B happens) and control (I plan, I achieve) towards an idea of complexity (many diverse forces drive change, and they can play differently) and adaptiveness (we have a purpose, we navigate options). 

It also requires us to put our principles first, and to understand and integrate different worldviews. 

As some students in a training c told me; “we had really to stretch and twist our thinking. we feel we, ourselves, changed”. 

If done properly, principled M&E exploring complex and adaptive systems is not only about setting a different way to collect data or process them. 

It is about asking the whole organization to think and work differently in relation to change. To decolonize our thinking. 

And this is the real challenge. 


Bruce Boyes

Hi Arnaldo,

Following on from Russell’s good advice, I would suggest exploring iterative impact-oriented monitoring, as recommended in one of the Overseas Development Institute (ODI) complexity publications, see https://realkm.com/2020/08/17/taking-responsibility-for-complexity-section-3-2-2-iterative-impact-oriented-monitoring/

Because of the uncertainty and unpredictability that occurs in complex environments, continuous and iterative M&E and the related adaptive management readjustment of actions aimed at achieving the original impacts is much more effective than hoping for the best and then doing MEL at the end. For an example of where I’ve successfully used this iterative MEL approach see https://realkm.com/2016/03/03/case-study-an-agile-approach-to-program-management/

For an analysis of RBM in the context of complexity see Box 2 in https://realkm.com/2019/11/11/managing-in-the-face-of-complexity-part-3-1-tailoring-management-approaches-to-complex-situations/


Thomas Winderl

I agree: RBM hasn’t turned out to be the revolution in adaptive management that it was supposed to be. But my feeling is: it’s not the idea that is flawed, but the implementation. In most cases, RBM turned into a fill-out-the-boxes, “do the same but give it a different name”, increasingly bureaucratic exercise.

That’s a shame. I think there IS value to the basic idea of RBM. My proposal to ‘rescue’ RBM: Follow five simple rules. These are:

1.      A relentless focus on outcomes, outcomes, and outcomes

2.      Keep adapting what you do based on what you learned from outcome monitoring

3.      Involve stakeholders at every step to create quick feedback loops

4.      Budget for outcomes and outputs, not for activities

5.      And overall: keep RBM simple, but not simplistic

I’ve written a longer post about this a few months ago. If you want to see more detailed argument, check out http://results-lab.com/five-rules-for-results-based-management.


John Hoven

Bruce, have you investigated qualitative methods for iterative, evidence-based investigation of cause-and-effect in a one-of-a-kind situation? The dominant method, process tracing, is used almost exclusively for backward-looking causal inference. However, it is easily adapted to forward-looking design and implementation of a context-specific development / peacebuilding project. Iterative updating of the causal theory proceeds at the pace of learning (every week or two, not every 6 or 12 months.) See https://www.researchgate.net/publication/324507012_Adaptive_Theories_of_Change_for_Peacebuilding.


Alix Tiernan

Hi Arnaldo

This is one of our favourite questions 😊.

Here are some practical thoughts on your three questions:

1) Key principles

a) Design the system around processes that repeat regularly and often, and expect to review project documentation at least twice a year, allowing for new strategies, approaches, and activities to creep in, and other originally planned ones to fall out – that’s the essence of adaptation

b) Try to avoid using targets if you possibly can – or if you can’t, then try to use the targets as a comparison to where you are at but NOT as a performance management tool

c) Find creative ways of triangulating and validating your data to ensure that you are taking account of all the different and changing variables and voices (ie. complement surveys with research studies, focus group discussions in communities with meetings with ministers, etc.)

2) Different MEL system and tools

a) you can adapt results frameworks to an adaptive approach but you need to identify some very good qualitative data collection mechanisms that allow you to take into account the effect of the complex context on your project/programme (we have been using Outcome Harvesting for that)

b) make sure you don’t use targets as a performance management tool, because not hitting your target could be the best possible indicator that you are being adaptive!

c) we like using theories of change instead of logframes or results frameworks, because they don’t have to be linear and don’t prescribe specific outcomes – whatever tool you use, make sure you have a way of picking up unexpected results of the work that relate to the expected outcomes, and that you analyse why they happened. Ideally that analysis would then lead to adaptation of further implementation, in an iterative manner.

3) What to read

I will just point you to this document: https://www.odi.org/publications/11191-learning-make-difference-christian-aid-ireland-s-adaptive-programme-management-governance-gender, which we co-authored, but there are many more from ODI exploring adaptive programming, if you google it (it’s sometimes called adaptive programming, sometimes adaptive management). 

Hope these (rather hands on) thoughts are helpful.


Bob Williams

I’m sorry if I appear to be trolling everyone on this list.  Maybe I’ve been around too long and have seen some of the histories and backstories the ideas being discussed here.  My understanding is that Adaptive Management was developed and adopted because of the widespread failure of RBM rather than being part of the adaptive management story.  

RBM has been criticised as a really bad idea both in theory and practice at least since the mid 1990s.  I recall a hugely negative evaluation of the use of RBM in the UN agencies written at least 15 years ago.  It concluded that it was a very bad idea because it wasn’t sufficiently adaptive for the kinds of circumstances that UN agencies.  The formal response from the UN was that RBM was a problem of implementation not the basic concept, without giving a single piece of evidence.  Hence my dismay to find myself working with a very large UN agency last year that was just introducing RBM at enormous cost.

None of which necessarily undermines the advice given by Thomas.  Just that a bit of history (or at least my version of it) might help a little.


Leslie Fox

… and yet, here we are with most of the major multi- and bilateral donor agencies not to mention a good number of international and local CSOs – those who pay our consultant fees — very much committed to an RBM approach. I’ve worked with most of these organizations and while evaluations have required an RBM approach (achieving results vis-à-vis measuring change in indicators), none of them has been limited to a single methodology to arrive at a full picture of whether change has/is taking place or why. I believe it’s called mixed method.


Frank Page

HI Arnando,

I am going to take a bit of a different tack than others who have answered this question more from the technical side. I am going to look at the institutional side. 

The simple answer I would give is to turn MEL systems into MIS – because, ultimately, M&E data/information is management information. 

For organizations to adapt quickly and effectively in complex environments a number of organizational practices are required.  Some organizational practices required include (this list is not complete) combining authority and responsibility in each position as far down the chain of command as to the right places in the organization possible (in other words, and I like this term, all positions has the agency to do their job and improve their performance) and then provide them and give them access to the information they need to make the best decisions. 

In other words, all staff from field workers, to project managers, to program managers, to VP’s and CEOs should be monitoring and evaluating performance in relation to the vision and mission and making adjustments themselves – as low down the ladder as possible. The M&E function of analysis (at least) needs to be integrated in their jobs, not in a separate department. In addition, doing this also moves much of the learning function into the chain of command, and the parts of the learning function that don’t go into the chain of command could very well go under R&D.

The current practice of separating program and m&el introduces many, some serious, coordination problems that are difficult overcome. 

Thus M&E when becomes MIS, it focusses on collecting original data, pulling data from the organization and its stakeholders, pushing data and information to where it is needed and allowing those in the chain of command to pull the data/information they need.  Then, those who are responsible for actually achieving goals in a complex environment have the information to make changes and adapt quickly. 

Photo credit: Tim Johnson on Unsplash

comment 0

Solving problems versus changing systems

I went for a run last week and listened the interview with Nora Bateson on the New Books in System and Cybernetics podcast.

Nora Bateson is the President of the International Bateson Institute and her web bio mentions “she is an award-winning filmmaker, writer and educator based in Sweden. Her work asks the question: How we can improve our perception of the complexity we live within, so we may improve our interaction with the world? “

The interview was about the book she published in 2006, Small Arcs of Larger Circles, a collection of essays, reflections and poems where she writes about systems and ecosystems and where she applies her own insights and those of her team at International Bateson Institute to education, organisations, complexity, academia, and the way that society organizes itself.

There are several points that struck me from the interview. One is that, solving probelms requires seeing the system in which the those problems emerge. This goes beyond understanding context and circumstances that produce a certain problems. The solution to a problem requires looking at the contexts / systems that contribute to the problem.

Here is an example. Say, I want to get or stay fit and decide to go for runs every second day. That is good for my body (system A). However, if the neighbourhood where I live (system B) has no green areas, just asphalt and cement and the air is polluted by the factory at the outskirt of the city. That will affect how fit I can become. If I struggle with finding a job to due to a recession or maybe robots taking up my job (system C), I might put running and getting fit on the back burner. So, in this example there are three systems that interact and in which my running will be part of. In reality the systems would be many more.

I need to change my behaviour to get fit and stay fit through running. But the systems around me, or better, the systems in which I exist determine how fit I can be. They can help me become more fit.

Photo credit: Photo by Daniel Fazio on Unsplash

comment 0

Snapshots from my journey into systems

When did I start to think about systems?

It goes back to mid 2017, when I was finishing writing with some colleagues the concluding chapter of Knowledge, Politics and Policymaking in Indonesia.

While the whole book is about the evolution of the knowlede sector in Indonesia (i.e. the institutional landscape of government, private sector, and civil society organisations that provide research and analysis to support the development of public policy), when we were writing the final chapter we realise that instead of the knowlede sector we had been writing about the knowledge system in Indonesia.

“This is a subtle but significant difference. The term sector, particularly when applied to actors, systems and processes related to evidence-informed policymaking, is reminiscent of a linear approach to the design and implementation of policy reforms and programmes to tackling persistent problems. This approach has often resulted in technocratic interventions that are solution-driven and measured by inadequate behavioural indicators. Utilising the sector definition of Indonesia’s knowledge sector carries the risk that the design of programmes and reforms by the Indonesian government and development partners will reflect traditional bureaucratic structures and accountabilities. Traditional sectors such as health and education have a ministry in charge of designing and implementing policies and managing the policy cycle for specific reforms. However, this is not the case for the knowledge  sector which, as mentioned above and in Chap. 1, is horizontal in nature with no one ministry or department responsible. The complexity, the politics and the diversity of actors and types of evidence in the knowledge sector are more reminiscent of a system or, to borrow a term from big data, a knowledge systems or evidence ecosystem.

Why is it important to see the knowledge sector as an evidence ecosystem?

First, to see the knowledge sector as an evidence ecosystem means to accept that the actors in the system are linked in a complex web of interlinked relationships whereby they not only want to produce and use evidence but also want to influence each other. This web cuts across policy areas and policy sectors. It is wider than any individual sector.

Second, it is because changes in capability within the evidence ecosystem are evolutionary, rather than linear and based on the principles of engineering, which are so common in development programmes (Green 2017). The evolution of a system, as noted by Jacobs (2000), involves a process that constantly produces increasing diversity and co-development relationships. As it evolves, it generates greater complexity. Importantly, the evolution of such a system is governed by uncertainty, rather than the certainty and linearity of results-based programmes.”

To be continued

Photo credit: Photo by Jesse Bowser on Unsplash