comment 0

The evidence on Climate Change is overwhelming, but ….

UN’s Intergovernmental Panel on Climate Change has published its latest report. The authors of the landmark report by the UN Intergovernmental Panel on Climate Change (IPCC) said urgent and unprecedented changes are needed to reach the target, which is affordable and feasible although it lies at the most ambitious end of the Paris agreement pledge to keep temperatures between 1.5C and 2C.

Leading scientists warn that there is only 12 years to be kept to a maximum of 1.5C and avoid catastrophic environmental breakdown.

The Guardian reports that Christiana Figueres, former UN climate chief who led the historic Paris agreement of 2015, said: “There is nothing opaque about this new data. The illustrations of mounting impacts, the fast-approaching and irreversible tipping points are visceral versions of a future that no policy-maker could wish to usher in or be responsible for.”

The scientific evidence is overwhelming but the political will struggle to match it. If climate change becomes the biggest failure of evidence-informed policy making ever, the costs will be incalculable and for generations to come.

Great if we manage to change our individual habits to live more sustainable lives and  pay greater attention to renewable sources of energy. That, however, in itself is not enough. Those individual changes and actions require the support from new policies and new laws.

More articles:

Nicholas Stern: We must reduce greenhouse gas emissions to net zero or face more floods.

Christiana Figueres: Limiting warming to 1.5C is possible – if there is political will

George Monbiot: We won’t save the Earth with a better kind of disposable coffee cup

comment 0

Knowledge Systems and Policy Innovation in the 4IR

A year or so ago, I worked with Diastika Rahwidiati and George Hodge of Pulse Lab Jakarta on a book chapter to reflect on the impact of data innovation on public policy in Indonesia. The lab was established in 2014 as a joint initiative via the United Nations Global Pulse and the Ministry of National Development and Planning (Bappenas). It works to close information gaps in the development and humanitarian sectors through the adoption of big data, real-time analytics and Artificial Intelligence.

The point we make in the book chapter is that in the space of just a few years, the near ubiquity of mobile phones, improvements in connectivity, and the availability of new and cheaper digital technologies are providing policy researchers and policymakers in Indonesia with access to new sources of real-time information and new tools to understand social and economic trends.

The changes in Indonesia are part of a wider global trend. According to the United Nations Global Pulse, growth in the volume of data produced globally has been exponential. In 1995 less than 1% of the world’s population used the Internet. Today that figure has risen to around 46%. The milestone of one billion internet users was reached in 2005; the second billion was reached in 2010 and the third in 2014.


Source: Internet Live Stats 2017

Over the last ten years, the remarkable progress in digital and information technology for collecting and analyzing data has been changing the way policymakers can source and use evidence, adding real-time big data analytics to their evidence toolkits.

This data revolution is one element of a wider process of rapid technological change that goes under the name of the Fourth Industrial Revolution (4IR) which Klaus Schwab describes as ‘a confluence of technological breakthroughs, covering wide-ranging fields such as Artificial Intelligence, robotics, internet of things, autonomous vehicles, nanotechnology, biotechnology, energy storage, and quantum computing.’

The future is closer than we think 

In their book, Next Humans (Prossimi Umani in the Italian edition), Francesco De Filippo and Maria Frega ask 13 Italian scientists working on genetics, robotics, astrophysics, Artificial Intelligence, etc. how life will look twenty years from now.  The picture they paint is breath-taking. Advances in genetics will allow us to be healthier and live longer. New materials (e.g., graphene) will help to build the products we need for our lives and be friendlier to nature. Computing capacity will grow exponentially thanks to quantum computers. Artificial Intelligence will become ever more sophisticated and able to interpret content in the digital world. Energy will be greener. Self -driving cars will travel our streets. The expansion of big data will continue thus increasing the accuracy of predictions about people’s behaviors and events.

In his bestseller Homo Deus, Yuval Noah Harari, argues that humanity is at a brink of a new evolutionary era. The three main problems which have preoccupied humanity over the last 3000 years, starvation, epidemics, and violence, ‘have been transformed from incomprehensible and uncontrollable forces of nature into manageable challenges.’ The unprecedented development of fields such as genetic engineering, regenerative medicine, and nanotechnology provide the possibilities to tackle three new challenges: re-engineering our bodies to live longer and maybe defeat death; find the psychological and biological keys to raise global happiness; and upgrade our human capabilities through biological and cyborg engineering.

The suggestions Harari’s makes may seem to belong to science fiction, but technologies that only a few years ago seemed also science fiction will soon be part of our lives. The World Economic Forum conducted in 2015 an analysis of the probability of the wider application of some new technologies by 2025, what they call the tipping points. The results, summarized in the table below, show that some of the changes linked to the 4IR are only a few years away.

% chance
10% of people wearing clothes connected to internet 91,2
1 trillion sensors connected to the internet 89,2
First robotic pharmacist in the US 86,5
80% of people with a digital presence on the internet 84,4
First 3D printed car production 84,1
The first government to replace its census with big-data sources 82,9
90% of world population using smartphones 80,7
Driverless cars equal 10% of all cars on US roads 78,2
30% of corporate audits performed by Artificial Intelligence 75,4
Tax collected for the first time by government via a blockchain 73,1

Source: Deep shift – Technology Tipping and Society Impact, Global Agenda Councils on the Future of Software and Society, World Economic Forum, September 2015 

The disruption to today’s governance systems

Some of the changes mentioned above will soon be with us. Some are further away. Some may never happen. What is certain, however, is that they will disrupt the governance systems in place today. It is not a matter of whether the changes brought by the 4IR will occur, but when and how. Hence, the pressure on governments to prepare and design policies and regulatory instruments aimed at avoiding the build-up of unbridgeable gaps between countries as well as tech-savvy individuals and the rest of their societies.

The governance and policy challenges are many. I draw here from Schwab (2016) to illustrate three of them:

First: The 4IR will bring considerable changes in labor markets. People will live longer and will remain employed for longer. This will have an impact on jobs markets and national welfare systems. At the same time, automation, Artificial Intelligence, and robotics will substitute many of the tasks performed by workers today. The risk of mass unemployment and social conflict is therefore real. Governments and businesses need to collaborate to design policies that enable workers to change and adapt skillsets by learning new skills and work alongside intelligent machines. In this new future, international competition will not be based on low-cost labor but on the capability to innovate and the quality of human capital required to do so.

Second: the governments’ power will change.  Big data and data analytics are helping the national and local government to respond quickly to emergencies and improve public services. Blockchain technology is opening up new opportunities to informing citizen about the way governments work and deliver public services.  Better knowledge and analysis allow to identify and address locally nominated problems, thus strengthening the demands from local governments for more autonomy and decision-making power. Milica Begovic has made the point that no single actor anymore has the solution to any given problem and that there is the need to engage new thinking from the local level to address context-specific challenges. New, more decentralized, and technology-driven policy decision-making systems will have to develop whereby government organizations engage with citizen and the data and information they have. The 4IR will disrupt existing governance systems and, at the same time, strengthen the need for collaborative decision-making processes that address context-specific problems.

Third: governance systems struggle to keep pace with multiple technological changes that are occurring simultaneously.  State institutions will continue to play a key role to ensure the adaptation and diffusion of new technologies, but the modality to do so will change. In terms of policy-making and innovation, processes that investigate specific or problems, design the necessary policies and regulatory frameworks, and deploy them through top-down systems will struggle in this new technology-driven context. Governance systems will have to be agile and adapt continuously to specific context and circumstances while testing innovation. Collaborations between state, civil society, and business will be key to develop an enabling environment where  space exists for greater policy experimentation and the generation of learning to identify context-specific policy solutions.

Knowledge systems as the foundation of governance systems in the 4IR

Together with colleagues based in different countries in the world (Fred Carden of Using Evidence in Canada, Vanesa Weyrauch of Politics and Ideas in Argentina, Andrés Morales/Luis Carrizo of UNESCO in Uruguay, and Maria Malho of Demos Helsinki in Finland), I have begun to work on these themes. Our interest lies in the consideration that the continued economic growth (and the achievement of the Sustainable Development Goals) in middle- and low-income countries depends also on the capabilities to adapt governance systems in order to become active participants in the 4IR. To do so, countries will have to develop an enabling environment of policies and regulations that supports innovation, helps in strengthening human capital, and ensures equitable growth. We argue that solid knowledge systems, encompassing national research, higher education, data innovation, and local and professional knowledge in government and outside of government, are needed to design and experiment policies in the 21st century.

Knowledge systems matter for any governance systems, regardless to a country ranking on the income ladder. They are not static and continuously evolve, influenced by politics and the technological advances that are also changing industrial production, labor markets, and service provision.

What does this mean for individual countries? What does it mean for countries in the Western Balkans, where the main audience of this publication is?

These countries face a double challenge: reform governance systems to be ready for the 4IR and, at the same time, to become eligible for membership of the European Union. It is a massive effort to design and implement structural reforms in all sectors of society and the economy. The strengthening of knowledge systems should be a high priority. However, the research sector is side-lined compared to other reform areas in the 35 Chapters of European accession.

The point I want to make here is that to take advantage of the opportunities brought by the 4IR (and the accession to the EU in the case of the Western Balkans), it is important to design policies and development assistance programmes that recognize the knowledge system as a system. For example, moving from a focus on one part of the system such as think tanks and knowledge production to situating them and their role in the larger knowledge context.

There is no blueprint to design these policies and programs. In the Western Balkans (as well as elsewhere) it is therefore important to research and understand how the knowledge systems have evolved over time and how knowledge informs (or does not inform) policy decisions today. This can help identify context-specific problems that local actors want to address and for which solutions can be tested and experimented.

The 4IR is imminent. We know it will change the way we live, work, and link with one another.  What is less certain is how. In this article, I have suggested that investments in the knowledge systems can help to develop governance systems that prepare countries to take an active part in the 4IR. The Western Balkans countries, through their process of accession to the EU and the ongoing reforms in the knowledge systems, provide an excellent platform to engage policymakers, researchers, and practitioners to investigate this and inform policy design and development programming.

This article was published originally on Helvetas Mosaic on 27. September 2018

comment 0

A question about policy experiments and adaptive development

I have a question about policy experiments and adaptive development that came to me after reading this article by George Monbiot in the Guardian: We won’t save the Earth with a better kind of disposable coffee cup.

He writes about Starbucks and Costa and that they have been thinking about replacing their plastic coffee cups with cups made from corn starch.

‘The problem is not just plastic: it is mass disposability. Or, to put it another way, the problem is pursuing, on the one planet known to harbour life, a four-planet lifestyle. Regardless of what we consume, the sheer volume of consumption is overwhelming the Earth’s living systems.”

I read the article and then thought about the processes and approaches I am working on at the moment as I believe they can improve the ways we do development, the way we work to change policies and improves people’s lives. I believe in problem-driven approaches, in adaptive and iterative projects, in policy experiments, etc. but are they sufficient to achieve the massive change of behaviors required to live within the Earth’s living systems described by Monbiot?

Are policy experiments, the testing of local and context specific solutions, too small and too slow given the magnitude of the climate and pollution problems we face?

comment 0

Can there be too much data to inform policy and what skills are needed for the policymaking of tomorrow

One of the things I enjoy doing during the summer in Finland is to work in the garden, in the late afternoon, and listen to a LSE public lecture podcast.

Last week I listened to two really interesting ones.

In the first podcast, Tracking the Rise in Global Economic Inequality: new evidence from the world inequality report 2018Lucas Chancel presented the main findings of the first World Inequality Report for 2018.  The report provides the first systemic assessment of globalization in terms of income and wealth inequality since 1980. The discussants were Duncan Green, Paul Segal, and Rebecca Simson.

LSE.png copy 2

In one sentence, the main finding of the report is that ‘income inequality has increased in nearly all world regions in recent decades, but at different speeds. The fact that inequality levels are so different among countries, even when countries share similar levels of development, highlights the important roles that national policies and institutions play in shaping inequality.’

To me,  these findings highlight the importance of developing the state capabilities for monitoring inequality at country, regional, and global level to inform national and  international policy decisions. The report’s webpage makes also available a rich set of data that can be accessed by researchers, practitioners, and civil servants to do some more analysis if they need or want. Will they?

Duncan Green, during his round of comments, made the point that while to have access to the data and the findings of the the report is important, it is also true that today there is a lot of data, analysis, and reports which are produced to inform public policy, maybe even too much.  There is plenty of data produced by development programme to inform government decisions. In other words, there is a lot of supply of evidence.  What is seems to be missing,  is a better understanding of the politics of change and the politics of policy decision making which by nature are very context specific.

To me this does not reduce the importance of producing data and analysis but is more like a call for researchers and practitioners and development agencies to get into the politics and uncertainties of policymaking.



Nick Clegg (Wikipedia)

In the second podcast, Flying the Flag for Openness: why liberalism still matter,  Sir Nick Clegg sets out the case for liberal values at a time of stark social and generational divisions. Towards the end of the Q&A, Clegg was asked by Professor Tony Travers, what skills should a school of public policy (such as the LSE) teach to the policy makers of tomorrow. Clegg said that based on his experience in government as deputy Prime Minister, one of the most critical skills is the understanding  of the trade-offs of any public policy decisions, big or small. Civil servants and policy makers are faced on a daily basis with many choices they have to make. Research, analysis, data, etc. can provide some help. However, the skill that often is most useful (and should be taught more) is about understanding the trade-off of those decisions: less budget for some programmes, the need to increase taxes, policies in some parts of the country but not in other, etc. The skill of understanding and managing trade-offs, according to Nick Clegg, is the one of the most important skills required in the future as the complexity of society and policy decisions continues to increase.

An interesting point, I thought.

Photo by Linus Nylund on Unsplash
comment 1

What is policy impact? Well, it depends from what we mean by policy change

During the last few weeks I have been working on the  design of an adaptive monitoring and learning framework to assess contribution to policy change on climate adaptation in few countries.  Working on the framework made me re-read some articles and papers about policy change which I had read some years ago. Below the key points I used for the development of the framework.

Photo by Linus Nylund on Unsplash

Photo by Linus Nylund on Unsplash

‘Rather than seeing policy as one single, discrete decision, it is important to broaden one’s view, so that policy is understood as a series of documents and decisions that are best described as a set of processes, activities or actions (Neilson, 2001).[1] If we accept this interpretation, it follows that throughout the policy decision process there are different changes that can be identified as the objective of a policy influence strategy or activity.

Jones and Villar (2008)[2] draw on the study that Keck and Sikkink[3] conducted in 1998 on transnational advocacy and the policy process, to describe different types of policy change or policy impact:

  • Framing the debate and getting issues on to the political agenda: this is about producing and communication evidence that draws the attention of policy makers, public opinion, and the media to new issues/solutions. It can result also in policy makers acquiring new knowledge about a specific policy problem. For example, in 2010 the World Bank published a two years study on governance institutions in Vietnam (Vietnam Development Report 2010: Vietnam development report 2010: Modern Institutions) which received considerable attention from media and prompted responses and comments from government officials on the issue of upward/downward accountability and what worked and what did not work well in the policies of devolution pursued by the government.
  • Attitudinal change is about drawing attention to changes in the way key policy stakeholders think about evidence-informed policymaking. For example, the Prime Minister’s Office expresses interest in setting up a policy analysis unit to coordinate the evidence strategy of the different ministries. A Ministry of Planning draws plans to establish a data analytics unit to test ways to use data analytics/dashboards to monitor forest fires.
  • Policy content change is about actual changes in legislation/regulations. For example, in Indonesia after four years of work involving academics, think tanks, the Ministry of Higher Education, the National procurement Agency, the Office of the President, etc., President Jokowi has signed on 15. March 2018 and new Presidential Decree (Perpres – No. 16/2018) that allows Indonesian non-government organisations to be procured by government organisations to conduct studies and policy research. The policy influence team worked and advised on the actual text of the Perpres.
  • Behavioural changes refer to changes in the way of working of government institutions with regard to the demand and use of evidence or the change in behaviour that a new legislation produces. This is the most comprehensive change, which requires more time to achieve and is more difficult to prove. For example, the Prime Minister’s Office in Finland in 2017 allocated 10ml Euro for 42 research topics requested by ministries. The research topics were defined by working groups involving policy makers, academics, and members of civil society organisations. In the Philippines, the Asia Foundation facilitated the process of establishing a coalition involving lawyers, land titling experts, NGOs, and researchers to amend the land titling legislation. President Arroyo signed the new legislation in March 2010. As a result, 50-60.000 land titles are being issued every year (up from ca. 10.000 per year) which has enabled families in urban disadvantaged areas to back up loans requests to banks and to start businesses.[4]

[1] Nielson, S. (2001) ‘Knowledge Utilization and Public Policy Processes: A Literature Review’, IDRC Evaluation Unit. Available at:

[2] Jones, N. with Villar, E. (2008) ‘Situating children in international development policy: challenges involved in successful evidence-informed policy influencing’ in Evidence and Policy, vol4, no.1: p53-73.

[3] Keck, M. and Sikkink, K. (1998) Activists beyond borders: advocacy networks in international politics. Ithaca, NY: Cornell University Press

[4] Watch the video at

comment 0

Reference societies and adaptive development: a conversation with Daniel Suryadarma about Indonesia’s participation in the PISA educational assessment

During a seminar organised by the EduKnow research group at the University of Tampere on comparative education, I came across the interesting concept of reference society. In a nutshell, countries which, like Finland, find themselves at the top of the PISA ranking may become models or references for countries looking for ways to improve their education policies and results.

The PISA ranking has a considerable policy influence internationally but, at the same time, the idea of reference societies seems a bit at odds with the emerging consensus that policy solutions to public policy problems have to be politically-aware, context specific, and that best-practices from elsewhere often lead to failures.

Researching to inform policy … Daniel Suryadarma

I got in touch with Daniel Suryadarma, who is the deputy team leader for the Research on Improving Systems of Education programme in Indonesia and a Research Associate at the SMERU research institute in Jakarta, to discuss about Indonesia’s involvement in the PISA assessment and the reference countries which may influence Indonesia’s education policy.

Daniel, maybe we can start by briefly describing what are the objectives of RISE programme and what kind of activities are implemented by the programme.

RISE is a research programme which focuses on how to make system-level changes that can result in improved learning. The research takes place in six countries: Ethiopia, India, Indonesia, Pakistan, Tanzania, and Vietnam. There are country research teams which focus on system-level questions which are adapted to the different country context. In Indonesia, for example, we focus our research and analysis on teachers by evaluating the impact of different national-level reforms on learning. At the same time, we collaborate with some local governments to design, test, and evaluate different policies and programmes aimed at improving learning. At the local government level, we focus on teachers and system-level questions that are relevant for our local government partners. I think that given Indonesia’s decentralized education service delivery, where local governments have significant freedom to design their policies and programmes, this approach is quite appropriate.

Indonesia has been part of PISA for a number of years now. What do you think is the main motivation for the government to be included in this international ranking?

I do not speak for the government, but I think the motivation differs by agencies, even within the Ministry of Education and Culture. The Assessment Research Centre, for example, is probably interested in having an internationally comparable measure of the quality of education to inform the design of robust assessments instruments to evaluate different learning domains. The Policy Research Centre is perhaps interested in an international assessment which can highlight if national education policies are having a positive effect on learning. The Ministry of Finance, for example, is probably interested in looking whether the increased investment in basic education results in improved learning. I have to admit, though, regardless of the motivation, the fact that Indonesia has continued to participate in PISA since 2003, even with all the criticisms directed at them due to the low ranking of the country in the PISA results, needs to be applauded.

I mentioned to you the concept of reference societies. What are the countries Indonesia is looking at in terms of education policies?

I think there is increasing tendency among policymakers in Indonesia, as well as donor agencies, to stop trying to transplant best practices from other countries to Indonesia. There have been hundreds of such attempts over the past forty years, with little success. In any case, with a country as large as Indonesia, with more than 500 autonomous districts spread across more than 17 thousand islands, it would be quite hard to determine what ‘Indonesian context’ means. This was partly the motivation for the decentralization reform that started in 2001 and which shifted the responsibilities to deliver basic education and health services from the central government to the district governments (although the central government still provides financial transfers to the district to deliver these services). This means that districts have some autonomy in designing their education systems and how to implement national education policies. Ultimately, elected district leaders are responsible for the performance of their education system. Therefore, while five to ten years ago we would still hear stories about the success of Finland or South Korea with their education policies and how Indonesia should adopt practices from those countries, we do not really hear those arguments anymore. 

So, the reference society idea does not apply anymore to Indonesia. What policy changes have occurred in Indonesia as a result of PISA which ranks Indonesia against OECD+ countries? 

I think the PISA results in 2003 provided the motivation for the government and the parliament to revise the law on education and the law on teachers in 2005. In addition, the Indonesian Constitution was amended to incorporate the 20-percent rule. Basically, the government has to invest at least 20 percent of the national budget on education. This is the only sector whose level of public investment is set by the constitution. The PISA ranking has therefore had some influence on education policy development in Indonesia. Overall, I think the government has to be praised for continuing to participate in international tests despite the unflattering results, and for their openness to discuss these challenges with researchers, donor agencies, and other stakeholders. I truly believe that eventually, Indonesia will get effective education policies in place.


Thank you, Daniel Suryadarma.

comment 0

Knowledge, policy, and adaptive development in Indonesia. A discussion at UNICEF’s Office of Research-Innocenti

I was recently in Florence to meet the team at the UNICEF’s Office of Research-Innocenti and present & discuss some of the experiences of the Knowledge Sector Initiative in Indonesia. I was part of the implementing team during phase 1 from 2013-2017 leading the research and learning work of the programme. The Knowledge Sector Initiative is now in its second phase which will run until mid 2022. I remain engaged managing the Overseas Development Institute’s inputs to the programme.

My research background is in education policy in international development and I have followed for some years the research work of the Innocenti Centre. However, this was the first time for me at the Innocenti Center. The sense of history struck me when entering the building.

The Innocenti Centre is hosted in a section of the nearly 600-year-old Ospedale degli Innocenti which was established in 1419 to house and care for the city’s orphans and abandoned children. The building was designed by Filippo Brunelleschi.

Adaptive and sustainable? The Ospedale degli Innocenti opened in 1419.

Adaptive and sustainable? The Ospedale degli Innocenti opened in 1419. (Arnaldo Pellini)

The Ospedale degli Innocenti is arguably the oldest operating children’s care institution in the world. During both World Wars it offered shelter for orphaned children as well as refugee families. Innocenti continues to provide temporary shelter for local women and children at risk. It is a reminder of the good that has been done by Italian institutions amidst the current refugees’ crisis and the battle between political parties around it.

The UNICEF International Child Development Centre was opened in 1988 with a broad mandate and with research quickly becoming the central component of its mission. Within UNICEF, the Innocenti Centre generates evidence and knowledge on a wide range of cutting-edge children’s issues to inform debate, development programming, and policymaking at country level. The current research themes are: child poverty, equity and well-being; child protection; children and the internet; adolescent well-being; children on the move and education policy.


The Centre is de facto UNICEF’s think tank and over the last five years it has invested in strengthening its profile and visibility within UNICEF and with the international community of researchers and development practitioners.

UNICEF country programmes and the Innocenti Centre collaborate to design and develop systems and processes that support an evidence-informed approach to programming and policymaking. Therefore, the team in Florence was interested to hear about the design and implementation of the Knowledge Sector Initiative.

I really enjoyed the discussion we had with the team as well as the preparation for the presentation. It gave me an opportunity to run through the journey of the programme from the initial idea, back in 2009, to the activities of the design phase which lasted until 2012, the first phase of the programme until 2017, the adaptation to a changing context and circumstances. It was great to revisit the knowledge-to-policy system approach of the programme which suggests that to address problems of limited use of evidence in policymaking, may require to experiment simultaneously with knowledge producers, government actors who demand and use evidence, knowledge intermediaries, and the rule and regulations that support the interaction between knowledge producers and policy makers.

The knowledge to policy cycle

The knowledge to policy cycle (KSI)

It is this knowledge system hypothesis that lend itself to building a close collaboration with government and non-government partners and to an adaptive approach to design and implement pilot activities and experiments which generate learning about what works and what does not work and why.

A knowledge system approach, and the contribution that evidence and knowledge can make to inform policy decisions, could provide interesting opportunities for the Innocenti Centre in its collaboration with UNICEF’s country programmes and national level policy initiatives.

I really enjoyed meeting the UNICEF team at Innocenti and learning about their work streams and policy research work and look forward to continuing the conversation.


comment 1

MLE or MEL in adaptive programming?

I was in Manila last week to attend the practitioners’ forum organised by the Asia Foundation on Adaptive Programming and Monitoring, Evaluation, and Learning.

A really good workshop where to share of experiences about tailoring MEL systems to fit the accountability demands from funders and support the adaptation and iteration of programmes. In attendance there were large and small development programmes from the region, funders, implementing organisations, advisors, and researchers.

I gave a presentation at the beginning of day one where I shared my personal experience in this area as well as findings from past and ongoing projects implemented by colleagues at ODI such as David Booth, Alina Rocha Menocal, Tiina Pasanen, Anne Buffardi and others.

One point I made was that the natural sequencing in the day-to-day of a programme is actually Monitoring, Learning and, more intermittently, Evaluation. So, MLE rather than MEL.

Jaime Faustino presented the six simple tools used by TAF’s Coalition for Change  project. These tools show that it is possible to design useful and (at the same time) simple tools for monitoring and learning. An interesting panel on ToCs and their use through an evolving programme. Some of the presentation focused on how programmes design feedback loops to inform the re-design of activities. There was a very interesting and open discussion about what counts as contribution to higher level outcomes for an adaptive and programme.

The Asia Foundation is working on a report of the workshop.

Sharing, discussing, learning .... Asia Foundation

Sharing, discussing, learning …. Asia Foundation

Many interesting points that I am still processing. Here my takeaways:

  • I have participated in few of the adaptive programming and DDD meetings over the last couple of years. I found that this workshop, with its focus on MLE, was a step forward in the discussion. Earlier meetings where more general, which was to be expected as the discussion around adaptive programming was starting. But now we are starting to look at specific elements of adaptive programmes.
  • Large programmes with large budgets, large teams, tackling wicked hard problems of governance capability have specific challenges and opportunities. They cannot change activities or work streams very quickly but, at the same time, they have sufficient budget to develop experimental and adaptive components and MLE systems as part of their design.
  • There was a discussion about the percentage that programmes allocate to MLE. The range was between 3% to 20%. The average at around 10%. This is probably low to support an adaptive implementation. Without insufficient MLE budgets, the risk is that programmes will be asked hard questions about their contribution to outcomes by funders to which they may struggle to answer without adequate MLE resources.
  • Large programme has to set up things quickly and often they develop ToC and M&E frameworks as one of the first deliverables without too much information about problems, possible solutions, and approach. One way to manage this tension, which was shared by some programmes, is to design MLE systems based on Key Evaluation Questions, letting indicators and means of verification emerge as the programme evolved.
  • Some programmes have shared their work with mobile apps and data analytics (TAF Myanmar / Open Data Labs Jakarta) to collect monitoring data. Interesting and innovative solutions, but overall, current MLE systems do not seem to take full advantage of digital technologies. The Excel spreadsheets are still going strong as well as formats in Word with a lot of time spent on the input and output sides by staff.
  • Linked to the previous point, one of the breakout session discussed what MLE will look like 10-15 years from now. We imagined AI that will give us new opportunities to come up with scenarios and ToCs. Monitoring data will be collected through videos and audio recordings. Software will automatically code, analyse, and synthesize the information. Monitoring data will be better at integrating data analysis and AI analysis of changes in social and political context in which a programme operates. Science fiction? Maybe not.

I learned a lot and left with some new questions. It is important to remember that adaptive programming is a mean towards an end. Not an end in itself. In the same ways, a MLE system is mean to support adaptation and iteration of a programme and provide the funder with the information they require. Being adaptive does not mean to try out solutions without a sense of direction.

The question I have is the following: let’s take the Coalition for Change project and some of the policy reforms that the project has contributed to like the property rights reform. The policy change objective has been achieved and is well documented. It is an important policy reform. Is that sufficient or does Coalition of Change (or similar projects) need always to demonstrate a contribution at an higher outcome level, which answer the so what or what does this all add up to question that funders ask?  I am in two minds about this.


comment 0

Serbia’s long journey to joining the EU and what the implications are for evidence informed policy making. A conversation with Nenad Čelarević

Eighteen years have passed since the big demonstrations that took place in the streets of Belgrade and which ultimately led to the overthrow of Slobodan Milošević on 5th October 2000.  Today Serbia is on a new journey which is not without challenges. Ten years ago, the government applied for membership to the EU and in 2012 the accession process kick started a large number of policy reforms required to join the EU. At the same time, former British ambassador to Yugoslavia, Ivor Roberts, has warned in an article published by the Guardian that the west must not just abandon the Balkans to Russia’s embrace.

How is Serbia doing in its journey to joining the EU? What role does the research community play in shaping and informing the reform processes? I have discussed these questions with Nenad Čelarević, Deputy Project Manager & Serbia Programme Officer with the PERFORM: Performing and Responsive Social Sciences project. Nenad is a political activist who is actively engaged in the political transition of the country and who has worked with civil society organizations on issues related to human rights and youth participation in the democratic transition of the country.


Nenad Čelarević…engaged in the conversation about the future of Serbia in the EU 

I travelled to Belgrade last November. It was my first time in the city and in Serbia. I did not know what to expect. My memories, growing up in Italy, were of the Balkan war and of nationalistic rhetoric of president Milošević. What are the main changes that you have seen since 2000.

A lot has changed in Serbia and in the Western Balkans. The main change for me have been elections. The trigger for the big protests against Milošević in October 2000 were allegations of electoral fraud. We now have, for the first time in our history, democratic elections to form our parliament (of course, still not a perfect process and with room for improvements). The governments that have emerged have undertaken policy reforms of almost all areas of social life. As in most countries, not all these reforms have been successful, but they have been designed and implemented through peaceful means and with the overall goal of improving the standard of living in Serbia. This is, in my opinion, the most significant change since the Milošević regime.

In addition to internal reforms, the fall of Milošević has also broken the international isolation of Serbia.

Yes, Serbia has re-engaged with the international community. The application for EU membership, alongside the same request by other Western Balkan countries, was a significant step.

Serbia could join the EU in 2025. It seems to me that the accession process is a massive bureaucratic exercise. There are 35 areas of reforms (or Chapters) that need to be fulfilled in a relatively short period of time. What do you think?

The process of joining the EU is as important as or even more important than the goal of being in the EU. I think that for the Serbian government, the negotiation process is in itself an opportunity to continue strengthening democratic values and the rule of law in our society. To me it is not so important for Serbia to join the EU in 2025. What matters is that we join when we are ready to do so. Being a member of the EU requires high administrative capability at national and local level as well as in the independent bodies that guarantee the check and balances vis-à-vis the state. It takes time to develop these capabilities.

I see two elements that are particularly crucial for the accession process. The first is that Serbia is the largest country in the Western Balkans to join the EU. Sooner or later, Serbia will be asked, as part of the EU, to support other countries in the region with their accession processes. One of these countries is likely to be Kosovo. The 35 accession chapters include a measure that is specific only to Serbia: the normalization of relations with Kosovo. To me, the development of a sustainable and peaceful relation between these two countries is a prerequisite for the enlargement of the EU and for the economic growth and social development in the whole region.

My second point concerns the balance of power in the negotiation process. On one side, there are 28 member states (soon 27 as the UK leaves the EU). On the other, the six countries of the Western Balkans. This division of power will inevitably result in a slow decision-making process. The only way to speed up this process is if the six Western Balkan countries align and coordinate their reform processes. A unity of intents in the region will allow a better negotiation with the EU block.

We both work on processes and systems for evidence-informed policy making. I think we share the belief that good quality and timely research-based evidence can help policy debate and policymaking. Is the research sector included in the 35 Chapters?

In PERFORM we are specifically interested in how the research sector, particularly social sciences, is contributing to the reform processes and the EU accession. While research is included in the 35 Chapters, it is a bit sidelined. The current reforms do not tackle sufficiently the blockages of the research system.  Moreover, the rhetoric against experts’ opinions, which have emerged in some Western democracies, is not helpful for making the case for a stronger evidence-informed policy system in Serbia.

Is the culture of demand and use of evidence in government changing as part of the accession process? If so, how?

If I think about the last 10 years, the way evidence informs specific policy decisions depends on the politics of the policy being debated and the power of line ministries vis-à-vis the overall strategy and direction set by the parliament and the government. It is a spectrum between two extremes: on one end, there are policy decisions where evidence is produced by the government for the government, without much participation by external actors such as researchers. On the other end, there are policy decisions where the government demands evidence and engages with academics, researchers, and activists.

In my opinion, at this point in time, the issue of using research to inform policy (i.e. production, demand, use, etc.) is not a priority. There are a number of important reforms which are highly political and where values and ideology play a stronger role than evidence does. One of these, as I said earlier, is the relation with Kosovo.

Having said this, I think there are areas in the reform process where greater attention is paid to the governance of the demand and use of evidence. For example, the public administration reform and the reform of higher education institutions.

Let’s try to look into the future. If we will have this conversation in 2028 what do you think you will say?

In 2028, I imagine Serbia being a country where young and mid-career researchers from overseas work and conduct their research and where researchers inform policy and debate. I imagine also a Western Balkans region which is more integrated than it is today and where regional disputes and tensions are something of the past, to be found only in history books. I imagine all six countries being members of a reformed EU, which is more democratic and with higher degree of solidarity among member states. I believe that Serbian scientists, researchers, and policy practitioners from different fields can contribute to these changes and shape the future of Serbia, the region, and the EU.



comment 0

What joining the European Social Survey means for social science research and evidence-informed policy making in Serbia. In conversation with Professor Dragan Stanojević.

The Faculty of Philosophy at the University of Belgrade has a long history. It was established in 1838 and today it is located in a modern building in downtown Belgrade. I went there to meet Dragan Stanojević, Assistant Professor at the Department of Sociology. I wanted to meet him because he led the initiative that resulted in Serbia joining the European Social Survey in 2017. The ESS, as it is known, is the most comprehensive cross-national survey that measures the attitudes, beliefs, and behavior patterns of the European population. I wanted to talk to him to understand what the ESS means for the Serbian research community and its significance for evidence informed policy making in the country.


As an academician and a social science researcher, what are the main changes you have observed during the last 15 years in Serbia? 

After a decade of political, economic, and international isolation during the nineties, the early 2000s marked the beginning of a new page in the history of the country with opening towards the European Union and also new opportunities for researchers and universities to be part of international research projects. However, the economic crisis which started in 2008, has somehow slowed down these positive changes and today Serbia, as well as other countries in the Western Balkans and Southern Europe, suffers from a weakening of social welfare systems and a migration of young and educated people overseas which involves also researchers from our universities.

What do you think is the current contribution that social science research is making to policy decisions in Serbia?

Policy impact is not always the main goal for academic researchers. At the same time, academic research does often address important policy issues such as social inequality, social inclusion (or exclusion), and economic growth. Unfortunately, the policy contribution of social sciences research in Serbia is not yet sufficiently recognized among researchers and policy actors.

Currently, there is limited discussion about the contribution that research from universities and social science institutes can make to policy decisions. Most of the policy research used in policy debates comes from think-thanks, with policies mainly designed for and driven by the process of joining the European Union. When universities do get involved, it is mainly because of projects and programmes funded by international development partners.

You have successfully led a team of social science researchers that has worked for 12 months for Serbia to join the European Social Survey? Why is this significant?

The European Social Survey is conducted under the auspices of the European Research Infrastructure Consortium. Methodologically and theoretically, it is the best international comparative and longitudinal social surveys run within the European Union.  For Serbia to join the countries involved in the ESS is significant for three main reasons:

  • Serbia can now bring the methodological rigour and expertise of the ESS into the Serbian research community and contribute to strengthen research and analytical capabilities
  • It will help Serbian researchers to gain a higher profile within the international research community and strengthen the participation of Serbian academic research organisations (rather than simply individual researchers) in international networks. This in turn can inform and influence European policies as well as national policies
  • The access to cross-country comparable datasets and analysis will strengthen Serbian government capability to design and enact public policies required by the process of EU accession.

What do you think were the key factors that helped to bring Serbia into the ESS and for you and your team to succeed?

The Serbian research community has tried for the last 15 years to bring Serbia into the ESS without succeeding. The political interest was simply not there. There main factors that in my opinion explain why we succeeded this time are:

  • The convening and facilitation role of the Performing and Responsive Social Sciences project (PERFORM) has been very important. The project provided the necessary financial support to support the process and, importantly, has taken the role of a broker between the research community and government stakeholders
  • The commitment and enthusiasm by the team of researchers driving the initiative which included young and mid-career researchers, which shows to me that the future is bright for social science research in Serbia
  • The involvement of the Ministry of Education, Science and Technological Development (URL). The ministry recognized the importance of joining the ESS and provided the political buy-in and financial resources that were needed to succeed.

Now that we are part of the ESS, we have an opportunity to generate good quality and longitudinal social development data. Up until now social changes were not tracked systematically over time. This means that on many social issues and problems we cannot say where we stand compared to other European countries. The ESS will give us, for the first time, a chance to generating such data and contribute to designing, costing, implementing and monitoring more evidence-informed policy measures to address social problems the country faces. It is a big step forward for Serbia.


Thank you Professor Dragan Stanojević.