comment 0

The brave new world of evaluating innovation. In conversation with Petri Uusikylä


In a recent blog, I discussed with Jordi del Bas what it meant to evaluate innovation. My main takeaways from the conversation with Jordi were that when evaluating innovation it is important “to generate evidence of whether innovative solutions work, how and why, so that they can go beyond a pilot and be scaled up to maximize their positive impact”. The OECD Development Aid Committee (DAC) evaluation criteria are not well suited to initiatives that test innovation, as they fit initiatives that are built on a reasonable degree of certainty, predictability and control. 

Talking with Jordi helped me to better understand the gap that exists between the established OECD DAC evaluation criteria of efficiency, effectiveness and impact, and criteria that better fit with and evaluate the experimentation of innovative solutions to social problems: criteria such as feasibility, desirability, viability, acceptability, usability and scalability. When evaluating innovation, it is first and foremost important to assess the learning that designing, testing and adjusting the innovation has generated versus whether the innovation achieved its planned objectives and goals.

I wanted to continue exploring this gap. What interests me is that this gap is not simply about evaluative tools and methods, but it is almost philosophical; it is about the principles, values and viewpoints, as noted by Gunnar Myrdal, that shape our idea of development.

Exploring complexity and innovation, Petri Uusikylä

I met Petri Uusikylä last winter at a presentation on monitoring, evaluation and learning, organized by Fingo in Helsinki. My Overseas Development Institute colleague Tiina Pasanen was presenting her work on monitoring and evaluating adaptive programmes. During the Q&A session I learned that Petri had worked in the area of evaluation for several years and that some years ago he took a strong interest in complexity and systems thinking. I reached out to him to learn more about how his interest in complexity had evolved and how it influences his view of evaluating development and social change.

Arnaldo Pellini – Tell me about your background and your evaluation work in international development

Petri Uusikylä – I have been doing evaluations, performance audits and capacity building in the international development context for 15 years. I have a background in political science and experience in teaching political theory, research methods and European policymaking at the University of Helsinki in the late 1990s. After that I worked at the Ministry of Finance as a senior advisor, and for the last 10 years I have been working as a public policy consultant, EU Twinning advisor and government public policy advisor. I was one of the founders of the Finnish Evaluation Society in 1999, and I am one of its board members. In the 2000s I worked on traditional policy and programme evaluations for about 10 years. In most cases, I worked with the OECD DAC evaluation criteria which I know very well.

AP – Tell me about your interest in complexity and systems thinking, and your research

PU – I have been interested in social network analysis for a long time. It started when I worked at the Political Science Department at the University of Helsinki between 1991 and 1997. That was during the early days of the Internet, before Facebook and other social networking platforms emerged. It was difficult at that time to analyse relational data. There was very little software to do that and it was not very good. In the team I was involved with, we adopted a trial and error approach to social network analysis, which led to some interesting results that we described in a book we published in 1999 called, The Network Society (Verkostoyhteiskunta:  käytännön johdatus verkostoanalyysiin).

I worked on traditional policy and programme evaluation for about 10 years after that. I learned and applied the OECD DAC evaluation criteria in policy evaluation, programmed evaluation and project evaluations. They were guiding my work – the questions I was asking and the evidence I was gathering. They always felt a bit too mechanical for the kind of initiatives I was evaluating, but they were widely used and I could not think of alternatives.

In the autumn of 2014, I remember reading Ben Ramalingam’s Aid at the Edge of Chaos. I was taken aback. It was by far the most brilliant book on international development I had ever read. It felt to me that Ben Ramalingam had put into words many of my thoughts and ideas about development and the evaluation of development initiatives.

Aid at the Edge of Chaos introduced me to the notion that cutting-edge ideas from complex science can be applied to the social, economic and political issues in development, and that those new ideas can also contribute to transforming the way international development works.

Soon after reading Ramalingam’s book, I applied some of its ideas to two evaluations: one for the Finnish Red Cross and one for the Innovation Partnership Program between Finland and Vietnam. The terms of reference for these evaluations required me and the team to follow the OECD DAC criteria, which we did. But we also managed to persuade the clients (Finnish Red Cross and the Ministry or Foreign Affairs) that we could complement the OCED by testing new systems evaluation methods. In the end it all worked out well and helped me to continue exploring the ideas of complexity science and how to apply them to development and evaluation work.

AP – What do you think evaluations capture and what do they miss in the way they are normally carried out?

PU – There has been an enormous growth in the overall evaluation landscape regarding the institutionalization of evaluation in the last decades. There is a new comparative study of the institutionalization of evaluation in Europe in which I was honoured to be one of the authors for the Finnish chapter. Finland compares well in most of these areas. According to this study, Finland is above average in the institutionalization of evaluation in its political and social systems, but lacks the procedures to utilise evaluation finding in political decision-making.

When it comes to the evaluation methodology, current evaluation approaches are rather static in applying mechanistic and linear causal logic, and often rely on rigid, a priori, defined evaluation criteria and methods. Thus, they are not suitable for understanding reasonably complicated or complex policy phenomena. My doctoral thesis last year was on the systems evaluation approach that relies heavily on systems thinking and complexity theories and utilizes systemic evaluation designs derived from these theories. Developmental evaluations, systemic models, social network analysis and the theory of change are examples of this approach. I also suggested that evaluators should adopt a more active role as knowledge brokers and policy interpreters between governments and citizens. This would make it easier for citizens to understand the complex regulatory and policymaking environment, and thus to support the development of an open and democratic society.

AP – How do you see the public sector, of which international development is an element, working with complexity and systems thinking?

PU – In the last 25 years, governance systems have changed. In my opinion, in western democracies hierarchical top-down governance structures are being slowly replaced by more bottom-up, collaborative, networked governance systems and capabilities.

Having said this, I am unsure as to whether governance systems have transformed sufficiently to tackle the wicked problems that are part of our interconnected societies and whether the evaluation methods and approaches which are used to assess the impact of public policies have evolved to meet the needs of new systems of governance.

The governance and leadership systems have to be agile to address and adapt to complex changes. Old management models based on best practices will not be sufficient to achieve success in public sector management. As my colleagues Petri Virtanen and Jari Kaivo-oja stated: “These changes are likely to be so challenging and pervasive that in some countries they go beyond the existing governance capabilities and will result in state failure.”

In a near future, there will be more need to strengthen and increase linkages and collaboration between state and non-state actors to address complex challenges such as inequality, climate change, and the pandemic we are experiencing today.

In a book I co-edited with Dr. Hanna Lehtimäki and Dr. Smedlund entitled Society as an Interaction Space, we present some new perspectives on these linkages which we call relational dynamics between governments, companies, and citizens. In doing so, we merge insights from public service science, political science, institutional logics and value co-creation.

I am currently working in collaboration with MDI Public, the University of Vaasa and Demos Helsinki on a project commissioned by the Prime Minister´s Office and some line ministries to develop a new systems governance model for Finland. As part of this study, we are benchmarking development in systems thinking in the United Kingdom, Denmark, Canada and Singapore. This study is aimed to be completed in spring 2021. 

AP – Tell me about your work with the University of Vaasa and your plans to have a research centre on complexity

PU – I started as a research director at the Complexity Research Group at the University of Vaasa this April. The university has built a strong research profile in the field of governance and complexity thinking over the past 10 years. The newly established Complexity Research Group (Kompleksisuustutkimuksen ryhmä) will start using complexity science in various domains to identify, map and better understand wicked problems in areas such as climate change, energy, public administration, anticipatory governance and so on.

With the help of complexity lenses and through cross-fertilization of findings from various disciplines, the research group is committed to exploring new niches within systems leadership and beyond. I am honoured to join the great team of complexity researchers. My aim as a research director is to bring a strong international and global context to the complexity research carried out at the University of Vaasa in the future.

Petri, thank you very much.

If you republish please add this text: This article is republished from Knowledge Counts, a blog by Arnaldo Pellini under a Creative Commons license. You can read the original article here

Photo credit: Mika Korhonen on Unsplash

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.