By: Yulye Jessica Romo Ramos, Thomas Kelly, Penny Hawkins

 

We are two evaluation practitioners and an evaluation advisor who work with NGOs, multilaterals and private foundations, and what did not surprise us about the McKinsey report on what makes a great foundation CEO and their conclusion of “discernment, relational intelligence, and storytelling” is that there is barely any mention of data analysis or evaluation playing a role in the strategy and decision-making of philanthropy executives — individual discernment seems enough. 

Globally, there has been a decrease in the level of influence and authority that evaluation and learning leaders and their departments have within their organisations. For example, in the US, 42% of foundations responding to the survey now report directly to CEOs/executives, down from 62% five years ago. The number and dollar amounts of commissioned evaluations have also been decreasing. Both this evidence and our observations across continents and sectors tell us that evaluation has been weakening in its ability to provide useful, representative, and credible evidence capable of influencing decisions and strategy adaptation at the highest levels of organisational decision-making.

Perhaps at the core of the crisis we face is the fallacy or assumption that evidence is the main driver of decision-making and behavioural change. Anyone who has worked directly with boards and executives will know that implicit knowledge and “gut feel” can drive strategies and innovations – usually mildly informed by landscape data. Evaluations and evaluators may have contributed to the strengthening of both the performance monitoring and internal learning of program and grantmaking teams; however, these often still fail to be robust, timely, and influential enough to contribute to the mission-related strategy decisions that CEOs need to make in every budget or strategic planning cycle. 

As the field of evaluation expanded, evaluators’ technical capacities, evaluators have paid insufficient attention to strengthening our communication and influence muscles to be taken seriously at the highest decision-making levels of organisations – CEOs and boards — especially where strategies and strategic directions are developed and approved.

Why is evaluation not meeting the critical decision-making needs of funders and their CEOs? 

• Timeframe and knowledge:  Many evaluations are designed to be done over longer time periods to answer specific questions about programmatic effectiveness (which takes time to achieve impact) and not around generating continuous forms of data analysis and information for insight in real time. Also, not enough attention and resources are applied to the key facilitators of success – organisational capacity, contextual knowledge and programmatic implementation.
• Level of analysis and data: Evaluation methodology was developed to focus on and answer questions about program activity and intervention effectiveness. However, a whole systems view is needed when implementing and assessing interventions and organisations. And even with the increasing influence of multiple forms of artificial intelligence (AI), more data has not always led to greater depth of insight, faster sensemaking, or better decision-making.  
• Wrong questions: When funders and evaluators only try to answer the question: “Did this program achieve the intended impact?”, generally, the answers are not robust enough or even relevant to the strategic and enterprise-level questions and decisions capable of propelling an organisation’s mission. 
• Inadequate theory of impact: An organisation’s expectations for an aggregation of outcomes from small, individual grants to multiple organisations (usually operating independently and not in a collective network of action and accountability) to scale to larger systems and population-level change is a fantasy, not a theory. 
• Technocratic and methodological solutions: the evaluation field and evaluators’ professional advancements have been mostly technocratic and methodological and have not kept pace with the ethical and moral challenges of the practice in today’s challenging political context and the polycrisis facing the world in current times. 
• Euro-Western colonialism: Despite the improvements that evaluation practice has made with participatory and equitable frameworks and methods, Euro-Western academic notions of rigour, such as  “objectivity”, “independence” and “evidence hierarchies” still prevail. (See Equitable Evaluation Initiative’s Orthodoxies). 

Of course, none of the above is new. Back in 2019, at the philanthropy sector’s Evaluation Roundtable, the prominent evaluator Thomas Schwandt advocated for “rethinking the moral compass for evaluation.” More recently, the International Evaluation Academy (IEA) and its partners ran a survey based on a three-horizons approach, finding that “Evaluation is not effective in addressing the deepening polycrises and the transformations required for a just and regenerative planet.” Moreover, global trends like fake news and post-truth politics have added to the relevance and credibility crises in evaluation, thereby threatening the legitimacy of evaluation as a field of inquiry.

Reimagining our Practice

In reflecting on our current evaluation practice, we wondered what else we need to meet the moment and play a more influential role in the C-suite and boardrooms and play a more meaningful role in the future we aspire to. We are not calling for a complete abandonment of key knowledge and skills, such as strategic foresight and systems thinking, as these will continue to be valuable in the future. Instead, we see a need for evaluators to apply a wider range of cultural and historical knowledge systems and include other human faculties such as imagination, emotion and recognition of emotions as powerful ways to support learning, as well as new ways of thinking that can stretch our practice beyond deterministic approaches.

We are not advocating for new technical and methodological solutions or other small improvements, including (usually weakly implemented) forms of participatory or trust-based practice. We call for a drastic and radical evolution and reimagination of evaluation and the role of evaluators. Without significant transformation, it’s increasingly likely that evaluation will fail to be of use in the expanding polycrisis we will continue to face. 

Transformative change requires shifting evaluation’s positionality, power, and purpose towards:

• Relational ontology and a practice with more heart and soul. One that centres more relational and imaginative approaches in how we position ourselves, how we interact with people, and other knowledge systems. For this, we need to unlearn that “neutrality” and “professionalism” equal quality of evidence. It is the quality of our relationship with others and their ideas that gets us closer to the truth and the potential for evaluation to contribute to impact. 
• Enabling upwards accountability and power shifting. This includes meaningful community-based (beyond participatory) approaches with a focus on learning from, listening to, and amplifying local knowledge holders’ voices, expertise and visions for transformative and sustained impact; and including community in the assessment, as well as rewards and sanctions, as components of public accountability of funders, nonprofits/NGOs, and governments. 
• A practice anchored in trauma-sensitive and culturally responsive approaches to play a meaningful role in healing historical injustices and creating the enabling conditions for positive transformation.

We could measure the success of evaluations not only by their technical quality but also by how much they influence decision-makers and contribute to their mission. As evaluators, our measures of success also need to include: 

• The way we make people feel: If our practice is truly relational and culturally-responsive, then those we work with would feel understood and respected.
• How we learn from people: This needs to go beyond training and sense-making approaches, and include reciprocal and collective growth and learning.
• How we communicate with people to support decisions that result in positive impacts: This goes beyond utilisation-focused approaches. We also need to employ arts-based methods, audio-visual tools and interpersonal approaches to extend the range of forms of human expression and foster greater mutual understanding. 
• Power shifting and healing: We need to understand community advocacy and action, as well as respecting community self-determination and data sovereignty. More time and safe spaces are needed when we engage with people. We must avoid perpetuating colonial mindsets and be able to identify and adopt appropriate practices in different situations, and know when to step aside.

Our experience tells us that all of these issues need further collective sense-making, possibility and solution-seeking.  We are interested in continuing this conversation in upcoming online discussions designed as a series of candid conversations with evaluation practitioners across the world. Consult this link to find out how to join the conversation!

 

Authors/ Bios:

Yulye Jessica Romo Ramos

Yulye Jessica Romo Ramos is the founding director of Nexus Evaluation and co-lead of the EES’s Equity & Gender thematic working group. Nexus is a mission-led consultancy company committed to maximising social and environmental impact with the use of systems thinking, equitable, trauma-informed and culturally responsive approaches to strategy, organisational strengthening, and MERL (monitoring, evaluation, research and learning).

Nexus Evaluation LTD has partnered with global foundations, NGOs, and development programs, including the Ford Foundation, GiZ, the School of Systems Change, Global Philanthropy Project and the UK Evaluation Society – to name some.

Jessica is a seasoned leader with 15 years of expertise spanning diverse thematic areas, including human rights, advocacy, gender equity, health, and climate change. This mission is supported by a global perspective, with experience across Europe, Latin America, Africa, and Asia. She prefers a relational and partnership-focused approach to her work. By understanding specific needs, contexts, and challenges, she develops innovative approaches and services that create real and lasting impact.

 

Thomas Kelly Jr.

Tom Kelly has been helping philanthropy, government, and investors define and measure social impact for more than 30 years. He was an elected board member of the American Evaluation Association (2020-2022) and former leader of evaluations at Hawai‘i Community Foundation and Annie E. Casey Foundation.

Tom’s current work includes evaluation and strategy planning within community, family, private, and corporate foundations with a focus on evaluation utilization and systems-level strategy and results. He also has designed impact measurement approaches for social impact investment portfolios and co-edited the special issue of New Directions in Evaluation on futures forecasting and evaluation.

He is a graduate of Harvard College and The George Washington University School of Public Health, and lives and works on the lands of the Agua Caliente Band of Cahuilla Indians in Palm Springs, California.

 

Penny Hawkins

Penny Hawkins is a social impact measurement and evaluation specialist with senior-level leadership and management experience in government and philanthropic sectors, and deep expertise in the design, establishment and ongoing development of impact measurement and learning systems in a wide range of organisations.

She has extensive senior-level leadership experience in New Zealand and UK central government, including policy, strategy, financial management, commissioning and management of impact measurement, evaluation, research, and performance monitoring and reporting. She also has wide-ranging experience in developing capacity and capability in the commissioning, production and use of high-quality evidence, including coaching and mentoring managers, technical and operational staff at all levels.

Penny is interested in creative approaches that blend art and science for impact measurement, serving current and emerging evidence needs at a systemic level in complex and rapidly changing contexts.