Programs need to change, and so does evaluation. There is no single method that would represent the “gold standard.” The main challenge and difficulty is adaptability and appropriateness. You need to understand what a particular program needs at a particular point in time.
If an evaluation involves, for example, the use of advanced sociological tools, which I am not familiar with as an evaluator, I would not be able to conduct an evaluation at the proper level, would I?
Absolutely. This is partly why it is so important to have professional associations. Today, thanks to them, to social media, to networking, you can always find someone to help you learn these methods. Many evaluations these days are done collectively, so no single evaluator would have all the necessary techniques and skills. However, if you can find the right people through an association, the Internet, or social media, you can find evaluators with all those skills.
Would you say that most clients today require the use of advanced sociological analysis methods?
They need to be able to get feedback from the beneficiaries, if that’s what you mean by sociological analysis. Programs should have good opportunities to get high-quality, real-world data from program participants, so they can find out what is really going on. Perhaps the most important and most common thing is that evaluators get direct, independent and real feedback from the target audiences involved in the program – how well the program is working, whether they are getting the help they need, and what can be improved.
You need to be a good interviewer, to know how to do surveys, use social networks, and get data directly from people.
You are a professional sociologist. In your experience, is it common for evaluators to have a background in sociology or psychology, a professional mastery of SPSS and similar programs, or is it really not necessary?
Fortunately, there is additional training available. Much of my sociological education was strictly academic, and I had to learn evaluation by doing. When I started, there were no courses, no seminars. I had to take interviews and surveys as sociological methods and adapt them to evaluation. Psychologists had to do the same with tests, and economists with cost-benefit analysis. Working in evaluation requires the ability to adapt these academic research methodologies. This is a key skill.
Today evaluation is not only about techniques, but also about interpersonal relationships. Evaluators must be good listeners, able to work with people from different cultures, be able to make connections, communicate well, solve problems and manage conflicts.
Let’s talk about the so-called “soft skills.” Are they really important to the evaluator?
That’s a good question. The American Evaluation Association has established a set of competencies that are essential to an evaluator. Of course, these include methodological competence, project management, knowledge of professional and ethical standards.
And then there are interpersonal competencies. Evaluators must be good listeners, able to work with people from different cultures. They must treat people with respect. Be able to make connections, communicate well, solve problems and manage conflicts.
One of my recent books focuses on the relevant skills for working with stakeholders at various levels. This is an important change, as evaluation today is not only about techniques, but also about interpersonal relationships.
In some of your publications, you mentioned the concept of evaluative thinking and suggested sharing this type of thinking with the project team, with the client. Doesn’t this threaten the existence of the evaluators as a profession? Wouldn’t evaluation become just a management function?
I don’t consider this a danger. You described the situation very well, but evaluative thinking in management only means that you have better communication because you speak the same language. People understand the concepts of logical frameworks, theories of change, SMART goals, different types of evaluation for different purposes, but in order to do the actual evaluation, you still need a fresh perspective, you still need people who can look at things differently.
When people do have an evaluative mindset, they begin to appreciate having a specialist who helps them look at things through the eyes of an outsider. It is something I do often. As an evaluator, I help managers, employees and even shareholders understand how evaluators think, how they define criteria and interpret data, how they draw conclusions – but they would still need an evaluator to make their job easier, to have another pair of eyes to look at things from a different perspective. Helping to think in an evaluative way is about improving communication and understanding of what’s going on.
So you don’t think that evaluation will become just a part of management, because the evaluator has other functions besides allowing you to understand what’s really going on.
Yes, that’s right. Evaluators also give an additional point of view. Even where there are internal evaluators in the management, their job is to do the evaluation and make sure there’s good data and make sure people take the time to interpret it. I believe the evaluator function will remain, but the better management understands evaluation, the better communication and use of evaluation results will be.
In your 2009 article, “The Future of Program Evaluation,” you wrote that the “gold standard” of evaluation methodologies would be hotly debated over the next few decades. Can we say that the answer to this question has changed significantly since then? Or do we still have the same leader – randomized controlled trials?
The debate continues, but it’s becoming increasingly clear that in a rapidly changing world, we need fast-turnaround techniques that produce real data in real time. Randomized controlled trials are really not very useful in a complex, dynamic, rapidly changing world. They work best in a stable situation where there are clear fixed interventions.
In a rapidly changing world, we need fast-turnaround techniques that produce real data in real time and provide them just as quickly to people.
If you have to work on topics like pandemics, climate change, or civil unrest when things are changing rapidly, randomized controlled trials are not appropriate – they are too rigid and too time-consuming.
The speed of change means that the evaluation must produce results quickly, in real time, and provide them just as quickly to people who need to make decisions in a time-constrained environment.
The evaluator’s profession is changing in response to what is going on, but evaluators themselves are also changing the world. What do you think this profession has already given to the world?
I think we are living in a time of a worldwide battle between approaches based on evidence, facts, and science, on the one hand, and a part of the world that ignores them and only seeks to promote ideology. Evaluation is part of this worldwide data. Evaluation proves that if you use data, you make better decisions and help people more.
We’ve seen it work with people who refused COVID vaccination, who don’t believe the pandemic and vaccination data. In the so-called post-truth or anti-science world, where people can say whatever they believe and assume it to be true, evaluators help people appreciate the importance of evidence, looking at reality and working with it rather than with subjective perceptions and beliefs. This is our contribution.
In the so-called post-truth or anti-science world evaluators help people appreciate the importance of evidence, looking at reality and working with it.
You are the author of many program evaluation concepts used by evaluators around the world. We would like to discuss some of them. The first question concerns the concept of utilization-focused evaluation. In it, you emphasize the client’s role and interest, their desire to participate in the use of the evaluation results. Nowadays, investors in social projects and even the state (as in the social impact bond model, SIB), often become the evaluators’ clients. They are not involved in the project itself, but are its customers. Is it possible to use the evaluation principle with an emphasis on its use in this specific situation? Are there any peculiarities of its application in this case?
This is an important question. Utilization-focused evaluation requires considering different levels and different types of stakeholders – we call them intended users. Who is this evaluation for? If the intended users are social investors, then I would want to engage them in defining the criteria they are seeking. What do they want to know to make a better investment?
When I work with social investors, part of what I do is help them understand as clients that there are different ways to do evaluation, because there are many different types of programs. For example, when I work with boards of directors investing in social projects or charitable foundations, I do an exercise with them. I make a list of different types of financial instruments: blue chip stocks, high growth stocks, undervalued stocks, that is, stocks of companies in transition or struggling, long-term bonds, short-term bonds, mutual funds. These people are well aware of the different types of financial instruments and the differences in their profitability. Then I take a set of projects and draw parallels between different kinds of projects and different kinds of financial instruments.
Blue chips are well known programs like the World Food Program or UNICEF; they are world-famous, stable, and have a good reputation. A program that seeks to scale, to enter other territories within or outside the country is the equivalent of high-growing stock in the financial market.
Undervalued stock is a program that is in transition and needs help to get through and adapt to change. Long-term bonds are the equivalent of operating support for a program without specifying what specific project the program will implement.
This exercise helps social investors understand that they value different financial instruments differently. They look at different types of stocks, investment funds or bonds using different criteria, and we also evaluate different programs using different criteria depending on the nature of the program and the nature of the investment. Using the logic of evaluating financial instruments helps them understand the logic of evaluating programs.
The theory of transformation, trying to change systems, must consist of many different theories of change. The Blue Marble evaluation is related to the theory of transformation: once we start dealing with large systems, we are dealing with global impacts.
Investors in social projects also have internal networks and have developed a number of methodological approaches. For example, they are very fond of the social return on investment (SROI) method. Sometimes it is even taken as the only possible, no-alternative method for evaluating the results of any project. This seems to go at odds with the evaluation specialists’ views – again, based on the utilization-focused evaluation concept.
The social effectiveness of investments still needs to be interpreted and adapted, depending on whether we are talking about long-term or short-term social effectiveness of investments, about innovative high-risk programs, high-risk programs with high performance potential, or low-risk, low-return programs. When I talk about parallels between different kinds of financial instruments and different kinds of programs, I’m talking about how much risk social project investors are willing to take, how innovative they want to be, what the time span is for investment and return, whether they are looking for several different indicators or just one measure of success.
There is a lot to negotiate and a lot to adapt, even with the general criteria of investing in social projects. It is also necessary to determine what it means to invest in social projects and the effectiveness of such investments in a particular situation. And this is where utilization-focused evaluation is needed, because it provides a dialogue with clients and helps them articulate the timing, the criteria, the degree of risk they are willing to accept, and the required level of return on investment.
Using the logic of evaluating financial instruments helps social investors understand the logic of evaluating programs.
The next question concerns the theory of transformation. What is the difference between your proposed term “theory of transformation” and theory of change?
That’s a great question! Theory of change was a concept suggested in 1995 by sociologist Carol Weiss, who observed how foundations began to implement programs without knowing what data on effectiveness the studies showed. They didn’t know about advances in social sciences, about how to fight poverty or drug addiction.
Now we have moved on to trying to change entire systems (climate, agriculture, food, healthcare). When you start changing the system, you go beyond one particular project.
The theory of change looks at an individual project or program, while the theory of transformation, trying to change systems, must consist of many different theories of change. For example, the anti-smoking campaign has more than one theory of change. You need to educate people about the dangers of smoking, work with advertising and marketing, with healthcare. You need regulations and incentives, active work with tobacco manufacturers, with young people, with the elderly.
There are many different approaches to fighting smoking, each with their own theory of change, and the theory of transformation attempts to integrate all of these different theories of change to achieve an overall systemic impact.
I think this also applies to the Blue Marble concept[8 - The Blue Marble is a photograph of Planet Earth taken on December 7, 1972 by the crew of Apollo 17 spacecraft from a distance of around 29,000 kilometers from the planet’s surface. For the astronauts, Earth was the size and appearance of a blue marble (a colored glass ball toy).]that you’re working on right now. Could you briefly describe the main ideas of this concept?1
The Blue Marble evaluation is related to the theory of transformation: once we start dealing with large systems, we are dealing with global impacts. For example, the state of agriculture in any country is affected by climate change, world markets, government policies, the policies of international agencies, trade, and transportation. The idea behind the Blue Marble evaluation is that we need to understand not only what is happening in a particular local program, but also how that program is affected by global and international trends, such as pandemics or climate change.
The Blue Marble evaluation concept connects the local with the global. The English word for this is “glocal,” which is a portmanteau of “local” and “global.” It means looking for interactions and interconnections between the local and the global, so that our understanding at the global level is based on an understanding of local processes, and our understanding of what is happening at the local level is conditioned by an understanding of global trends.
Let’s talk about the implementation of this concept in the evaluators’ practical work. For example, if an evaluator decides to use the Blue Marble concept in his work, where should they begin?
Let’s take investing in social projects as an example again. For any social project investment, it is important to consider the specific project towards which the funds will be invested, but from the Blue Marble perspective, evaluators also need to know what other social project investors are doing. What is happening in other countries? What are the global trends in social project investment? What networks are involved? The more money goes into social projects on a global level, the more important it is to apply global trends and patterns to each specific project.
The work of the evaluator in this case includes not only developing a specific social impact evaluation, but also knowing the global context, what is going on in other social impact evaluation projects, what can be learned from them, what the indicators are, and how this relates to the sustainable development goals. For example, is a particular social project investment linked to one sustainability goal or several sustainability goals? Specifically, this means that the evaluator needs to learn how to place the individual project in a broader global context.
Do you have a manual or a list of questions that the evaluator should ask? Like not forgetting climate change and other global issues, for example.
Yes, I do have such a list of questions. As you suggested, the first part relates to climate – how does the project relate to sustainable development, recovery, climate change mitigation? The thing is that we are living in a climate emergency, and any action we take can have an impact on it.
Another set of questions concerns social justice, because the gap between rich people and poor people, between richer and poorer countries, keeps growing endlessly. There is an ethical criterion for how a particular project supports gender, social and economic equality and racial justice. Thus, social justice and equity issues and climate change impacts have become universal aspects to be considered in all program evaluations.
My final question is about the future. If we try to look into the near future of program evaluation, what changes can we expect?
I think there will be more networks of evaluators exchanging data and communicating with each other, because we are dealing with systemic change. For example, I work with the Global Alliance for the Future of Food, which combines 30 charities on four continents. This includes foundations from North America, Europe, South America and Asia.
Solving global problems and tracking systemic change requires a collaborative team of evaluators from different countries, speaking different languages and with different methodological skills. Last year, me and a team of Blue Marble evaluators participated in the evaluation of the UN Food Systems Summit.
Currently I am working with the new International Evaluation Academy, which has participants from all over the world. I am part of the Russian Association of Specialists in Program and Policy Evaluation, and I’m assisting the International Organization for Cooperation and Evaluation. International and global networks and evaluation teams are definitely part of our future.
Экспертные мнения / Expert Opinions
Без стандартов, но по призванию. Является ли оценка проектов и программ профессией?
Тема профессионализации оценки проектов и программ (в социальной сфере, культуре, науки, экологии и т. д.) существует в повестке международных организаций второй десяток лет, в российской практике – около десяти. Появляется много публикаций, проводятся дискуссии, специалисты по оценке обсуждают статус ее как профессии на региональном и международном уровне. В нашей стране эта тема рассматривается на ежегодных конференциях Ассоциации специалистов по оценке программ и политик (АСОПП). Так является ли оценка проектов и программ профессией, и если да, то как это можно понять и узнать?
Юлия Вяткина
Редактор журнала «Позитивные изменения»
СТАДИИ ПРЕВРАЩЕНИЯ ЗАНЯТИЯ В ПРОФЕССИЮ
Прежде чем начать разговор о профессионализации оценки (здесь и далее под оценкой мы будем понимать оценку проектов и программ – прим. ред), рассмотрим, каким образом то или иное занятие превращается в профессию.
Американский социолог Гарольд Виленски, исследовавший функционирование различных организационных структур в современном обществе, предложил в 60-х гг. прошлого века модель, согласно которой формирование профессии проходит несколько стадий[9 - Wilensky, L. (1964). The professionalization of everyone? American Journal of Sociology, 70(2), 137–158. Ayoo, S., Wilcox, Y., LaVelle, J. M., Podems, D., & Barrington, G. V. (2020). Grounding the 2018 AEA Evaluator Competencies in the broader context of professionalization. In J. A King (Ed.), The American Evaluation Association’s Program Evaluator Competencies. New Directions for Evaluation, 2020, 13 30.].
Все начинается с того, что некое занятие приобретает статус полной занятости. Т. е. достаточно много людей занимаются некой деятельностью с утра до вечера и зарабатывают этим на жизнь. Полная занятость – это отправная точка, иначе занятие не может превратиться в профессию. Кроме того, она возможна, когда есть спрос на услуги.
Вторая стадия – это появление школы профессиональной подготовки людей, которые занимаются этим видом деятельности. Такая школа чаще всего ассоциирована с университетами. Далее появляется профессиональная ассоциация – объединение людей, которые занимаются этой деятельностью на постоянной основе. Ассоциация делает определенные шаги, чтобы со стороны государства были приняты какие-то нормативные документы, которые бы позволяли в установленном порядке лицензировать, сертифицировать деятельность, чтобы появилась своего рода профессиональная монополия. Также профессиональная ассоциация влияет на профподготовку специалистов, появляется теория, методы, стратегии деятельности и компетенции специалистов.
Следующий шаг – этический кодекс, который обеспечивает предотвращение злоупотреблений авторитетом и властью. Профессиональные ассоциации демонстрируют общественности, что они контролируют соблюдение этического кодекса и являются саморегулируемыми.
Еще один этап – формирование позитивного имиджа в обществе. Когда имидж есть, можно говорить о том, что появляется профессия.
