Interview with Dr. Bidisha Barooah

Introduction to Impact Evaluation Methods

by Rhea Bisaria and Aditi Acharya

The Economics Society, in collaboration with the Centre for Economic Research, St. Stephen’s College hosted Academic Summit 2.0,  a series of lectures on  Applied Economics from 25th to 27th of March, 2019.

The society was privileged to host Dr. Bidisha Barooah who delivered a lecture on the topic ‘An Introduction to Impact Evaluation Methods’. Dr. Bidisha Barooah is Senior Evaluation Specialist at the International Initiative for Impact Evaluation (3ie) and is also a senior research assistant at the Delhi School Of Economics. 

Dr. Barooah spoke about the unique field of impact evaluation, the different methods involved and contexts in which impact evaluation plays an important role, while illustrating the key principles required for good evaluations.

Rhea Bisaria (RB) and Aditi Acharya (AA) had the privilege of interviewing ma’am and are delighted to share the transcript of the same.

RB: In recent years, policies based on impact evaluation have been receiving a lot of focus in developing as well as developed countries. In this regard, what do you think of India’s attention to these policies, has it been adequate or is there a lack of attention in this field?

BB: I think the field is certainly changing and we are finding that Indian policy makers are becoming more open to evidence-based, informed decision making. There have been a number of cases where similar evaluations have gone on to become the foundation for informed large-scale policy making. However, compared to many other countries, we are lagging behind. I think the focus in India is very much still on an input- based evaluation rather than an outcome- based one. But we do see that in some pockets, particularly in some government departments, there has been a lot of buy-in for evidence based decisions.

AA: People have often been skeptical about the impact of the impact evaluation programs, considering how governments have continued with projects which have not been successful empirically. What do you think are the reasons for the same and how do we overcome it?

BB: Well, that’s something that even we at 3ie are grappling with, on how to increase accountability. We saw, in the lecture, that the constraints of a policymaker are very different from the constraints of an enumerator. A policymaker typically has to complete a task by a deadline and one of the feedbacks that we’ve constantly received, from our interactions with policymakers is that the evaluation results don’t reach them on time and they have to proceed without them. So while there’s a demand for impact evaluations, maybe they don’t have the time to wait for these results or there are many other political considerations, that we as evaluators are only slowly becoming aware of, which tells us that we need to scale our programs up and we have to learn to be adaptable and to navigate our way around these constraints and make our evaluations more useful to policymakers. We also try to explain to policymakers that a well-done evaluation can inform them more in the long-run than something quick and small. However, this is a principle- agent kind of a problem and they have different motivations and tasks from evaluators. We have to find a way to work together.

RB: Impact evaluation for policies, done by a country on a large scale, seems to be a method that requires a lot of resources and manpower and there’s a also a time constraint involved in the evaluation process. Is there an alternative to this option for countries that do not have the required means?

BB: The quickest and perhaps the best use of resources is to build an impact evaluation monitoring system that tracks outcomes. The government, even though is investing in a lot of MIS (management information system) and already has a few good systems in place in some schemes, we find that they only stop at inputs. Perhaps having outcome level data would be really great. Secondly, along with robust MIS systems, we need good qualitative work that gives us better insight into a field. Thirdly, we need a synthesis system which basically compiles findings from different small studies and is able to better inform impact evaluators. These are just methods which are one-time investments and won’t have too many costs involved after setting up. All methods concentrate on getting the correct information out to the public.

AA: The decision regarding an impact evaluation project will involve a cost benefit analysis wherein the cost can reasonably be estimated, yet the benefits- evaluation of a program- is not easy to measure, how then are decisions taken about which evaluation should be done and which not?

BB: It’s not so much about the evaluation to be considered but about which programme should be undertaken. The costs are of the evaluation but the benefits are of the program. These benefits can actually be captured only if we find out whether it was taken up by policymakers, did it create change or inform any of the stakeholders in a better manner etc. It’s very hard to attribute a money value to these things. In this case, we look for other methods that try to calculate the benefits of the program.
RB: What are some fields in india, where you feel impact evaluation should play an essential role?

BB: Flagship programs and innovative programs that are likely to be scaled up in the future are the basic areas that definitely require impact evaluations. Innovative programs are piloted and have potential and these need evaluations to inform and assess the possibility of scaling them up. Not only impact evaluation, but they also require what’s known as process evaluation. These assess the benefits of a program going from a pilot stage to a principal stage. Impact evaluations, in themselves tell us whether a program works or not. To understand why and how it works involves process evaluations, which are embedded in impact evaluations.

AA: On a lighter note, we are curious to know, what motivated you to enter the field of impact evaluation?

BB: I’ve always wanted to work in the development sector. This is a niche field that is methodologically rigorous but at the same time, you don’t lose sight of the real problems. It allows me to do my own research, making use of my training in this field and also keeps me grounded in the realities of what’s going on.

 

You can access the presentation used by Dr. Barooah here: Teaching impact evaluations

 

Interview with Dr. Jorgen Weibull

By Antora Bhattacharya (AB) & Rhea Sinha (RS)

We are extremely pleased to share with you, our interview with Dr. Jorgen Weibull (JB). His ground-breaking research on morality in economics is intriguing and provides ample reasons to ponder over the basic assumptions of economic models. Read on to rethink economics!

RS: A lot of your work has been on evolutionary game theory and the application of game theory to the Darwinian model. What inspired you to take up such a field in economics?

JW: I was always intrigued by evolution and the beauty that nature and life has to offer. My fascination with this evolutionary phenomenon propelled me further to take up evolutionary game theory academically. I read up on this topic and I realised you don’t have to rely on fully rational agents as proposed by conventional Economic theories. It became imperative to consider that there are other forces that move us to behave in a particular way and to survive. I made an attempt to lay these other factors bare so that I could view them in a more structured manner. With the help of others and also myself, I was able to prove new results and was able to relate ‘economic thinking’ as a dominant strategy. There were many questions that needed to be kept into consideration such as, ‘What is a dominant strategy?’, ‘What is rationality?’, ‘What is common knowledge?’ and ‘What does evolution say?’. I think for me it is very important to base Economics in something more fundamental. This is because Economics is all about human species interacting with each other which indeed is a more complex phenomenon than what most Economic theories pre assume. We are biological creatures and once we gain insight about our various behaviours, we would go a long way in this discipline.

 

AB: Economics is called a science of assumptions. Sometimes, it is critiqued to be somewhat inapplicable in real life situations. How do we, as students, learn to bridge the gap between theoretical knowledge and the practical applications of these theories and ideas?

JW: This is a very important and big question and I don’t think I’m the right person to give a full answer. I think economics is extremely complex and a difficult field. It studies complex creatures like humans and what they do together. They can create companies, trade in markets; they can make families, organizations. It is very hard to make predictions about human behavior. Economics can do many things but we have much to learn. We could simply apply economic models, look at the data and see if we can explain the data. Sometimes, it works surprisingly well and sometimes, not well at all. Some market predictions of economists are amazingly accurate. For example, game theory is an amazing tool to predict the outcomes of auctions. It is powerful in some fields. In other fields, it’s lacking and there are several reasons for that. One reason is, one has to understand human motivation and it’s not enough to just think about your material self interest. Another reason is cognitive limitations. We are not infinitely smart. In economics we assume that we are infinitely smart. What is the defense for that assumption? If you want to look into that, you can read Milton Friedman’s ‘Methodology of Positive Economics’. The argument is evolutionary. It says that if firms behave not in a way to maximize their expected profits, they would be out of business or would change their practice. It says that we believe that well functioning markets will sort themselves out, like evolution. For example, if the other firms in a market raise their prices, so will an individual firm. This could maybe be an approximation for what could best be applied in game theoretic terms. We definitely need to bridge the gap.

 

RS: You have used the word ‘motivation’ repeatedly. Human beings are inherently motivated and that is what guides decisions and that is what we call a rational human being. So if you say that morality plays an important role in taking decisions, what would be essentially irrational in that case?

JW. By using the term ‘irrationality’ in economic theories, we don’t necessarily mean any particular way of action but rather that the person isn’t coherent or consistent in their actions. You should have what are called ‘transitive preferences’. This means if I prefer A over B and B over C then I should logically prefer A over C as well. If you have logic in your preferences, then there will be a utility function, which you would want to maximise. Economic theories work on assumptions rather than hard facts. We can look at ‘Homo Moralis’ which suggests that you care about what you do. For example “it’s not right to pollute here if I can avoid it, even if I understand that my contribution to pollution wouldn’t be significant due to my small size”, this suggests that you are rational. Rationality doesn’t say that it should be your material consequence that drives your actions. It simply says that there should be coherence in your behaviour and there should be some underlying structure in your behaviour. This revelation surprised me as well, I couldn’t believe it. We tried to make sense of what was happening through mathematics. We can try to think of this, as a form of morality. Essentially, if you don’t contradict yourselves and you have transitive preferences then that would fulfill the definition of rationality.

 

RS: When we hear the word morality, it’s often linked to a positive notion or idea. Also we just discussed it is evident that we link rational behaviour with consistency in thoughts and actions. If we extend this argument to that of the situation of a prisoner or any criminal, it is very much possible that he has abnormal thoughts in a consistent manner. So even though these thoughts aren’t socially acceptable, there is an element of consistency. Does this mean that the person is still rational?

JW: So let’s look at it from a revealed preferences point of view like that in Economics. You give him a set of choices between different actions and if he has transitive preferences, it is a mark of rationality. If he’s consistent with those choices then it would still be considered rational even if the choices are considered strange or abnormal. If we want to understand rationality from a deeper sense then there is something called savage rational by Leonard Savage. He wrote a book on the foundations of statistics. His notion was that by rationality we don’t only mean that we aren’t contradicting our choices but also in addition to transitive preferences we form beliefs that are consistent. So if I form a populist belief and then following that I get new information, I should be able to integrate that into my new belief and ideas. Rationality means I’m able to use Bayes’ law and updates. So using this new information I should be capable of updating my beliefs which can be positive. Therefore if this prisoner has strange views about society and what goes on in society then Savage would say that he’s not rational because his beliefs contradict the information he has. I remember that we had a terrorist act going on Stockholm and that guy had killed people by driving trucks into crowds. And I heard his trial and what he had said; his beliefs were completely false about Sweden and the nation. So maybe he was consistent in his awful thoughts of killing people but he wasn’t consistent in the information he was acting on. His thoughts were indoctrinated.

 

AB: In most undergraduate economics courses in India, game theory is introduced quite late into the program. For those who are eager to learn more about this topic earlier than it is taught, do you have any suggestions or recommendations regarding books or courses to access?

JW: I think you should study it early. Nowadays, universities in many countries teach it earlier. When I started teaching game theory, students at the masters’ level didn’t know what Nash Equilibrium was. Now, it is taught differently. There is a very nice book by Avinash Dixit, who is of Indian origin at Princeton, and Barry Nalebuff. It is called ‘Thinking Strategically’. They are very clever people and know game theory very well, but the book is written quite informally. I would think that’s a good start. There are many books in game theory for different levels.

 

 

Interview with Dr. Anirban Kar

By Khushi Gaur (KG) and Ananya Iyengar (AI)

It gives us great joy with share with you the transcript of our interview with Dr. Anirban Kar (AK), following his very enlightening talk on Behavioural Economics and its applications. Happy Reading!

AI:  Sir, in your lecture, you mentioned the concept of measuring intentions. Can behavioural economics create mathematical models to incorporate qualitative aspects like altruism, norms and reputation? Usually, economic models measure only quantitative aspects. So, can we measure intentions mathematically?

AK:  We can. In fact, the paper I wanted to cover actually does that. So, let’s take a step back. Any mathematical construction in economics has a certain structure, a certain mathematical representation of it. It’s another question, and an important question as to how much of that is empirically true. A mathematical structure will have certain parameters which are taken as exogenous. So, one particular empirical objective could be to estimate those exogenous parameters: compute that from real life data. A second objective could be- even after computing that, to check that the structure that we had is really capturing that qualitative aspect you were talking about. The empirical validity of a theoretical model will always remain a question.

So, think of it this way. As economists, we are observing certain real-life situations and theoretical economists essentially are trying to tell a story- about why such situations occur. No matter how mathematically you represent it, it is ultimately a story. You are only expressing that story in terms of some mathematical equations. That mathematical representation is important because the implications of the story has to be consistent with the assumption of the story so that we can make sure that our conclusion is consistent with the starting point. Consistency is important.  So, all that theoretical-mathematical models are doing is telling a consistent story; making sure that you didn’t bring in something you didn’t specify. Often, with the verbal telling of a story, you can bring in certain aspects you didn’t start off with. Here, there is a rigorous check that you have actually specified everything that you need for that story.

 Now, the next question is whether the story is the right story or not. That is the job of empirical economics. And, that is a major area of research right now. In the last twenty to thirty years, the discipline has changed. In the times before 2000, the emphasis was on theory.  Now, it is entirely that empirical aspect of those theories that is being investigated.

KG:  Sir, behavioural economics has often been critiqued for its paternalistic approach. It can be used by parties in power to reap more dividends-electoral and strategic. Some say that an economic, imperialism has come about. So, in this context, how can this criticism be justified and how can the activities of nudge units be regulated to prevent manipulation of behaviours in the self-interest of the state?

AK:  See, in the way you posed the question, I think you have a starting assumption that paternalistic behaviour is bad.  Let us detach this from behavioural economics and go to the standard rational theory. A paternalistic attitude can be embedded in any regulation. It doesn’t have to be the behavioural understanding of human behaviour of the “rational” understanding of human behaviour. So, the first question I will take, the way I interpret it, is about the understanding that paternalistic behaviour is bad. I don’t agree. Paternalistic behaviour, as it is, can be studied in different contexts. For example, smoking. Public smoking is banned and a very heavy tax is imposed on cigarettes; and this is of course coming from a paternalistic attitude, that you are doing harm to yourself as well as to others as it is a public bad. So, we are making a value judgement, right? That something is not good for you. And that is what you are calling paternalistic.

Let us think about it. Not making this value judgement is also a value judgement in itself. When economists pretend that they are not making a value judgement, they are not letting nature ease on its own. That’s not what they are doing. When they argue that no rule is to be imposed, say no tax is to be imposed on cigarettes, they are also making a value judgement. That can take two forms. One, that we have no business is telling people what to do. That could be one moral point. The second philosophical point is that whatever people do is the right thing. Based on that, they are making a value judgement that interfering is not the right thing to do. That itself is a value judgement. So, all this pretension that the 101 Economics textbook is not making value judgements is rubbish. Now, think of this. What is the starting point of economics? Property rights. Isn’t that a value judgement; that property rights are someone’s fundamental rights? Whenever we are deciding that the state has no role in XYZ, it is on the basis of a value judgement.

So, it is not the value judgement that has to be questioned. I think the real question is what is the right value judgement and what is the wrong value judgement. We need to go beyond this façade that value judgements are wrong. Every minute we make value judgements. Every policy is a value judgement. Not implementing a policy and leaving it to the free market is also a value judgement. So, we have to go beyond making such comparisons. If there is no intervention in anything, what you are saying is that the market price, given that you have ensured property rights, takes care of everything. That itself is a value judgement. Think of another extreme. The idea of justice- whether you are right of wrong- is up for bidding.  Whoever can pay the highest price can get the judgement in his or her favour. No economist would go that far. Are not they making value judgements here? This blanket vilification of moral judgement is wrong. We need to think carefully about what spheres the state should intervene in. For example, it is okay for the state to intervene in the smoking behaviour. But it cannot intervene by saying that non-veg food is banned. So, we have to make a distinction there and think thoroughly about why one intervention is right and why one intervention is wrong.

AI:  In 2016, the NITI Ayog came up with a Nudge Unit so that behavioural economics could play a major role in policy making in India. However, until now a large-scale impact hasn’t been made. In the Indian context, especially since the country is so diverse and as you said we have things like lynch mobs happening in the country. So what sort of role can behavioural economics have in policy making in the future?

AK:  Well, I don’t differentiate between behavioural economics and say, the neo-classical paradigm. I would say the broader question is what role economics can play in policy making and there I think that indeed economics can make a difference. What behavioural economics is doing is that it is broadening the horizon and if we think of it that way, then naturally it has some role to play. Once again, the question is what kind of role it is playing and once again there’s no general answer and you have to look and understand it on that basis. Behavioural economics is allowing for such possibilities that norms can be different in different places and there is a possibility that it can be more effective than the standard solution. However, that being said, there is a lot to learn from the rational paradigm. Behavioural economics should not become a fad. It is important if it is saying something extra but if it is not, then there’s no point in jumping into this fashion of behavioural economics.

KG:  The most famous experiments on the nudge theory have been implemented on a small scale in privileged countries. For example, in Spain, the organ donation rates have increased because of automatic enrolment and in the UK, the savings in the private sector have increased due to enrolment by default. So, can behavioural economics help in solving basic fundamental problems in third world countries like unemployment, poverty, crime?

AK:  Well, once again, I wouldn’t suggest behavioural economics. Economics has to get richer and so it’s an interaction of many things. Behavioural economics is identifying patterns everywhere. You need to ask where this pattern is coming from, what is triggering such behavior? So, it must be embedded in the institutional structure and the ‘political economy’ structure of a place. We need to establish that link between that institutional structure with the behavioral pattern and the feedback on the institutional structure. So, this is a complicated cycle and it’s an economist’s job to understand this cycle. Behavioral economics has opened up that possibility. It says that if you change the institutional structure, the nudge experiments there, you can change the behavior. But that change in behavior has an impact on the institutional structure as well. To take a concrete example, if you have a more democratic country where your voice is heard, you might feel that a judge can change your aspirational level and that can have an effect on whether you can move out of the poverty trap or not. However, in a developing country like ours, the constraint might just not be my ambition. There might be a real poverty trap embedded in the political economy structure. So, my lack of aspiration is probably holding me back, that’s true, but I need to understand why I don’t have that aspiration. Is there a lack of democratic structure in the society which is stopping me from having that aspiration and hence preventing me from getting out of that poverty trap? So, the more you understand these linkages, you’ll be better informed about these policy changes.

AI:  Economics, as a discipline, is going in a completely different direction due to behavioural experiments and increasing technologisation of the workforce. So as students, what can we expect the future of economics to look like?

AK: (A pause and then laughs) I don’t think economists are good at making predictions! They’re really bad at explaining things. So, let’s just not predict the future!