RCTs in Development: A critical perspective Book Launch

10 December 2020

You can find a recording of the meeting here.

Moderator note: The book is an edited volume, not a co-authored work. The views expressed here are therefore those of the editors and the individual authors named and who participated in the IDEAS-facilitated book launch event on 10 December 2020. They do not represent the views of the other chapter authors.

You can also download a PDF file with all questions.

Approach to the book

• Why was EVALUATION not included as a discipline debating RCTs?
• How many RCT studies did the authors analyse to inform their findings and conclusions?

Suitability of  RCTs for humanitarian and development

• What did you find about how suitable RCTs are for emergency situations? Are there ethical and human rights considerations particular to their use in such situations?
• Considering the actual experiences in the world, did RCTs (in social policy, not clinical trials) contribute in any respect to emergency measures implemented?
• How appropriate is the use of RCTs in the context of LICs (or LMICs), given the practical challenges that accompany their execution in low-resource, low-capacity and often fragile institutional and societal contexts? What considerations are then most important?
• And what can you say about the cost of RCTs and their application in such countries compared to “developed” countries, and across sectors?
• What are your perspectives on the following statement? “The historical four-fold transformation of countries as paradigm of development is just as kinky, but on another level. True development will also require transformation of global relations, including solving global crises like pandemics, climate change, biodiversity loss and plastic waste, economic exploitation and so on. I see no role for RCTs there either.”
• Given Lant’s arguments, are most international relief and development organizations (from Save the Children, World Vision, CARE International, etc.) “kinky” organizations – using large amount of funds for inconsequential “tweaks” and “twists” until real development occurs?
• A question for Lant: Do your data and argument mean that the degree of income inequality is the same in each country? Is it not a counter intuitive conclusion?

Polling results

• The following were the results from the polls during this book launch event. How do they compare with earlier launch events? What are the “correct” responses?

Methodologies, and ideologies about methodologies

• What are your perspectives on the following comment? “The controversies around RCTs are often cast in methodological terms. However, the central issue is more a matter of how comfortable we are in dealing with uncertainty in our knowledge. Many of the strong proponents of RCTs are perhaps very uncomfortable with the idea of epistemological fallibilism.“
• How are the issues you are raising with RCTs not also present in most other available methodologies and what are your proposing be put in their place to simultaneously achieve causal identification and to solve the issues you are raising. This conversation feels like a reductive view of RCTs reminiscent of the same debate we were having ten years ago. No so-called randomista would use the term “gold standard” or suggest using the tool widely. Why is micro work and systemic change mutually exclusive? Why can’t qualitative work (ex-ante, during, and ex-post) and implementation research as components of RCTs not complement impact estimates? In fact, how well does the book go beyond the “usual” claims about the weaknesses of RCTs that randomistas generally oppose?
• Are the “randomistas” answering to the methodological and ethical critiques put forward in the book? Is there at present a playing field for scientific confrontation?
• Did any author change or modify their view about RCTs in some way by participating in this book, given that they would have read one another’s work?
• Among others, why doesn’t James Heckman’s thinking temper the fervour of randomistas – or does it?
• To what extent do you agree with the following comment? “I would add that as we have entered an era of global systems change, mirrored by the problems in the SDGs, the conventional fixation on singles measures for complex issues is increasing being challenged – reflected by this very webinar. The transformational Global 2030 Agenda requires complex systems analysis that recognizes the inherent interdependencies and thus complexities of the SDGs. This is problematic for evaluation methods such as RCTs, that rely on assessing effects of single treatments on individual outcomes that control for all other observable and non-observable characteristics that can influence the casual change process. Such approaches fail to assess the effects of multiple inputs on multiple outcomes that characterize large-scale, open-system comparisons required for highly complex challenges like climate change, chronic poverty, etc. They also miss out on unintended outcomes outside the casual logic, which is critical for transformation, which, in addition to being.”
• Do these debates on RCTs not raise the question of models of behavior and theories of change – and how do the authors view this?

RCTs and ethics

• To what extent do RCTs always note and absolutely mitigate ethical issues and their consequences on participants in the long run?
• What are your perspectives on the following comment? “There is an ethical consideration related to knowledge production – namely, the failure to acknowledge uncertainty and complexity is not simply a technical error but also an ethical one. It is an ethical failure when we fail to take responsibility for our knowing. Ethics is not something that is supplementary to our understanding of the world. Ethics is always already part of what we do.”
• Ariane, can you clarify the following: In the bail example you mentioned, would you say it is unethical if you WOULD have the money to pay bail for everybody in need and you CHOOSE not to – but if you have limited resources (as I assume it is the case) I see no wrong in assigning randomly to the program?
• How should we regard the ethical situation when an intervention involves providing incentives – as many RCT-focused interventions do – among destitute participants who in practice really cannot say “no” to any donation? How problematic is this for issues of consent?

The bottom line

• What is, from your perspective, the “bottom-line” statement about the usefulness of RCTs for impact assessment in development?