In 2020 a guest post on this blog noted the launch of the Social Science Prediction Platform (SSPP). This platform is intended to facilitate the systematic collection and assessment of forecasts of social science results. We are both on the advisory board of the SSPP, and have used it in some of our own work, so have had a chance to see its evolution and growth, and thought it might be a good time to share an update and give examples of how it is being used and could be used to improve development economics research.
A brief summary of what the platform does
The SSPP provides an online infrastructure that you can use to collect predictions for your own studies, as well as provide your predictions for other studies – typically predictions of treatment impacts, but they could also be predictions of other outcomes or statistics of interest as we will illustrate in our example. It has templates you can use, as well as integration with Qualtrics, to enable easy writing and hosting of a prediction survey. You can then send the survey to a sample of your choosing to collect their predictions or make it open to the public (or both). One of the innovations over the past couple of years is also to put together a panel of just over 60 superforecasters who are graduate students, academic researchers, and faculty that are committed to regularly providing predictions to surveys on the platform. This ensures that all surveys now published can get a decent number of responses. You can also sign up and make your own predictions on open studies. And of course, the SSPP is a public good and is free and open to use.
The SSPP just reached 100 projects on the platform, with almost 10,000 forecasting surveys taken to date, and a median of 79 forecasts per completed project. There are also now more than 50 published articles or working papers that use these forecasts, which start to show some of the variety of use cases that people are finding for collecting them. We pick a few examples of how this can be useful for development economics, while noting that similar uses apply for other impact evaluations (and lab experimental studies). The forecasts are also being used for a whole methodological meta-research agenda on learning across multiple studies on how to best collect these forecasts, and what determines the accuracy with which people can forecast results.
1. Dealing with the “isn’t this obvious?” objection, especially to benchmark null results
Duncan Watts has a book called Everything is obvious: how common sense fails us, which documents how people can often immediately think of an explanation for why whatever result they are told is ex post obvious (even when they are told the wrong answer). We both speak from experience that there can be a tendency with international development interventions to either say “well of course, it’s obvious that giving people stuff makes them better off” when a positive treatment effect is found, or “it’s obvious that we wouldn’t expect such an intervention to do much since it doesn’t fix other underlying constraints” when no significant effect is found. One common use of the SSPP is to document ex ante what experts/policymakers/your target audience thinks the results will be before anyone sees the results, to see how much “news” there is in the results. This can be particularly useful when you find a null result: a null result for an intervention that people were pretty certain would work may be more interesting and easier to publish than a null result where everyone expected a null result (you can of course think of exceptions).
For example, in work on attempting three different interventions to reduce the flow of irregular migrants from The Gambia, David and co-authors used the SSPP to elicit predictions about the likely impacts, finding the estimated impacts were smaller than experts predicted. Deserrano et al. (2024) use the SSPP to compare predictions of which of three different types of incentive structure for community health workers and supervisors in Sierra Leone would be most effective, using this to document that most successful treatment differs from what most people thought ex ante would be the best. In a recent working paper on housing subsidies to refugees in Jordan, Ted and co-authors summarize their primary outcomes along with what researchers and policy experts predicted in the figure below. They find null and even negative effects for some primary outcomes, with impacts typically smaller (but overlapping with) predictions. This type of figure also shows how the predictions can be used to benchmark the results and how they can be displayed without needing to add another figure or table.
2. Corollary: potentially using this to modify or change the intervention
Most of the time when we launch interventions and run studies, it is because we think/hope that they will work. So imagine designing an intervention, putting it up on the SSPP, and then finding that almost everyone expects it to have no effect, or such a tiny effect that you would not be powered to detect it with the research budget in hand. While this would be disappointing, if the predictions are collected ex ante, you might have time to rethink the intervention logic, the theory of change, and either convince yourself that they are all just haters, or else to consider what changes you could make to strengthen the intervention or increase sample size. With multiple treatments, this approach could also be used to help prune from a list of possible treatments to determine which ones to test.
3. Predicting context, as well as treatment effects
Rosenzweig and Udry (2020) show how much the returns to different investments can vary with aggregate shocks such as rainfall and price fluctuations. Many of us have recently been in the situation of interventions being affected by COVID-19, which affected the effectiveness of many interventions (as well as the ability to implement them). If we think that the impacts of our interventions are likely to vary depending on the macro context, then the SSPP could also be used to understand what people were expecting about this context. For example, in their paper on reducing irregular migration, David and co-authors designed the intervention at a time when irregular migration flows from The Gambia were the highest per capita in Africa. However, a combination of a change in government and European asylum laws, coupled with the pandemic overlapping with part of the follow-up period, meant that irregular migration fell by itself by the time the intervention occurred. We didn’t think ex ante to ask expectations of the control group migration rate, but ex post documented that local experts still believed it was much higher than it actually was. You might imagine going further and eliciting a distribution of anticipated treatment effects under different macro environments to see how sensitive people expect the results to be to these conditions.
4. Using the predictions as formal priors in Bayesian impact evaluation
In this blog post, David discusses how eliciting not just point estimates, but entire distributions can be used as informative priors in Bayesian analysis. Here the knowledge of policymakers or researchers can enter as an additional input in estimating treatment effects. This also relates to a broader research agenda of learning how policymakers and academics update their beliefs when faced with new evidence (e.g. Vivalt and Coville, 2023).
What are some other SSPP potential uses?
The SSPP actively welcomes new studies on the platform and its uses are evolving. A couple of development-specific thoughts that might be of interest for people to pursue:
· Incorporating more the predictions of low and middle income country participants in interventions: most of the studies are eliciting predictions from graduate students, academics, or, on occasion, policymakers. But decisions of whether to enroll in a study, or how much effort to exert in an intervention will depend on the expectations of study participants as to the likely treatment effects. Here researchers may wish to consider whether they want to ask participants about their predictions of the effect on the average participant in the study (as David and co-authors did for a sample of firms in their Colombian study) and/or for their predicted individual treatment effects. An even earlier example of this approach conducted among urban residents in Kenya (carried out before SSPP was set up) is Thomas et al (2020).
· These predictions of individual treatment effects (when appropriately incentivized for truthful reporting) could then be potentially used to better select which individuals to include in a program, as in Hussam et al. (2022).
source : World Bank Blog
