Over the past few decades, economics has undergone a ‘credibility revolution’, emphasising techniques to establish causal relationships. This column analyses over 44,000 working papers over 1980-2023 to assess the evolution of these methods. Papers with a higher proportion of causal claims and those introducing novel causal relationships have a higher likelihood of a top-tier publication. However, the use of causal methods does not necessarily lead to a higher citation impact once published. These findings suggest the need for a more holistic approach to research, combining methodological rigor and broader academic impact.
Economists play a crucial role in informing policies on pressing issues such as inequality, education, and public health. Over the past few decades, the discipline has undergone a ‘credibility revolution’, emphasising rigorous programme evaluation techniques to establish causal relationships. This shift has enhanced the credibility of economic analyses, but could, on the other hand, have had broader implications for what could broadly be considered marketable research in economics.
The rise of causal inference methods
The credibility revolution is characterised by the adoption of empirical strategies designed to strengthen causal claims. Seminal works by Angrist and Krueger (1991) and Card (1990) introduced natural experiments and instrumental variable techniques to address endogeneity concerns, laying the groundwork for more credible causal inference. Subsequently, methods like difference-in-differences (DiD), regression discontinuity design (RDD), and randomised controlled trials (RCTs) have gained prominence, signalling a paradigm shift toward ‘design-based’ empirical strategies (Angrist and Pischke 2010, Pischke 2021).
To assess the evolution of these methods, in a recent paper (Garg and Fetzer 2024) we analysed over 44,000 working papers from the NBER and CEPR spanning 1980 to 2023. Our analysis reveals a significant increase in the use of causal inference methods over the past four decades. Figure 1 illustrates the proliferation of key empirical methods used in these papers.
Figure 1 Proliferation of empirical methods over time in NBER and CEPR working papers
Notes: This figure shows the increasing use of methods such as DiD, IV, RCTs, and RDDs from 1980 to 2023. DiD usage rose from about 4% to over 15% of papers, reflecting its utility in exploiting policy changes and natural experiments. RCTs grew from near zero to over 7%, highlighting the rising feasibility and acceptance of experimental designs in economics.
To visualise how these methods contribute to constructing economic narratives, we use knowledge graphs to map the relationships between concepts in economic research. Figure 2 presents an example of such a knowledge graph from Banerjee et al. (2015), illustrating the causal impact of introducing microfinance in India.
Figure 2 Knowledge graph of Banerjee et al. (2015)
Notes: This graph visualises the causal relationships identified in the study, where nodes represent economic concepts mapped to JEL codes, and edges represent causal links evidenced using RCTs. The introduction of microfinance leads to increased business investment and affects various household outcomes, illustrating a complex causal narrative with multiple interconnected effects and a longest causal path length of 3.
In this knowledge graph, the authors examine how access to microfinance influences a range of outcomes, from business creation to household expenditure patterns. The high number of causal edges and unique paths indicates a rich and interconnected causal narrative, reflecting the complexity of economic relationships explored in the study. Such detailed mappings can help understand how empirical methods contribute to advancing knowledge in economics.
Publication success versus citation impact
Despite the methodological advancements, there is an ongoing debate about the implications for research dissemination and influence. A particular concern may be that the credibility revolution has given rise to a specific style of economic research that may put more emphasis on the methodological toolbox, rather than the underlying question that policymakers and decisionmakers have to contend with on a day-to-day basis (Jiménez-Gómez et al. 2019). Further, assessing what is economically significant, vis-a-vis what merits consideration on statistical grounds may lead to publication bias, disadvantaging studies that, for example, produce null findings (Chopra et al. 2022) or generate a broad range of theoretically consistent, high dimensional empirical patterns on a broad range of variables of interest that may be jointly significant when viewed as being embedded in a causal chain or a graph.
To explore this, we use knowledge graphs to represent the relationships between economic concepts in each paper. We quantified narrative complexity through measures such as the number of unique causal paths and the depth of causal chains. Our findings suggest a nuanced relationship between methodological rigor, narrative complexity, and research impact.
Figure 3 shows that papers with a higher proportion of causal claims are more likely to be published in ‘top five’ economics journals. Additionally, papers introducing novel causal relationships and engaging with less central, specialised concepts have a higher likelihood of top-tier publication.
Figure 3 Knowledge graph measures and publication outcomes
Notes: This figure displays the relationship between various knowledge graph measures — such as the proportion of causal edges, narrative complexity (measured by unique paths and path length), and novelty — and the likelihood of publication in top journals. Papers with higher narrative complexity and more causal claims are favoured by top-tier journals, indicating a preference for innovative methods and intricate causal narratives.
Yet, when examining citation counts – a proxy for academic influence – we observe a different pattern. As depicted in Figure 4, while the complexity of a narrative positively correlates with citation counts in top journals, the use of causal inference methods does not necessarily lead to higher citation impact once published. Instead, papers focusing on central, widely recognised concepts tend to receive more citations.
Figure 4 Knowledge graph measures and citation counts by journal category
Notes: This figure illustrates how narrative complexity and engagement with central concepts relate to citation counts across different journal tiers. While complex causal narratives are associated with higher citations in top journals, engaging with central economic concepts enhances citation impact across all journal categories, suggesting that broader relevance drives academic influence.
This divergence suggests that while top journals prioritise methodological innovation and complex narratives, broader academic impact is driven more by the relevance of research topics. This raises important questions about the direction and priorities of economic research, highlighting the need for a balance between methodological rigor and engagement with central economic debates (Deaton and Cartwright 2016, Pischke 2021). There is a concern that prominently published research in leading journals could encourage a shift in research focus into areas that may be of marginal broader interest, possibly creating deep ‘rabbit holes’ that may subsequently generate a self-reinforcing publication dynamic, hindering innovation more broadly.
Challenges in replication and data accessibility
The increased emphasis on sophisticated empirical methods brings challenges related to replication and research transparency. For example, Chopra et al. (2022) find a substantial perceived penalty against null results in the publication process, which can distort the scientific record and hinder cumulative knowledge. Such biases can lead to an overrepresentation of significant findings, inflating false-positive rates and undermining the reliability of published research (Brodeur et al. 2016).
Moreover, we observe a rise in the use of proprietary data, with the proportion of papers using private company data doubling from about 4% in 1980 to over 8% in 2023. The use of private data in fields like finance and industrial organisation exhibit the highest proportions. Proprietary data can provide granular insights, but it can also raise concerns about replicability and transparency. Limited access to such data hampers other researchers’ ability to verify findings or explore alternative hypotheses (Jiménez-Gómez et al. 2019). Further, the provision of research access to proprietary private data may be skewed towards academics with a broad profile, which could further exacerbate the inequalities in the profession in terms of research access (Fetzer 2022). Alternatively, companies could strategically use (publicly funded) researchers to produce private knowledge goods, outsourcing research and development. Alternatively, they may leverage the credentials of academics or higher education institution to foster brand recognition or to boost corporate social responsibility credentials strategically (Bounie et al. 2021).
Deaton and Cartwright (2016) caution against overreliance on randomised control trials (RCTs) and emphasise the importance of understanding the mechanisms behind observed effects. They argue that without a theoretical framework, findings from RCTs may not be generalisable to other contexts, limiting their policy relevance. The generalisability and scalability of experimental results are crucial for informing policy decisions (Jiménez-Gómez et al. 2019).
Implications for the economics profession
These findings have significant implications for the economics profession. The trade-off between methodological rigor and broader academic impact suggests the need for a more holistic approach to research. As Jiménez-Gómez et al. (2019) argue, experimental economists must tackle the generalisability and applicability of their evidence, ensuring that findings contribute meaningfully to theory and policy discussions. This involves embracing diverse methodologies and focusing on questions with substantial policy relevance (Deaton and Cartwright 2016).
Encouraging transparency and the reporting of null results is essential to maintain the integrity of the scientific process. Miguel et al. (2014) advocate for practices that enhance credibility and accessibility, such as pre-registration and data sharing. Addressing the challenges posed by proprietary data requires balancing the benefits of rich datasets with the need for verifiable and replicable research. Initiatives promoting open science and replication studies can help mitigate these issues (Jiménez-Gómez et al. 2019, Brodeur et al. 2016).
Furthermore, there is a growing recognition of the limitations of focusing solely on statistical significance. As Brodeur et al. (2016) highlight, an overemphasis on significant results can lead to ‘p-hacking’ and inflate false-positive rates. Adopting robust statistical practices and valuing studies based on their methodological soundness and relevance, rather than just significant findings, can mitigate these issues. Emphasising the economic significance and practical implications of research findings is vital for advancing the field (Chopra et al. 2022).
Source : VOXeu
