Sample surveys have been a fundamental tool for capturing policy-relevant heterogeneities among agents, especially firms, and remain crucial for providing insights into key analytical issues. This column argues that the digitalisation of information systems, the rise of artificial intelligence, and the growing availability of administrative and big data offer significant opportunities for improving surveys. Integrating these data sources with carefully designed sample surveys can enhance their value, particularly in areas like sample selection, weighting, and questionnaire design. Advances in AI and machine learning also offer potential benefits throughout the survey process, providing promising directions for further exploration and research.
Carefully designed sample surveys have long been an essential tool for capturing policy-relevant differences between groups of economic agents in a timely manner. Despite their crucial role in informing policy, surveys face growing challenges, including declining participation rates and the emergence of faster and more competitive data sources. In a recent contribution (Banca d‘Italia 2024a), we discuss the advantages and limitations of traditional business surveys, and then explore new directions to develop ‘business surveys 2.0’ to seize the opportunities presented by new technologies and digitalisation.
The usefulness of firm-level surveys for policymakers has become so clear that an increasing number of central banks from both sides of the Atlantic have added them to their analytical toolkits. In the early 1960s, Banca d’Italia, which has a long tradition in this field, launched the Survey on Household Income and Wealth, and soon after, in 1972, the annual Survey of Industrial and Service Firms (INVIND); in 1999, the toolkit was expanded with the quarterly Survey on Inflation and Growth Expectations, which systematically collects firms’ expectations of consumer price inflation at different time horizons, as well as quantitative information on past and expected changes in their own selling prices. Other institutions have also initiated their own business surveys on specific topics, such as the Survey of Business Uncertainty by the Federal Reserve Bank of Atlanta and more recently the Survey on the Access to Finance of Enterprises (SAFE) by the ECB.
Business surveys remain crucial to this day for providing timely insights into key analytical issues (Banca d’Italia 2024b). As shown in many academic papers, they have allowed economists to achieve a deeper understanding of expectations (see for example, Manski 2004, Bartiloro et al. 2019) and their relation with firm-level dynamics (Coibion et al. 2020, Rosolia 2024), of uncertainty (Guiso and Parigi 1999, Altig et al. 2022), and other outcomes (for the interpretation of the ECB’s inflation target by firms, see Bottone et al. 2022b) that would have been otherwise unattainable. Because they can provide timely, specific, and flexible information, business surveys have proved to be particularly useful in recent years, which have been marked by large, unexpected shocks. During the COVID-19 pandemic, for example, surveys were crucial for understanding the underlying heterogeneity that drove the surge in bank borrowing and deposits (Bottone et al. 2021). More recently, in a world of heightened geopolitical fragmentation, they have been useful to understand the potential impact of supply disruptions from high-risk countries on European regions, sectors, and firms, and, thanks to the coordinated efforts of several central banks, they have allowed researchers to conduct harmonised cross-country analyses (Balteanu et al. 2024, Borin et al. 2024, Panon et al. 2024).
However, conducting traditional business surveys based on sound methodologies such as probability sampling is becoming increasingly challenging and costly for central banks (Veronese et al. 2020). Examples of key challenges include the rise in both the item non-response rate – which has tripled to 15% for investment plans since the inception of the Banca d’Italia INVIND survey (Figure 1) – and the unit non-response rate, especially among firms that perceive a high response burden (Bottone et al. 2022a). Moreover, the growing awareness and concerns about data privacy and time constraints (for example, Galesic and Bosnjak 2009) not only complicate the conduct of surveys, but also pose serious risks to data quality and inevitably raise questions about how to use and potentially adapt sample surveys in the future.
Figure 1 Features of the INVIND survey
Sources: Survey of Industrial and Service Firms (INVIND), Banca d’Italia, Statistics Series. The number of variables represents the total number of questions included in the questionnaire, while the share of missing data is calculated as the proportion of item non-responses on expected investment relative to the total number of firms participating in the survey.
Where could we go next?
The digitalisation of information systems, the increasing availability of administrative data, big data, non-probabilistic and online surveys, which may seem like quick, easy, and inexpensive ways to collect data (Gambacorta et al. 2018, Neri and Zanichelli 2020, Cummings and Tedeschi 2024), as well as the rise of artificial intelligence (AI), bring potential opportunities for the development of surveys that must be carefully assessed. There are some promising directions. For example, full advantage can be taken of the growing availability of administrative and other unstructured data. Combining these data with methodologically sound and carefully designed sample surveys can greatly enhance the informative value of both. This integration can occur at various levels, such as sample selection, weighting, and streamlined questionnaires. In addition, it is important to harness the rapid advances in AI and machine learning techniques, which have potential benefits at various stages of the survey process. In the latter area, it is possible to identify some avenues for further exploration and research.
Survey research focuses primarily on the wording of questions and answers in an attempt to gather soft information that is not readily available in more traditional data, or ‘hard information’. Thus, a first way to improve traditional surveys relies on the use of large language models (LLMs). In the preparation phase of surveys, for example, LLMs could help create questions, identify inconsistencies, and suggest effective response options. In data cleaning and management, LLMs could help detect inconsistent responses and prevent low-quality entries. In the final stage of reporting results, LLMs could help ensure accessible formats, whether in the form of summaries, visualisations, presentations, or written text.
A second way to improve traditional surveys is to use natural language processing and text analysis to quickly extract signals from text. These methods are already widely used in the literature to extract signals about economic variables, such as industrial production, from text analysis of survey responses from manufacturing firms (for example, Cajner et al. 2024), or economic activity from earnings calls (Gosselin and Taskin 2023), which are an important channel of communication between market participants and the management of publicly traded companies. In light of these experiences, the possibility of extracting information from informal interviews with managers that are typically more difficult to interpret might be further explored, moving away from traditional surveys in which respondents read and select the most appropriate answer.
Third, even in the conduct of more formal interviews, surveys could be facilitated by the use of AI-assisted interviewing, which integrates advanced artificial intelligence into the interviewing process, particularly as a tool to assist interviewers in improving data quality (minimising human error and bias). Such a development would enable more dynamic and responsive interviews that adapt in real time to respondents’ answers, automating routine tasks, providing a more engaging and personalised interaction, possibly leading to higher response rates and greater satisfaction with participation.
Fourth, machine learning models could replace current methods for imputing missing data in business surveys, as they are able to capture the complexity in data structures, recognising non-linear relationships and allowing a large number of predictors to be included. 1 Machine learning techniques could also help to detect outliers in surveys, with algorithms that compare responses within the same questionnaire and across waves.
Finally, LLMs could, in principle, revolutionise traditional data collection methods. For example, instead of sending out questionnaires, one could provide a programme that firms could adapt to their own databases using LLMs, allowing them to extract the hard information they need, leaving more time for qualitative or soft information, such as plans and assessments. Clearly, the significant benefits of reducing the burden on respondents would have to be carefully weighed against the potential problems of data confidentiality that might discourage firms from participating in the survey.
These are just some examples of the potential evolution of business surveys. Understanding whether and how new technologies and tools could be integrated into survey production processes will surely be a challenge, but it is a task worth embracing.
Conclusions
Sample surveys are an essential tool for economic analysis, providing timely and otherwise unavailable information on the behaviour and expectations of economic agents. Technological advances and the increasing availability of administrative and big data offer significant opportunities to improve the conduct of surveys and to reduce the perceived burden on respondents.
Source : VOXeu
