Large language models increasingly write the code behind empirical research. This column reports a benchmark of 40 econometric tasks in Stata, R, and Python. For each task, two leading models write a script, as a chatbot that never runs it and as an agent that runs and revises it. Letting the models run their code raises the share of correct results from 79% to 97% and makes the prompt and the software matter far less. The failures that remain under the agent come from scripts that run without errors but return wrong numbers, and detecting them is still the researcher’s work.
Economists are debating what artificial intelligence will do to their own research. Weder di Mauro (2026) argues that AI is raising the supply of competent papers while the number of slots in the top five journals remains fixed. Vansteenkiste (2026) suggests that AI may make the researcher’s role less about having the right answers and more about asking the right questions. Much of that debate is about how research is judged and what role remains for the researcher. This column concerns a more mechanical layer, the code that turns a dataset and a method into an estimate.
Code written by people has never been free of errors. An audit of 110 studies recently published in prestigious economics and political science journals finds coding errors in about a quarter of them, and such errors can be present even when the posted code reproduces the published numbers exactly (Brodeur et al. 2026). Economists often delegate estimation code to research assistants, and delegation does not remove the need to review it. Large language models can now write this code, and, like assistants, they can introduce such errors. Each generation of AI coding tools increases the volume of code that developers produce, while their software releases rise by far less, in part because AI-written code still has to be reviewed, tested, and integrated by people (Demirer et al. 2026).
Many users interact with language models through a chat window (Chatterji et al. 2025, Johnston et al. 2026) where, if the model answers with a script, the researcher must run it and request any corrections. Coding agents such as Claude Code and Codex are programs that surround the model, execute what it writes, and let it read the output and revise its code until it decides that it is finished. Korinek (2025) shows how economists can use such agents and build their own research agents. In a recent paper (Galiani et al. 2026), we ask how moving from a chat window to a coding agent changes the correctness of econometric code and what this implies for empirical researchers.
Ash and Hansen (2023) called for evaluating language models on the tasks that economists use them for, and econometric coding is one such task. We built a benchmark of 40 econometric tasks covering linear regression, difference-in-differences, instrumental variables, regression discontinuity, discrete choice, survival analysis, and time series. Each task states the method and comes with a simulated dataset and a reference output computed in advance. A run succeeds when the numbers it reports match that reference within a tolerance. We ran every task in Stata, R, and Python, with and without an example of a related task in the prompt, and under three degrees of agency, which differ in how much of the running and revising falls to the model.
Under the chatbot, the model writes one script and never sees it run. Under self-repair, we run the script and, if it fails to execute or its output lacks a required value, return the output so that the model can revise the script, up to three times. Under the constrained agent, so called because it has only a command line and its prompt confines it to the task, the model runs and revises its code as often as it chooses, is instructed to verify its calculation with an independent check, and decides when it is finished. We repeated every combination 10 times with each of two models, Claude Sonnet 4.6 in Claude Code and GPT-5.6 Luna in Codex, for 14,400 runs in total.
Claude Sonnet 4.6 returns a correct result in 81.4% of chatbot runs, 93.5% of self-repair runs, and 99.2% of agent runs, while GPT-5.6 Luna moves from 77.3% to 90.3% and then to 95.1%. Self-repair, with at most three revisions, already achieves 68% of the agent’s improvement over the chatbot for Claude Sonnet 4.6 and 73% for GPT-5.6 Luna.
Figure 1, which pools the two models, shows that the prompt and the software matter less as agency increases. An example in the prompt (Brown et al. 2020) raises task success under the chatbot by 5.8 percentage points for Claude Sonnet 4.6 and 7.3 points for GPT-5.6 Luna but adds 0.7 and 1.5 points under the agent (Panel A). The choice of software shows the same pattern (Panel B). Under the chatbot with no example, Stata trails Python by 25.5 percentage points for Claude Sonnet 4.6 and by 30.8 points for GPT-5.6 Luna. Yet Stata is the most widely used software in empirical economics (Vilhuber et al. 2020). Under the agent with no example, the gap falls to 1.3 points for Claude Sonnet 4.6 and 6.8 points for GPT-5.6 Luna.
Figure 1 Task success by agency condition
We call a failure loud when the script does not run or returns no complete output, so that the researcher sees an error message or a missing result. We call it silent when the script runs and returns its output, but at least one of the numbers it reports is wrong, so that nothing in the output flags the error. More than two-thirds of the silent failures whose scripts we read come from a coding error in an otherwise correct method, such as a wrong index or a wrong term in a formula. Under the chatbot, most failures are loud. The example mainly reduces loud failures, and Stata scripts produce them far more often than scripts in R or Python. The agent removes loud failures in all three environments, so that few failures remain for the example to prevent and most of the Stata gap closes.
Pooled over the two models, loud failures fall from 15.8 per 100 runs under the chatbot to zero under the agent. Silent failures fall from 4.9 to 2.9 per 100 runs (Figure 2). Total failures therefore fall from 20.7 to 2.9 per 100 runs, and every failure that remains is silent. The fall in silent failures comes almost entirely from Claude Sonnet 4.6, from 4.7 to 0.8 per 100 runs, against 5.1 to 4.9 for GPT-5.6 Luna. Under the agent, the task with the most failures is shift-share instrumental variables, where every failing run returns all its estimates but reports a standard error outside the tolerance.
Figure 2 Loud and silent failures by agency condition
Running and revising code requires additional model calls, valued here at API list prices. For Claude Sonnet 4.6, the mean cost of a run rises by a factor of 3.1 between the chatbot and the agent, from 4.56 to 14.09 cents. For GPT-5.6 Luna, the factor is 5.0, but an agent run still costs less than one cent on average. The median agent run also takes 5.8 times as long as the median chatbot run for Claude Sonnet 4.6 and 3.4 times as long for GPT-5.6 Luna. For Claude Sonnet 4.6, each correct result that the agent adds over the chatbot costs about $0.54. At research-assistant wages of $15 to $50 an hour, this amount pays for 0.6 to 2.2 minutes of an assistant’s time.
For a researcher choosing how to use these tools, the degree of agency matters more, on average, than the prompt or the software, whose effects fall sharply once the model can run its code. Delegating the running and revising of the code to the model removes the errors that are cheapest to detect. It removes only about two-fifths of those that the output does not flag, which the researcher must find by checking whether the model implemented the requested specification and whether its numbers are right. Rerunning the code would not reveal such errors, since, as Brodeur et al. (2026) note, code can reproduce the published numbers and still be wrong.
If researchers move from chat windows to coding agents, they encounter fewer errors, and a larger share of those errors is silent. Vansteenkiste (2026) argues that when accurate results grow harder to tell apart from plausible but inaccurate ones, journals should require more transparency about code and data. Weder di Mauro (2026) expects that as AI generates the technical analysis behind policy advice, what will stay scarce is a human relationship built on trust and accountability. Our results suggest a similar division of labour in empirical research, since agents run and revise the code at little cost but leave the researcher to judge whether its numbers are right.
Source : VOXeu
Financial crises routinely trigger sweeping overhauls of financial regulation, but whose interests do these reforms…
Bilateral pushback against China is failing, making joint WTO and IMF action key to tackling…
Public debt is at or near record highs across the OECD, increasing the pressure on…
Mortality is one of the most salient consequences of climate change. Existing estimates have largely…
In a green transition, workers will have to move from polluting, carbon-intensive sectors to other…
In response to digital dependence, Europe has decided to think bigger. This includes AI gigafactories,…