AI agents can now write substantial amounts of research code. In a recent self-selected survey, 39 percent of economists reported using coding agents more than once a week. As tools such as Claude Code and GitHub Copilot become more widely available, producing code is getting easier. But who is reviewing what the agents produce? 

In our earlier tests, agents demonstrated their ability to solve coding tasks, scoring between 77 and 81 percent on the assessment we use to hire research assistants. Yet their code was complex, and the rationale for their analytical decisions was unclear. Another example is a recent benchmark of econometric tasks, letting AI agents run and revise their own code raised the share of correct results from 79 to 97 percent, but every remaining failure was silent: the code ran and returned wrong numbers. But code that runs is not necessarily code a team can work with. Research projects often last several years, and the code needs to be understandable to whoever picks it up next. It also needs to meet team standards, such as documenting analytical decisions and validating merges.

As more journals require researchers to share data and code, research code is receiving more scrutiny. And that scrutiny has exposed two different problems: code may fail to reproduce the published results, and even when it does reproduce them, it may still be wrong. Only one in five packages in our checks reproduce as submitted. At the American Economic Association, just 31 of 404 manuscripts at conditional acceptance were recommended for acceptance without changes to their replication packages.

Reproducibility, however, does not imply the code is correct. An audit of 110 articles in leading economics and political-science journals found substantive coding errors in 26, even though 85 percent were computationally reproducible. Errors included duplicate observations, incorrectly coded treatment variables, and models that differed from those described in the papers.

AI thus poses a challenge, and offers an opportunity. By greatly expanding research code production, it makes systematic review more important. Yet these tools can also put careful code review within reach of researchers who lack expert coding skills or a research assistant or colleague to check their work.

From manual peer review to review on demand

Before AI coding tools became available, our DECDI team organized structured peer code reviews. Colleagues reviewed each other’s code using standard checklists. Participants reported learning something new even in repeat rounds; one described the checklists as “very helpful starting points for the review process.” About 15 people took part each quarter, but finding time and coordinating reviews limited engagement. Since then, we have combined these checklists with lessons from reproducibility reviews into a reusable AI skill. It applies the same checks a peer reviewer would, but researchers can run it whenever their code is ready, without waiting for a review round.

Why use a skill rather than a prompt?

When you ask an agent to “review my code”, it can produce different results depending on the model, the type of errors it checks for, and the depth of its review. A skill, on the other hand, makes the review more consistent because the agent follows the same steps each time. It runs the same checks, uses the same reference material, and produces the report in the same format every time.

How does AI code-review skill work? The skill first checks safeguards and reviews code and documentation without executing code or changing files. It reports findings by severity, and the researcher decides whether any fixes should be implemented.

Errors can go unnoticed. Many consequential research coding errors trigger no error message When non-unique identifiers cause a merge to fail, merge m:m may let the code run, but it can match records incorrectly and produce unstable results.  Likewise, duplicates drop, force removes duplicate IDs by retaining the first record encountered. A comment may say the analyses clusters standard errors at the village level, while the code uses heteroskedasticity-robust errors instead. None of these errors stop the code from running, and all of them look fine on a quick read. The skill flags all three as high severity and points to the exact lines, explaining why each issue matters. We tested the skill on a small synthetic package containing the three errors described above. It flagged all three as high-severity and caught a hard-coded file path. The table shows part of its report.

Example output. Excerpt from a Phase 1 report on a synthetic test package. “Can the agent fix it?” shows whether the agent can make the change once approved (Yes), needs a fact or decision from the researcher first (Conditional), or must leave the decision to the researcher (No). The full report also lists a recommendation and proposed next action for each finding.

Findings are rated Critical, High, Medium or Low. The rating reflects how much the problem could affect the research, such as a privacy exposure or wrong results are rated “high”, and readability as “low”. It does not reflect how hard the fix is. The skill covers more than 60 checks drawn from your peer review checklists.

A read-only review followed by approved changes

The review is done in 2 phases. Phase 1 starts after the researcher confirms that the folder has no confidential data. The agent then reads the code and documentation only. It doesn’t open data, run scripts or edit files. The completed report provides the location and severity of each finding, along with a recommended next step. The report also indicates which issues the agent can address. Issues marked “Yes” involve mechanical errors that the agent can easily fix after approval. “Conditional” issues require the researcher to supply additional information first, and “No” issues require an analytical judgment, such as defining the sample or choosing a model.

Phase 2 starts only after the researcher chooses which findings to fix. Changes are tracked in Git, so they can be reviewed and reversed.

The limits of the review

The skill does not run code or inspect data, so it cannot confirm reproducibility or data cleanliness. It also treats methodological concerns as questions for the researcher, rather than conclusions. The idea is for the skill is to act like a reviewer that can highlight potential problems systematically. The researcher’s responsibility is to then address the problem. The skill supports Stata, R, and Python and works with common coding agents.

Source : World Bank