METR and Redwood Research Find Gaps in OpenAI's Internal Model Monitoring
An on-site audit revealed that 6 to 25 percent of top-tier model traffic escaped internal checks, while other issues remain unresolved.
By Jason Kovalenko ·
Two independent researchers completed an on-site review of OpenAI's internal monitoring systems in late July and early August.
The audit, conducted by METR and Redwood Research, examined how the company tracks the behavior of its most advanced AI models. The team spent three weeks, from July 21 to August 11, inside the company. They reviewed internal code and documents and interviewed approximately twelve employees. This pilot program was designed to test the limits of external oversight in AI infrastructure.
The primary finding concerns the volume of traffic that escapes monitoring. The auditors estimate that between 6 and 25 percent of internal traffic from the GPT-5.5 class and above was not monitored at all during the review period. METR noted that the uncertainty around this range is substantial. A separate issue affected internal Codex traffic. The researchers estimate that roughly 10 percent of this traffic went untracked between May and July. The system failed to process compaction events, which are context summaries. OpenAI identified and fixed this specific bug while the audit was still ongoing.
The audit identified structural weaknesses in how the monitors trigger. Oversight only activated when an agent returned control to the user. This meant an agent could continue executing actions indefinitely without being checked. The auditors also found methods to hide hundreds of actions from the monitor and bypass the Codex sandbox. They concluded that specially crafted inputs could likely crash the monitoring system entirely. Additionally, model behavior during reinforcement learning training was not monitored at the start of the audit.
OpenAI stated that it paused reinforcement learning training following the Hugging Face incident in August. The company expanded its monitoring to include this phase after the auditors' access period ended. OpenAI claims it has now fixed all the weaknesses identified in the report.
The full audit report consists of 61 pages and was shared internally. After redactions, METR and Redwood Research leadership received a 38-page version. Only a short summary was made available to the public. METR stated it did not have the right to disclose the redacted sections independently.
METR said it will push for greater public transparency in future audits of AI systems.
