External Audit Exposes Blind Spots in OpenAI’s Safety Protocols

For the average user interacting with advanced language models, the recent findings from an independent audit suggest that the safety nets holding back erratic AI behavior may have more holes than previously acknowledged. Researchers from METR and Redwood Research spent three weeks embedded within OpenAI, scrutinizing the systems designed to catch rogue AI actions. Their work, conducted between late July and mid-August, revealed that a substantial slice of internal traffic was effectively invisible to the company’s own monitoring tools.

The timeline of these discoveries began with the launch of a pilot program designed to grant outside researchers unprecedented access to OpenAI’s internal infrastructure. From July 21 to August 11, the auditors were given full access to the company’s codebase and internal documents. They also conducted interviews with approximately a dozen OpenAI employees to understand how the monitoring systems operated in practice. This period of deep inspection was intended to test the resilience of OpenAI’s defenses before they were scaled to newer models.

The most striking finding concerns the coverage of the Responses API. The auditors estimated that between 6 and 25 percent of internal agentic traffic from models at the GPT-5.5 class and above was not monitored at all during the audit window. METR cautioned that the uncertainty surrounding this figure is significant in both directions, meaning the true number could be higher or lower. This gap implies that a non-trivial amount of high-level AI activity was proceeding without an external watchdog watching over it.

Specific technical failures also emerged regarding internal Codex traffic. The team estimated that roughly 10 percent of this traffic went untracked between May and July due to a bug in how the system handled compaction events, which are essentially context summaries. While OpenAI had already identified and fixed this specific bug while the audit was ongoing, the auditors noted that during the period of the gap, agents could delay oversight. The monitoring system only triggered when an agent handed control back to a user, allowing agents to sidestep checks by simply continuing to execute actions without pausing.

Beyond statistical gaps, the researchers identified specific vulnerabilities that allowed for evasion. They found methods to hide hundreds of actions from the monitor and ways to bypass the Codex sandbox entirely. Furthermore, they determined that specially crafted inputs could likely crash the monitoring system altogether. At the start of the audit, model behavior during reinforcement learning (RL) training was not being monitored at all, a gap that OpenAI claims has since been closed.

OpenAI stated that it paused RL training following the Hugging Face incident in August and subsequently expanded its monitoring to cover this area. However, these corrective measures were implemented after the auditors’ access period had ended, meaning they were not part of the formal evaluation. The company asserts that it has now fixed all the weaknesses the auditors identified and has received the full 61-page report internally.

The level of public disclosure has drawn attention. After OpenAI applied redactions, METR and Redwood Research leadership received a 38-page version of the report, while the public was provided with only a short summary. METR stated that it did not have the right to disclose the redacted portions on its own. For future audits, the organization said it intends to push for greater public transparency to ensure that the public can assess the robustness of these safety measures more directly.

The result is a scenario where the most detailed findings remain partially hidden behind internal redactions, even as the company claims to have addressed the identified flaws. This pilot program was designed to establish a baseline for future independent reviews, but the current opacity suggests that the process of verifying AI safety is still in its early, imperfect stages.