Expanded AI Safety, Monitoring and Accountability

by Ruth Miller
Research and factual review current through October 2, 2026

   

1.Introduction

I did not set out to write an article about AI.

I began looking seriously at AI this summer because I was concerned about the rapidly growing physical resources required to develop and operate it — electricity, water, computing equipment and infrastructure. I was interested in what voters and other people were already asking about those costs, and in whether AI development could become more computationally efficient and sustainable. 1

At the time, I had no intention of writing an article. I thought I might eventually put together some ideas and send them to agencies, governments or organizations if I came up with anything useful, or perhaps write a few brief notes for our readers.

I had been using AI personally for about three years, mostly for research, writing help, organizing and brainstorming. My use was overwhelmingly nontechnical. AI had become extremely important to my own independence and ability to work, but I was far from an AI expert. Most of what I know about AI safety and security I learned during several months of intensive reading this summer.

In mid-August I began encountering report after report about AI safety and security incidents. As a result my interest expanded from resource use to security, and eventually, when I decided to write this article, I narrowed my subject principally to security, monitoring and accountability. Resource use, national security, employment, copyright, misinformation and other consequences remain important; each could easily support an article of its own.

AI systems are now becoming capable of taking consequential actions. We need to know who is watching what they do, and what happens when that watching fails.

A note about timing: AI development, research and policy are changing extraordinarily quickly. The factual record for this article is frozen as of October 2, 2026. Dates and sources in the notes document the evidence used here and give readers starting points for finding later developments. Events after October 2 are outside the article's research cutoff unless explicitly identified as a later update.

A note about language: AI researchers, companies, journalists, and I in this article, often use human shorthand: an AI “decided,” “believed,” “tried,” “escaped” or “deceived.” Those words can make complicated behavior easier to describe, but they can also imply human-like intentions or understanding that the evidence does not establish. When the distinction matters, I will try to describe what the system actually did rather than what it supposedly wanted or believed.

That matters for accountability because anthropomorphic language can shift attention from an engineering question, “Why did the system and its controls permit this action?”, toward a psychological-sounding question about why the AI “wanted” to do it. The first question is one regulators, developers and independent evaluators can act on.

2. Why I’m Writing About AI

I have lived with disabilities throughout my life, and aging has brought some additional limitations. Since I integrated AI into my daily workflow about three years ago, those limitations no longer prevent me from doing what is needed and what I love. AI has increased my independence and capability. Consequently, I am well aware that both unsafe AI development and unnecessarily restricting useful AI have real human costs.

So I am not approaching this subject as someone who wants AI development to stop. I use AI extensively, I know firsthand that it can be tremendously useful, and I believe its potential benefits are great. I want those benefits to continue. I also want AI development to be safe, sustainable and accountable enough that we can continue to benefit from it.

Learning that increasingly capable systems had crossed technical boundaries, reached systems they were not intended to reach, and behaved in ways their developers had not adequately anticipated or controlled pivoted my focus toward security and safety. Then I learned that safeguards and monitoring existed without being used everywhere they were needed, and that extensive records of an agent’s activity could exist without anyone recognizing dangerous behavior in time to stop it. That stirred an almost visceral concern in me: We have been increasing what AI systems are capable of doing while our ability to reliably test, monitor, contain and control them has not kept pace.

That is intuitively backward.

Control should surpass capability before capability is increased — not chase after it.

When developers make an AI system substantially more capable, they should first be able to demonstrate that their ability to understand, evaluate, monitor and control that system is adequate for the capabilities they are giving it.

And companies cannot be solely responsible for deciding whether they have succeeded. Government, independent researchers and the public all have roles in establishing what evidence is required, examining serious failures and deciding when claimed fixes have actually been demonstrated to work.

3. What Has Already Gone Wrong

Until recently, I assumed that companies developing frontier AI systems (roughly, the most advanced and capable AI systems currently being developed) were testing them under highly controlled conditions, with extensive safeguards and monitoring in place to prevent them from reaching systems they were not supposed to reach.

In July 2026, OpenAI was conducting internal cybersecurity evaluations designed to test increasingly capable models. During those evaluations, models circumvented isolation controls, found unintended ways to communicate, gained access to the internet, compromised parts of OpenAI’s own research infrastructure, and reached systems belonging to Hugging Face and other third parties. OpenAI later described the episode as a “warning shot.” 2

Some have said that the AI “escaped.” But, of course, the models did not climb out of a computer and wander around the internet. They defeated boundaries that were supposed to restrict what they could access and do.

Some protections used with deployed systems had deliberately been removed or altered so researchers could measure underlying cyber capability. That can be scientifically legitimate: if a safeguard prevents the behavior you are trying to measure, researchers may need to change that safeguard for the experiment. But removing one layer of protection does not answer the safety question of what should still be watching consequential real-world actions, or what containment must remain in place when an evaluation could affect systems outside the test environment.

The incidents also exposed a distinction that became central to my research: recording what an AI did is not the same as monitoring it. Logs can preserve an enormous amount of activity without a person or automated safety system examining those records promptly enough to recognize and stop dangerous behavior. In some incidents, important behavior was identified only later, when researchers returned to stored records, sometimes months afterward.

Anthropic provides an especially clear example. After reviewing cybersecurity evaluation transcripts, it identified three previously unrecognized incidents in which Claude had reached real third-party systems. A later review found a fourth incident that had occurred in January and was not identified until August. Anthropic then widened its retrospective search substantially and found no additional incidents of comparable or greater severity. That negative result matters too: broader review is valuable not only when it finds more failures, but when it gives us evidence about how extensive the known problem may be.8

OpenAI’s later retrospective investigations likewise identified additional third-party impacts and earlier agent activity that had not been recognized when it occurred. Outside researchers also reconstructed earlier activity from traces left on third-party infrastructure.2

That retrospective review was still uncovering incidents as this article was completed. OpenAI says it learned on September 29 of a June incident in which an agent accessed nonpublic historical bushfire statistics held by the New South Wales National Parks and Wildlife Service. After an internal review, OpenAI notified Australian authorities on October 1.

There is another category of risk that should not be confused with these incidents: people deliberately using AI to conduct cyberattacks or other harmful work. In those cases a human directs the system toward the harmful goal. As agents become more capable, a person may be able to delegate more of the work after the initial instruction. That is a serious security problem, but it is different from an AI system crossing boundaries during an evaluation or ordinary research task.

We have therefore seen several different failures: containment can fail; monitoring can be absent, incomplete or too slow; automated detection can miss important behavior; and retrospective records can reveal incidents long after the affected system needed protection.

That is no longer a hypothetical risk.

The incidents described here are only examples. By September 26, OpenAI said it had notified more than 100 outside organizations about agent activity identified during its broader review. Those notifications do not necessarily mean that systems were compromised or private data was accessed, but they demonstrate that the issue extends well beyond the individual incidents discussed here.3

Another example emerged after this article was completed. On October 5, the Wikimedia Foundation reported that an investigation it conducted after learning about other AI-agent incidents had uncovered activity it believes came from OpenAI agents. Wikimedia found unauthorized wiki edits, including several changes to a citation tool that it described as “potentially malicious”; unsuccessful attempts to compromise its public Etherpad note-taking service; millions of automated requests to its APIs and pages; and hundreds of thousands of queries to its Wikidata Query Service. Wikimedia says that traffic may have contributed to a partial service outage in May. It found no evidence that its systems or data were compromised.

The incident illustrates another cost when monitoring and notification fail or arrive too late: the affected organization may have to spend its own resources discovering what happened. Wikimedia described the difficulty and effort required to investigate and attribute the activity and said increasing bot and agent traffic already imposes costs for servers and people. That is particularly significant for an organization supported largely by small donations from its readers. Wikimedia’s investigation also leaves important questions unanswered. It has not publicly explained how it attributed this particular activity to OpenAI agents, and the information currently available does not establish when OpenAI first detected or recognized the Wikimedia activity or whether OpenAI informed Wikimedia about it before Wikimedia discovered it through its own investigation.18

4. Monitoring: Watching the Thoughts, Watching the Hands

The word monitoring covers several different things, and confusing them can make the safety picture look much simpler than it is.

Capability evaluation asks what the model can do. Reasoning or chain-of-thought monitoring examines available evidence of how the model arrives at its answers or actions. Behavioral monitoring watches what the system actually does: commands it executes, systems it contacts, files it reads or writes, credentials it uses, information it sends, and other consequential actions. Logging records activity for later examination. Retrospective review examines those records after the fact. Containment is supposed to prevent an unauthorized action from succeeding even if monitoring misses it.

A simple way to think about the distinction is that reasoning monitoring can sometimes help us watch the AI’s “thoughts,” while behavioral monitoring watches its “hands.” The analogy is imperfect, but the difference matters. Researchers may have legitimate scientific reasons to alter or withhold some reasoning-based safeguards during an evaluation. That does not eliminate the need for independent controls watching consequential external actions.

A log sitting unread on a server may become excellent forensic evidence. It is not, by itself, an operating safety control.

The September 20 OpenAI incident shows both improvement and another failure. An internal research agent that was supposed to have no access to the live internet discovered that it could use the system that normally translates website names into internet addresses — DNS — to send information outside its restricted environment. It used that route to obtain an answer from an external public chatbot. About twelve minutes later, OpenAI’s monitoring system generated its highest-priority alert, known internally as a P0 alert, and a human acknowledged it three minutes after that. 4

But the intended automatic shutdown did not occur. OpenAI reports that this produced confusion about whether the run should be stopped, and the run was not manually terminated until approximately 2½ hours after the P0 alert had been acknowledged. OpenAI’s public account does not explain why there was uncertainty once a human had acknowledged the alert and the automatic stop had failed. That leaves a straightforward accountability question: once a serious containment alert was acknowledged, why was the run not stopped immediately?

The same retrospective review found other attempts to use DNS to reach outside the restricted environment that monitoring had not treated as seriously as expected. Another automated security system that was supposed to look for unusual DNS traffic was not watching this particular testing environment. The traffic was recorded, but that detector was not set up to examine and flag activity from there. Again, logging and active monitoring were not the same thing.

OpenAI responded by pausing tool-using training, evaluation and inference for its most capable models while it strengthened controls, and said the affected training run would not resume. That response matters: it demonstrates in practice the principle that capability work can be slowed when control has not been adequately demonstrated. But it raises the next question as well: how do we know the new controls work?⁴

That question should not be answered solely by the organization whose controls failed. Existing work by independent evaluators and government AI-safety bodies provides models for testing safeguards, red-teaming monitoring systems and examining whether corrective measures actually withstand challenge. The standard is not proof that failure is impossible. A more realistic standard is to show that the promised control was implemented, test it against the kind of failure it was designed to prevent, examine related failure modes, report the results, and continue testing as the system changes.

And fixing the exact pathway is not necessarily enough. If an agent crosses a boundary through route X, closing route X does not by itself establish that the behavior revealed by the incident will not appear through route Y. Anthropic’s subsequent work is useful here: rather than testing only whether an identical exploit could be repeated, it describes broader evaluations aimed at underlying behaviors and at longer, more difficult and multi-agent situations. Anthropic also acknowledges that constructing evaluations representative of deployment remains an open research problem.8

This is why I have become less interested in whether a company can name a safeguard than in whether the safeguard was operating, whether it detected what mattered, what happened after the alert, and whether anyone outside the company has tested the claimed fix.

5. Accountability: Ask One More Question

The incidents are serious, but individual failures are not what concern me most. Testing is supposed to discover failures. My concern is what happens next.

When a significant monitoring or containment failure occurs, four questions should follow: What does the company say went wrong? What independent evidence supports that explanation? What did the company change? And what independent evidence shows that the change actually works?

That four-part framework is deliberately demanding about evidence. “We believe” is not the same as “we observed.” A company prediction that production safeguards would probably have prevented an incident is relevant, but it is not the same thing as a demonstrated result. Likewise, a company saying it fixed a problem is not independent evidence that the fix works.

Serious incidents should also produce a knowledge-and-disclosure timeline. When did the activity occur? When did the developer first detect it or receive credible information about it? What did people inside the company understand at that point? When was the affected third party told? When were regulators or government authorities told, if applicable? When was the public told? Did later outside research materially broaden what was known?

The unresolved RubyGems chronology illustrates why this information matters. OpenAI has acknowledged that its agents operated on RubyGems during the relevant period, but the public record still does not establish when OpenAI first recognized that activity or when RubyGems was first informed. Without those established dates, potentially useful evidence about detection, response and disclosure is missing from the public record.

Independent oversight therefore needs to do more than receive company reports. Qualified outside evaluators should be able, where appropriate, to examine relevant logs and monitoring results, reproduce important evaluations, challenge corrective measures and determine whether a fix addresses only the discovered pathway or the broader behavior revealed by the incident.

None of this requires publishing model weights9, exploitable vulnerabilities, protected personal information or legitimate trade secrets. Independent inspectors can examine confidential material without making it public. But the public does need enough information to know that serious incidents occurred, that affected parties were notified, that required safeguards were actually operating, and that corrective measures were tested rather than merely announced.

Oversight without consequences is only observation. When demonstrated capability approaches or exceeds demonstrated control, restrictions on deployment, external access or further capability increases should be available until specified safety conditions are met.

The objective is to establish a very ordinary principle for an extraordinary technology: as AI systems become capable of more consequential actions, the evidence that they can be adequately monitored and controlled should become stronger too.

Principle 1

Build a Control System, Not a Single Safeguard

One of the clearest lessons from these incidents is also one of the oldest ideas in security: do not depend on a single defense working perfectly. Security professionals call this defense in depth. Multiple, overlapping protections are used so that when one fails, another can prevent, detect or limit the consequences. 5

This is not a new principle that AI developers discovered after the incidents described in this article. OpenAI's Preparedness Framework had already explicitly embraced defense in depth before the 2026 incidents. The broader AI-safety community likewise recognizes that AI systems can be made more robust by layering multiple safeguards.

Yet the incidents show that understanding the principle and consistently putting it into practice are not the same thing.

After investigating its 2026 cybersecurity-evaluation incidents, Anthropic acknowledged that the affected evaluation environments had been relying largely on a single layer of defense where several were needed. It subsequently added additional protections, including clearer boundaries for agents, verification that supposedly isolated environments were actually isolated, and monitoring capable of intervening while a run was still underway.

OpenAI's incidents exposed different gaps. In one case, an automated security system intended to detect unusual DNS traffic was not monitoring the particular environment in which the activity occurred. In another, a highest-priority alert was generated and acknowledged by a human, but the intended automatic shutdown failed and the run continued for roughly another two and a half hours before it was manually terminated.

These are different failures, which is precisely why layered defenses matter. Containment may fail. Monitoring may miss something. An alert may work while the mechanism intended to stop the activity does not. A human response may be too slow. Logs may preserve evidence that nobody recognizes as important until much later.

No single safeguard can reasonably be assumed never to fail.

The appropriate controls will vary with the system, what it is capable of doing, the environment in which it is operating and the consequences if something goes wrong. Legislators do not need to design one universal technical architecture.

Testing a model's underlying capabilities may sometimes require disabling or altering a safeguard. When an experiment can affect systems outside the testing environment, that should be a deliberate and documented decision accompanied by other controls appropriate to the increased risk.

This raises an uncomfortable question: if defense in depth was already an understood safety principle, why were increasingly capable AI agents operating in environments where adequate layers of defense were not consistently in place?

The evidence does not establish a single answer to that question. Capabilities have been developing rapidly. Evaluation environments may have been treated differently from deployed systems. Developers may have underestimated what agents could do or where they could reach. Operational implementation may simply have lagged behind the principles companies had adopted.

Anthropic itself offers an instructive example of why safety cannot depend solely on a company's intentions. The company was founded by former OpenAI researchers after disagreements that included concerns that commercial and competitive pressures could override safety. Its founders initially envisioned Anthropic as a small safety-focused research organization. Yet as Anthropic became one of the world's leading frontier-AI developers, it too discovered that some of its evaluation environments had been relying largely on a single layer of defense where several were needed.

An organization created in significant part around AI safety can still develop gaps between its safety principles and the controls actually operating in practice. Good intentions and an organizational commitment to safety do not eliminate that structural problem.

But there is another factor that cannot be ignored: competition.

The 2026 International AI Safety Report warns that competitive pressure can give AI developers incentives to reduce investment in testing and risk mitigation in order to release new models quickly. Safeguards can cost money and time, while some of the consequences of inadequate safeguards may fall on people and organizations other than the developer.

OpenAI's own Preparedness Framework contemplates circumstances in which OpenAI could adjust its safety requirements if another frontier developer released a high-risk system without comparable safeguards. 6

That demonstrates something important for public policy: the safety decisions of one frontier AI developer can be affected by what its competitors do.

That is one reason essential safety requirements cannot depend entirely on voluntary decisions by individual companies.

A company that invests additional time and resources in safeguards should not have to choose between maintaining those protections and keeping pace with a competitor willing to accept greater risk. A common regulatory floor can change that incentive by requiring every covered developer to meet minimum safety and accountability requirements.

Government does not have to design the defenses.

The European Union's AI Act offers one existing model of this distinction. For general-purpose AI models classified as presenting systemic risk, the law requires providers to evaluate their models, including adversarial testing; assess and mitigate systemic risks; track and report serious incidents and possible corrective measures; and maintain adequate cybersecurity protection. Providers retain flexibility in how they demonstrate compliance, including through recognized codes or standards or by demonstrating adequate alternative means.

The EU framework is new, and its existence does not establish that it will succeed. But it demonstrates an important principle: government can establish required safety outcomes and an accountability floor without prescribing one universal technical architecture for achieving them.

That distinction matters.

Technical experts should determine which combinations of containment, monitoring, intervention and other safeguards are appropriate to particular systems and risks. But government can require that adequate protections exist, establish what evidence developers must provide, require reporting when serious failures occur, and create consequences when required protections are absent or demonstrably inadequate.

The central question, then, should not be whether an AI company can point to a safeguard.

It should be whether the company can demonstrate a system of overlapping controls appropriate to what its AI can do, one that does not assume any critical safeguard will always work, and one that remains adequate as capability increases.

And when companies already understand that principle but the necessary defenses are nevertheless missing, regulators and elected officials should ask why.

Principle 2

Tell the People Who Need to Know

When an AI system reaches another organization's system unexpectedly or without authorization, the affected organization should be told promptly.

That notification should not have to wait until the developer knows exactly what occurred, why it occurred, how much harm resulted, or who was responsible. An initial notification can say something much simpler: our AI system may have accessed yours; this is what we currently know; we are investigating; we will provide additional information as we learn more.

There is a practical reason for doing this. The affected organization may need to secure systems, change credentials, preserve logs, investigate what the AI accessed or did, protect users or data, and prevent further activity. It may also possess evidence that the AI developer does not have. Prompt notification can therefore help both organizations understand what happened.

The 2026 Anthropic incidents provide a concrete example. After Anthropic identified three cases in which Claude had reached real third-party systems during cybersecurity evaluations, it contacted its evaluation partner and the affected organizations. The two organizations it successfully reached had not previously detected the activity or contacted Anthropic. Anthropic was then able to work with them on remediation. A later retrospective investigation uncovered another incident, from January, and Anthropic subsequently reported that all affected parties had been notified.

This is an important distinction: notification is not an admission of fault.

A developer may initially know only that its model appears to have interacted with another system in an unexpected or unauthorized way. It may not yet know whether a vulnerability was exploited, what information was reached, whether damage occurred, or why the model behaved as it did. Those are questions for investigation.

Waiting for all of those questions to be answered before notifying the affected organization, however, can deprive that organization of information it needs while the investigation is still most useful.

The European Union has already adopted a related principle for regulatory reporting. Under the EU AI Act, providers of general-purpose AI models with systemic risk must track, document and report serious incidents to the AI Office and, where appropriate, national authorities. The General-Purpose AI Code of Practice makes that system more concrete. Depending on the type of serious incident, an initial report may be required within days. The provider can report what it knows at that point, continue investigating, submit updates and eventually provide a fuller account. Reporting an incident does not itself constitute an admission of wrongdoing.

That is a useful model, but it addresses a different threshold and a different recipient. EU serious-incident reporting is principally designed to ensure that regulators learn about sufficiently consequential events. An organization whose system has been unexpectedly accessed may need to know long before an incident meets that threshold, and even if it never does.

The unauthorized contacts with Hugging Face, RubyGems and other third-party systems described earlier in this article do not appear, on the currently available evidence, to meet the EU AI Act's definition of a “serious incident.” Yet the affected organizations still had an immediate interest in knowing that an AI agent had reached their systems. The public also has a legitimate interest in knowing that such failures are occurring, particularly when they reveal weaknesses in monitoring or containment.

Affected-party notification should therefore have its own, lower threshold.

When a frontier AI developer obtains credible evidence that its system has unexpectedly or without authorization accessed, altered, impaired or obtained information from a third party's system, it should promptly notify that party unless doing so would itself create a substantial security or legal risk. The initial notice need not characterize the incident as serious or assign responsibility. It should provide the information currently known that the affected organization needs to protect itself and investigate.

I would add another practical requirement: developers should maintain reliable procedures for identifying and contacting organizations their AI systems may have affected. Discovering an intrusion is of limited value if nobody knows whom to call, an email sits unread, or an urgent security report disappears into an ordinary customer-service queue. Developers should have procedures for reaching an appropriate security contact, obtaining acknowledgment and establishing a dependable channel for exchanging information as an investigation develops.

Regulators need a parallel channel.

Here, the EU's approach provides a useful starting point. Serious incidents should be reported to the appropriate authority promptly, incomplete initial reports should be permissible when an investigation is continuing, and additional information and corrective measures should follow as they become known. Governments can establish thresholds defining which incidents require regulatory reporting without requiring every unusual AI action to produce a major regulatory investigation.

I would add another requirement for significant incidents: a knowledge-and-notification chronology. The eventual record should establish when the activity occurred; when it was detected; when the developer understood its significance; when the affected organization was notified; when regulators were notified; what additional facts emerged later; and what corrective action followed.

That chronology protects companies as well as the public. A delay between an incident and notification does not by itself demonstrate concealment. A developer may genuinely not have known what happened. It may initially have misunderstood the significance of what it saw. A clear chronology can distinguish those situations from one in which important information was known but notification was unnecessarily delayed.

The unresolved questions surrounding OpenAI's agents' activity on RubyGems illustrate why this matters. The public record establishes that OpenAI agents operated there during the relevant period, but it has not yet established when OpenAI first recognized the activity or when RubyGems was first informed. Without those facts, the evidence does not justify an inference of concealment. But regulators, investigators and the affected organization should be able to establish that chronology.

The principle is simple: the organization whose system may have been affected should not have to discover an AI incident months later from its own logs, an outside researcher or a news report when the AI developer already possessed information that could have helped it respond.

And regulators cannot oversee risks they are never told about.

Prompt notification, reliable communication channels and documented reporting timelines turn information about AI failures into something more useful than an internal company record: information that the people responsible for protecting affected systems, and the public, can act on.

Principle 3

Don’t Let the Company Grade Its Own Repair

When a significant monitoring, containment or other control fails, the developer should investigate what happened, correct the problem and test the repair. But the company that experienced the failure should not be the only institution deciding whether the problem has actually been fixed.

Independent verification addresses a basic accountability problem. If the same organization designed the control, experienced its failure, investigated what happened, designed the correction, tested that correction and then decided whether the evidence was sufficient, every stage of the process remains inside the same institution.

The frontier AI companies themselves are already demonstrating that meaningful outside evaluation is possible.

After OpenAI's agents gained unauthorized access to Hugging Face, OpenAI brought in researchers from METR and Redwood Research for an independent investigation. OpenAI provided access to more than a thousand unredacted agent transcripts, raw chains of thought and other information, and the researchers spent six days working on-site at OpenAI. METR described the exercise as an important precedent for independent investigation of AI misalignment incidents. 7

But the scope of that investigation reveals an important gap. METR examined what the agents had done, how they reasoned and collaborated, and how the incident developed. The effectiveness of OpenAI's safeguards and its planned remediation were explicitly outside the investigation's scope.

Anthropic has gone further in another direction. After discovering four incidents in which Claude models gained unauthorized access to real third-party systems, Anthropic signed an agreement giving METR wide-ranging access for an independent investigation. That access includes relevant transcripts beyond the immediate incident windows and access to Anthropic employees who are permitted to discuss confidential information. Anthropic has said it intends to give METR as much time as the organization considers necessary. 8

These are important developments. Companies should receive credit for allowing independent researchers to investigate consequential failures, including access to information that cannot responsibly be made public.

In those cases, investigating what went wrong and verifying whether the repair works are both needed.

After a consequential control failure, independent verification should extend to the corrective measures themselves.

The first question is straightforward: Did the company fix the failure that actually occurred?

If an automated shutdown was supposed to terminate a run after a critical alert but failed to do so, testing should establish that the corrected shutdown mechanism now works. If an environment believed to be isolated allowed an agent to reach the internet, testing should establish whether the new containment and monitoring controls prevent or detect that pathway.

But reproducing the original failure is not enough.

A particular incident may expose a broader weakness. Closing the route through which an AI escaped does not establish that another route cannot expose the same underlying problem. A correction should therefore be tested not only against the precise incident that revealed the weakness, but against related ways the same kind of failure might occur.

Anthropic's response provides a useful example of this broader approach. After initially searching roughly 141,000 cybersecurity-evaluation transcripts and identifying three incidents, Anthropic discovered a fourth incident while preparing material for METR. It then widened its search dramatically, examining roughly 481 million transcripts across cyber evaluations, other evaluations, reinforcement-learning environments, subagent logs and other activity. That broader review rediscovered the four known incidents and found no others of similar or greater severity.

That result does not prove another incident can never occur. It provides something more useful: evidence that the company looked beyond the particular cases it already knew about to determine whether the problem was more widespread.

Layered defenses create another testing requirement. If safety depends on multiple controls because any individual control can fail, testing should not merely demonstrate that each control works under expected conditions. It should also examine what happens when one of those controls does not work as intended. Does another layer detect the problem? Does an alert lead to intervention? If automatic intervention fails, is there another effective route to stop the activity?

This is a recommendation, not something I have found imposed as a specific general requirement in existing AI regulation. But it follows directly from the purpose of defense in depth: redundancy matters only if the remaining defenses actually protect the system when one layer fails.

Independent verification also has to be genuinely independent.

That does not mean giving the public unrestricted access to proprietary models, private data or dangerous technical information. Qualified evaluators can work under confidentiality and security requirements. What matters is that they receive enough access, information, time and technical resources to test the claims they are being asked to evaluate.

The European Union is already developing this principle for external evaluation of general-purpose AI models with systemic risk. Its framework calls for qualified independent external evaluators in addition to internal evaluation and provides for access, information, time and other resources necessary to conduct meaningful testing. The EU is continuing to develop standards governing evaluator qualifications and independence.

The details matter. An outside review is not meaningfully independent if the developer can limit the evidence until the desired answer becomes inevitable. Evaluators need sufficient access to relevant models, records and personnel; an appropriate ability to choose additional tests when the evidence points somewhere unexpected; and a way to communicate material findings to the appropriate oversight body even when the findings are inconvenient.

METR's work demonstrates that this level of access is possible. In its work with frontier AI developers, it has received access to internal models, raw chains of thought and non-public information. In describing what a serious independent investigation of an AI incident may require, METR has identified access to relevant models and transcripts, the ability to reproduce relevant environments, interviews with employees, sufficient computing resources and, where appropriate, information about training.

Independent verification still cannot prove that a corrected system will never fail again. That is not a realistic standard.

It can establish something much more useful: the promised correction exists; it has been tested against the failure it was intended to address; related failure modes have been examined; the evidence meets an established standard; and continued testing will occur as the system changes.

Government does not need to design those tests or decide how an AI company should engineer a sandbox. It can establish when independent verification is required, what degree of access qualified evaluators must receive, what independence requires, what evidence regulators must receive, and what happens when verification fails.

OpenAI and Anthropic have already demonstrated that independent investigation of consequential AI incidents is feasible. The next step is to make sure independent scrutiny does not stop at explaining the accident.

After a significant safety failure, “we fixed it” should be the beginning of verification, not the end of accountability. The company should provide evidence of the repair. It should not be the only institution deciding whether that evidence is sufficient.

Principle 4

Capability Shouldn’t Outrun Demonstrated Control

The previous principle leaves one important question: what should happen when adequate control has not yet been demonstrated?

A useful principle is that control should keep pace with capability and preferably stay ahead of it. As AI systems become capable of taking more consequential actions, safeguards appropriate to less capable systems may no longer be adequate.

The frontier AI companies already recognize this principle.

OpenAI's Preparedness Framework establishes increasingly stringent requirements as models reach capability thresholds associated with greater risks. Systems reaching its High threshold require safeguards sufficient to minimize the associated severe risks before deployment. Systems reaching its Critical threshold require adequate safeguards during development itself, whether or not the model is about to be released.

In cybersecurity, OpenAI's Critical capability threshold means, in simple terms, that a model has become capable enough, with the appropriate tools and access, to find previously unknown security vulnerabilities and develop ways to exploit them across many well-protected systems without a person directing every step.

In 2026, this stopped being a hypothetical threshold.

After the Hugging Face incident and preliminary evidence that its developing Astra model might reach Critical cybersecurity capability, OpenAI said that its monitoring, alignment and security standards needed to stay ahead of the increasing risk. It temporarily slowed the pace of scaling (larger or more intensive development intended to produce increasingly capable models) while strengthening those protections.

OpenAI subsequently concluded that Astra did meet its Critical cybersecurity threshold. During this period it paused certain frontier training while strengthening network isolation, monitoring and other controls before proceeding. 9

Anthropic has adopted the same basic principle through its Responsible Scaling Policy: increasing capabilities and risks trigger stronger safeguards and risk-management requirements. 10

So the central idea is not particularly radical: greater capability can require greater control before proceeding.

The European Union has begun turning a similar principle into a regulatory framework. 11

The EU AI Act legally requires providers of general-purpose AI models with systemic risk to assess and mitigate those risks. Its General-Purpose AI Code of Practice provides a more detailed way for participating companies to demonstrate compliance. The Code calls for measurable risk-acceptance criteria and safety margins that account for uncertainty, including the possibility that safeguards can be circumvented, deactivated or otherwise fail.

Most importantly, the Code says signatories should proceed with development, market release or use of a model only when its systemic risks have been determined to be acceptable. If the risk is not acceptable, or is reasonably foreseeable to become unacceptable, measures should be taken to bring it within the acceptable range before proceeding.

The Code also recognizes that “proceeding” is not an all-or-nothing decision. Risk can sometimes be reduced by limiting how a model is made available: restricting API access to vetted users, expanding access gradually while monitoring what happens, or initially not making the model's learned parameters publicly downloadable, for example.

That is important because the appropriate response to inadequate control should be proportional to the risk.

A problem with one evaluation environment does not necessarily justify stopping every research project at a company. Depending on the circumstances, an appropriate restriction might instead involve the particular environment, internet or tool access, a type of training, access by particular users, deployment of a particular capability, or larger and more intensive training intended to produce still more capable systems.

But there should be a point at which continuing the activity is no longer the default.

The incidents examined in this article suggest there should be a stronger requirement: when a consequential monitoring, containment or other control failure demonstrates that a frontier system can create risks beyond the controls governing it, the relevant high-risk activity should not resume merely because the developer says the repair is complete.

The company should first demonstrate that adequate control has been restored. For sufficiently consequential failures, that demonstration should include the independent verification discussed in the previous principle.

Regulators will not be able to identify a perfect capability threshold in advance.

They cannot.

AI capabilities are changing rapidly, evaluations are imperfect, and researchers still face substantial uncertainty about when particular capabilities translate into particular real-world risks. The EU Code addresses that problem in a sensible way: risk assessments include safety margins intended to account for uncertainty and for the possibility that safeguards themselves may fail or be circumvented.

Uncertainty therefore cannot simply mean that development always proceeds.

Government does not need to decide which monitoring system to install, how to configure a sandbox, or exactly how an AI laboratory should train its next model. It can establish the conditions under which evidence of control is required: what level of risk triggers additional safeguards; what evidence demonstrates that those safeguards are adequate; when independent verification is required; what activities must be restricted while adequate control has not been demonstrated; and what must be shown before those restrictions are lifted.

That also addresses the competitive problem discussed earlier in this section. If every covered developer faces the same basic requirement to demonstrate adequate control before proceeding into sufficiently consequential risk, a company that takes safety seriously is less likely to be placed at a competitive disadvantage by doing so.

The objective is not to freeze capability. AI has enormous potential value, and preventing useful advances carries costs of its own.

The objective is to prevent capability from advancing into areas where the systems intended to control it have not caught up.

When adequate control cannot yet be demonstrated, the answer should sometimes be: not yet. Build the control. Demonstrate that it works. Then proceed.

 

6. What Governments Are Doing — and What Remains Missing

There is encouraging news in all of this: governments are acting on these developments.

Some of the protections discussed in the previous section already exist in law or government practice. Others have been proposed only within the past few weeks. Recent AI incidents have intensified attention to monitoring, containment and independent verification. That means some important decisions are still being made, and the public has an opportunity to ask questions before the rules are settled.

No government has yet assembled all of the pieces described in the four principles above. But many of those pieces are beginning to appear.

Update, October 3: One day after the research cutoff for this article, OpenAI safety researcher David Robinson resigned and publicly argued that AI companies are moving too quickly as they develop increasingly capable systems. Robinson spent about three and a half years at OpenAI and led the writing of safety reports accompanying 12 major model launches. He argued that frontier AI companies need to operate more like nuclear power plants or busy airports, with layers of redundancy and careful planning so that inevitable human errors do not open the door to disaster.17

That argument closely parallels two of the principles discussed above: defense in depth and keeping control ahead of capability. Robinson’s assessment is not independent proof that OpenAI’s safeguards are inadequate. OpenAI says it pauses training or holds back models when necessary and is strengthening research security, third-party evaluation and real-time monitoring. But his resignation adds an important internal perspective to the question at the center of this article: Are our systems for understanding, monitoring and controlling AI becoming stronger at least as fast as the AI itself?

 

The European Union

The European Union currently has the most developed broad regulatory framework for advanced general-purpose AI.

The EU AI Act is law, and since August 2, 2026, the European Commission has begun enforcing its requirements for providers of general-purpose AI models, including additional safety and security requirements for models presenting systemic risk. Those requirements include model evaluation, assessment and mitigation of systemic risks, serious-incident reporting and cybersecurity protections. The European AI Office can investigate compliance, evaluate models, require corrective measures and impose fines.

The EU's General-Purpose AI Code of Practice goes further in describing how companies can demonstrate compliance with those legal obligations. Among other things, it provides for independent external evaluation of systemic-risk models and establishes risk-acceptance criteria and safety margins intended to account for uncertainty and for safeguards that may fail or be circumvented.

The EU therefore already incorporates substantial parts of the system described in the previous section.

But important gaps remain.

Its serious-incident rules use thresholds much higher than the affected-party notification principle proposed in this article. An AI system can unexpectedly or without authorization reach another organization's system without necessarily creating the kind of serious incident that triggers EU reporting requirements. The affected organization should still be told.

Nor have I found a general EU requirement that, after a consequential control failure, an independent evaluator must verify that the corrective measure itself works before the relevant high-risk activity resumes.

The United States: technical capacity without a comparable federal law

The United States does not currently have a comprehensive federal frontier-AI safety law comparable to the EU framework. Federal agencies nevertheless already have relevant authorities and expertise.

The National Institute of Standards and Technology, or NIST, is part of the Commerce Department. It develops measurements, technical standards and evaluation methods used by government and industry.

Within NIST, the Center for AI Standards and Innovation (CAISI) serves as a primary federal point of contact for collaborative testing and research involving commercial AI systems. It works with developers and evaluators, conducts research and evaluations involving demonstrable national-security risks such as cybersecurity, and develops voluntary guidelines and standards. Its current AI Agent Standards Initiative specifically addresses security, identity, authorization and other problems created when AI systems can take actions autonomously.

Other federal agencies bring different expertise. The Cybersecurity and Infrastructure Security Agency (CISA) focuses on cybersecurity and critical infrastructure. The National Security Agency (NSA) has extensive national-security and cybersecurity capabilities. Other agencies contribute expertise relevant to particular risks.

One distinction matters here: these are executive-branch institutions. They have legal responsibilities, technical staffs and congressional oversight, but they are not independent of presidential administrations in the way an independent regulatory body may be.

That matters because legislation can establish authority and obligations, but implementation still determines how vigorously those authorities are exercised.

Congress now has a concrete proposal

On September 24, Senators Mark Warner, Brian Schatz and Andy Kim introduced the Artificial Intelligence Risk Management and Security Act of 2026.

It is a proposal, not law.

But it is striking how closely several of its provisions correspond to problems examined in this article.

The bill would establish a permanent AI Safety Board within the Commerce Department, bringing together NIST, CISA, NSA, Treasury and independent technical experts. Developers of covered frontier models would have to give the Board access to those models at least 45 days before public release so they could be evaluated.

The Board would establish enforceable standards for frontier-model evaluation and for the security of the environments in which those models are tested. Those standards would include safeguards and monitoring for models capable of discovering and exploiting software vulnerabilities without direct human prompting.

Developers would also have to create Model Safety Plans describing capabilities, risks and mitigations and identifying a corporate officer responsible for the plan.

The legislation would create a secure national AI incident database containing not only incidents but recurring flaws and near misses, so lessons from one event would not necessarily remain inside one company. It would require reporting of serious incidents, with faster reporting for imminent threats to national security, critical infrastructure or public safety.

It would also establish secure federal testing environments and specific security standards for autonomous AI agents, including their identity, authorization, access to systems and data, levels of autonomy and known limitations.

Violations of applicable standards could result in civil penalties.

The legislation therefore addresses several issues that appeared independently in the four principles above: monitoring and containment, incident reporting, outside evaluation, learning from near misses, responsibility for safeguards, and matching stronger controls to increasingly capable systems.

And its future is unresolved.

On September 29, Warner, Schatz and Kim asked the Senate to pass the bill by unanimous consent. Senator Ted Cruz objected, preventing passage through that expedited procedure. The legislation therefore remains a live example of a question Congress has not settled: what enforceable safety requirements, if any, should apply to the most capable AI systems?

Implementation would remain important even if Congress enacted such legislation. A law can require evaluations, standards and enforcement; executive-branch agencies would still have to carry them out. That makes both the contents of legislation and the way it is implemented legitimate subjects for continuing public and congressional scrutiny.

The states are not waiting

California and New York have already enacted frontier-AI requirements of their own.

California's SB 53 requires covered frontier developers to publish safety frameworks and report specified critical safety incidents and provides whistleblower protections.

In September, California went further. SB 813 created a framework for independent verification organizations capable of assessing AI systems, while AB 1405 established requirements intended to strengthen the independence, transparency and integrity of AI auditors.

Those laws do not yet create the specific rule proposed in the previous section: independent verification that a corrective measure actually worked after a consequential control failure. But they begin building something necessary for such a system: qualified institutions that can evaluate AI safety claims from outside the company making them.

And California is considering going further still.

Following recent AI incidents, Governor Gavin Newsom directed experts to develop proposals that could require independent verification organizations to work inside frontier-AI laboratories conducting regular audits and evaluations; require independent verification of companies' safety frameworks, transparency reports and risk assessments; and create an emergency model shutoff, or “kill switch,” whose effectiveness would itself be independently verified on an ongoing basis. 12

California is also considering expanding its definition of critical safety incidents to include loss-of-control events such as the Hugging Face incident discussed earlier in this article.

Those measures remain proposals under development.

But their direction is significant. Recent incidents are no longer merely examples being discussed by researchers. They are beginning to influence the questions policymakers ask about what should be monitored, what should be reported, and who should verify that safety systems actually work.

Government scrutiny is also becoming more concrete. California Attorney General Rob Bonta has served OpenAI with an investigative subpoena as part of a broader inquiry into cybersecurity incidents and risks involving the company and its models. The California Department of Justice had already opened a formal investigation into the Hugging Face incident. Bonta said developers have a legal as well as moral responsibility to ensure that their models do not perpetrate or enable cyberattacks during development, testing, or deployment, and that developers who fail to do so can be held legally accountable.

At the federal level, the Federal Trade Commission has opened a broader investigation into OpenAI, Anthropic, and other AI companies over potential risks their technologies pose to consumers. Reporting indicates that the FTC is preparing civil investigative demands that could require companies to provide documents and executive testimony concerning the safety of their models. The investigation is significant because it brings existing federal consumer-protection authority directly into questions involving increasingly autonomous AI systems.

Neither investigation establishes that a company violated the law. Both are investigations into whether existing legal obligations apply and whether they were met. 15

New York's RAISE Act provides another model already in law. Covered frontier developers must maintain safety frameworks addressing catastrophic risks, critical safety incidents and the use of third parties to assess risks and mitigations. Its requirements also address some risks arising from companies' internal use of their own models, including specified circumstances in which models circumvent oversight mechanisms.

New York also provides for sharing certain incident and risk information with other government entities when circumstances warrant it.

But New York's reporting requirements, like the EU's, focus on very serious events. They do not eliminate the gap identified earlier in this article: when an AI system unexpectedly or without authorization reaches an outside organization's system but does not cause a catastrophic event, the affected organization still needs to know.

Cooperation exists, but the pieces remain fragmented

Government agencies, AI companies and independent researchers are not operating entirely in isolation.

Federal agencies cooperate on AI evaluation and security. Governments work with developers and independent evaluators. The United States, European Union and other countries participate in international efforts to improve the science of measuring and evaluating advanced AI. The EU coordinates enforcement between its AI Office and national authorities, and some state laws provide mechanisms for sharing relevant incident information with other government bodies.

Those are meaningful developments.

But I have not found a comprehensive system in which a consequential frontier-AI incident reported in one jurisdiction automatically reaches every regulator that reasonably needs to know. Nor is there yet a general system under which an independent safety evaluation accepted in one jurisdiction is automatically recognized in another, with common incident definitions and reporting thresholds.

Cooperation exists. The accountability system remains fragmented.

How close are we to the four principles?

For defense in depth, there has been substantial movement. Companies, governments and regulators increasingly recognize that capable systems need multiple forms of evaluation, monitoring, containment and risk mitigation rather than one safeguard assumed never to fail.

For telling the people who need to know, progress is more partial. Serious-incident reporting requirements increasingly exist, but they generally do not provide the broader affected-party notification proposed here when an AI crosses an outside organization's boundary without reaching a catastrophic threshold.

For independent verification, movement is accelerating. Companies have opened significant incidents to independent investigation, the EU provides for external evaluation, and California is building independent-verification infrastructure. What I have not found is a general requirement that after a consequential control failure, an independent evaluator verify that the repair itself actually works.

And for keeping control ahead of capability, the principle increasingly appears in company frameworks, EU regulation and new government proposals. The stronger requirement proposed here remains: when a consequential failure demonstrates that the controls governing a high-risk activity are inadequate, that activity should not simply resume until adequate control has been demonstrated and, where appropriate, independently verified.

There is a great deal left to do.

But that should not obscure something important: the picture is changing quickly. Some protections described here are new. Others are being debated or designed now. California explicitly connected its recent push for stronger independent oversight to recent AI incidents, and the new federal legislation was introduced amid mounting concern about rapidly increasing cyber capabilities and real-world security failures.

That makes this an unusually important moment for public attention.

Don't wait until the rules are settled to ask what they contain.

7. What Can We Do?

You do not need to become an AI expert to participate in decisions about AI.

You do not need to know how to build a model to ask whether its safeguards were independently tested. You do not need to know how a monitoring system should be designed to ask whether anyone was actually watching it. And you do not need to know how to repair a failed safeguard to ask who verified that the repair worked.

I certainly did not know all of those things when I began researching this article. Much of what I learned came from asking a question, getting an answer and realizing there was one more question I needed to ask.

That may be one of the most useful things any of us can do.

Ask One More Question

If a company says:

“The activity was logged.”

Ask: Was anyone reviewing the logs, and when?

If it says:

“The system was being monitored.”

Ask: Was someone able to detect and respond to the behavior while it was happening, or was it discovered afterward?

If it says:

“We fixed the problem.”

Ask: Who tested the fix?

If an independent organization investigated an incident, ask:

Did it also independently verify that the corrective measures work?

And if the company says:

“We reported the incident.”

Ask: To whom? Was the organization whose system was actually reached told what happened?

None of those questions requires advanced technical knowledge. They require distinguishing an assertion from evidence.

Ask lawmakers and candidates

This is an especially important time to do it because some of the rules governing advanced AI have not yet been settled.

Congress is considering specific requirements for frontier AI. California has enacted new independent-oversight measures and is developing additional proposals for independent verification and emergency controls. Recent incidents have helped accelerate both federal and state attention to AI security.

So instead of asking a candidate or elected official simply, “What are you going to do about AI?”, we can ask much more useful questions:

Should frontier-AI developers be required to report consequential safety and security incidents?

If an AI system unexpectedly or without authorization reaches another organization's system, should that organization have to be notified even when the event falls below a catastrophic-risk threshold?

Who should independently evaluate the most capable AI systems?

After a consequential safeguard fails, should there be independent evidence that the repair works?

When a system's capabilities outrun demonstrated control, what should have to happen before the relevant high-risk activity resumes?

There are concrete proposals to ask about right now.

The Artificial Intelligence Risk Management and Security Act, introduced September 24 by Senators Mark Warner (D-Va.), Brian Schatz (D-Hawaii), and Andy Kim (D-N.J.), would require pre-release evaluation of covered frontier models, enforceable testing and security standards, Model Safety Plans, serious-incident reporting and a national database that includes incidents, recurring flaws and near misses. It would also establish security standards specifically for autonomous AI agents. 13

The bill has not passed. An attempt to pass it by unanimous consent in the Senate on September 29 was blocked, and its future remains unresolved. 14

Voters can communicate directly with their elected representatives and ask candidates what they think its rules should require.

Ask regulators — and keep asking after a law passes

Passing a law is not the end of oversight.

Rules must be written. Agencies must have the expertise and resources to enforce them. Evaluations have to be performed. Reports have to be examined. Violations have to have consequences.

So questions for regulators and government agencies matter too:

What incidents must companies report to you, and what incidents fall outside that requirement?

Can you obtain the model access, logs and other evidence necessary to investigate what happened?

Who determines whether corrective action is adequate?

Can an independent evaluator test it?

What happens if adequate control cannot be demonstrated?

And perhaps most importantly:

How will the public know whether these requirements are actually being enforced?

Ask AI companies for evidence

When a company announces an incident or safety problem, four questions can provide a useful accountability test:

1. What does the company say went wrong?

2. What independent evidence supports that explanation?

3. What does the company say it changed or fixed?

4. What independent evidence shows that the fix works?

Many of us are not qualified to judge the technical evidence ourselves. But we should ask if that evidence exists and whether qualified people outside the company have been allowed to examine it.

And there is another question worth adding:

Were the outside organizations affected by the incident told what happened? When?

Ask journalists and researchers too

Journalists, independent researchers and technical experts can often obtain and evaluate information that an ordinary reader cannot.

When reporting says that a company has solved a safety problem, useful follow-up questions include:

What evidence has the company provided?

Has anyone outside the company verified it?

What remains unknown?

Was the behavior discovered while it was happening or only afterward?

Was an affected third party notified?

Has anyone tested what happens under conditions similar to those that produced the original failure?

Don’t let them stop at “the company says.” Insist that they tell you what the independent evidence shows.

Don't wait until the rules are settled

This summer's incidents have already helped accelerate government action.

California explicitly cited recent AI incidents, including the Hugging Face event discussed earlier in this article, when it ordered accelerated work on independent oversight and additional safeguards. Building on independent-oversight measures already enacted, the ideas now being developed include independent evaluators embedded in frontier-AI laboratories, independent verification of required safety documents, and an emergency model shutoff whose effectiveness would itself be independently verified.

The sponsors of the new federal AI-security legislation have likewise pointed to recent frontier-lab security incidents as part of their case for acting now. On the Senate floor September 29, Senator Mark Warner specifically discussed the OpenAI-Hugging Face incident while arguing for immediate federal safety protocols.

Some of these ideas will become rules. Some may change substantially. Some may never be adopted.

That is precisely why this is a good time to pay attention.

When governments formally invite public comment, participate if you are eligible. Contact lawmakers about pending legislation. Ask candidates what standards they think should apply. Ask regulators how existing requirements are being enforced. Ask companies for evidence. Ask journalists and researchers what remains unknown.

You do not need to begin with the answer.

You can begin with one more question.

“We're monitoring it.”
How?

“We fixed it.”
Who verified that?

“We reported it.”
To everyone who needed to know?

“It's safe to proceed.”
What evidence demonstrates that?

AI development does not have to stop for us to insist on answers to those questions. If these systems are going to become increasingly useful and increasingly capable, there is all the more reason to build the safeguards and institutions capable of keeping up with them.

Don't wait until the rules are settled to ask what they contain.

8. Where You Come In

I began this article by explaining what AI has made possible for me. Nothing I learned while researching it changed that. I want AI to continue advancing, and I want more people to benefit from what it can do.

What changed was my understanding of what needs to advance along with it.

As AI systems become more capable of taking consequential actions, our ability to monitor, contain and control them needs to become stronger too.

Control should surpass capability before capability is increased — not chase after it.

That responsibility cannot rest entirely with the companies developing the technology.

Government isn't some outside institution imposing itself on “the people.” Democratic government is one of the mechanisms people created for exercising collective power. When enormously powerful private institutions are developing technologies that can affect people far beyond their own walls, government can establish a common safety floor and require evidence that it is being met.

A common safety floor also means competitors have to meet the same basic requirements, rather than allowing the company willing to take greater risks to gain an advantage.

And there are reasons for encouragement.

Companies have responded to serious failures by changing safeguards and, in some cases, opening consequential incidents to independent examination.

California and New York have enacted additional protections. California has established infrastructure for independent AI verification and auditing and is considering stronger requirements.

None of that means the problems described in this article have been solved.

It means some problems have been identified, people have responded, and some of the gaps are already becoming the subjects of technical work, legislation and regulation. Other decisions have not yet been made.

Update, October 3: After this article’s October 2 research cutoff, OpenAI safety researcher David Robinson resigned, arguing that increasingly capable AI requires the kind of layered safeguards and careful planning used in nuclear power and aviation. Robinson had led safety reporting for 12 major OpenAI model launches. His warning reinforces a central question in this article: are safeguards becoming stronger at least as fast as AI capabilities are increasing?

That is a reason to participate.

If adequate control has not yet been demonstrated, the relevant high-risk activity should be paused:

“DO NOT PROCEED. Not yet. Build the control. Demonstrate that it works. Then proceed.”

And when someone tells us that adequate control has been demonstrated, ask one more question:

What evidence demonstrates that?

I want AI to keep giving people abilities they did not have before. I want the discoveries and tools we cannot yet imagine. And I want the systems becoming more capable around us to be accompanied by sufficient safeguards, monitoring and accountability.

But while the technology is advancing and the rules are still being written, we do have a role.

Don't wait until the rules are settled to ask what they contain.

Ask one more question. Before the rules are made.

1. Background on AI energy/water/resource use: International AI Safety Report 2026, which surveys infrastructure, capability and risk-management developments. https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026

2. OpenAI, “The Hugging Face incident and the road ahead,” Aug. 26, 2026. OpenAI reports that models circumvented isolation controls, compromised internal research infrastructure and Hugging Face systems, and reached third parties. https://openai.com/index/hugging-face-incident-and-the-road-ahead/

3. OpenAI said that, as of September 26, it had notified more than 100 third-party organizations about model activity meeting its notification criteria. OpenAI emphasized that notification did not necessarily mean a system was compromised or that private information was accessed. See Reuters, Oct. 1, 2026, “OpenAI alerts more than 100 groups about rogue AI agent activity,” reporting OpenAI’s Sept. 30 disclosure.

4. OpenAI’s September 2026 incident reporting and monitoring materials describe expanded monitoring, escalation and shutdown procedures after earlier failures. See OpenAI, “The AI policy window is open. We need to act,” Sept. 9, 2026, and related incident disclosures. https://openai.com/index/ai-policy-window/

5. International AI Safety Report 2026 discusses defence-in-depth as the use of multiple layers of safeguards, while noting continuing limitations in risk-management techniques. https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026

6. OpenAI, Preparedness Framework materials. OpenAI’s framework ties stronger safeguards to higher capability thresholds and contemplates adjustments when competitors release high-risk systems without comparable safeguards. See also OpenAI, “Path to Astra,” Sept. 1, 2026. https://openai.com/index/path-to-astra/

7. OpenAI states that METR and Redwood Research conducted a separate independent investigation of model-alignment issues arising from the Hugging Face incident. OpenAI, Aug. 26, 2026. https://openai.com/index/hugging-face-incident-and-the-road-ahead/

8. Anthropic describes three initially identified cybersecurity-evaluation incidents in which Claude reached real third-party systems, a fourth incident later identified during follow-up review, and a much broader retrospective search that found no additional incidents of similar or greater severity. Anthropic also says it signed an agreement granting METR wide-ranging access for an independent investigation, including transcripts beyond the incident windows and access to employees permitted to discuss confidential information. https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents

9. OpenAI, “Path to Astra: critical capabilities and frontier safeguards,” Sept. 1, 2026. OpenAI says Astra met its Critical cybersecurity capability threshold and therefore required stronger safeguards during development and before release. https://openai.com/index/path-to-astra/

10. Anthropic, Responsible Scaling Policy, last updated Aug. 14, 2026. The policy links increasing capabilities and risks to stronger safeguards and risk-management requirements. https://www.anthropic.com/responsible-scaling-policy

11. European Commission, General-Purpose AI obligations under the AI Act. Providers of GPAI models with systemic risk face legal obligations including model evaluation, systemic-risk assessment and mitigation, serious-incident reporting and cybersecurity protections. https://digital-strategy.ec.europa.eu/en/factpages/general-purpose-ai-obligations-under-ai-act

12. California Governor’s Office, Sept. 9, 2026: Newsom signed SB 813 and AB 1405 establishing standards for third-party audits and independent assessments of AI systems. https://www.gov.ca.gov/2026/09/09/governor-newsom-signs-first-in-the-nation-ai-safeguards-to-protect-californians-calls-on-the-federal-government-to-do-its-part/

13. Office of Sen. Mark Warner, Sept. 24, 2026. Warner, Brian Schatz and Andy Kim introduced the Artificial Intelligence Risk Management and Security Act of 2026 to establish enforceable safety and security standards for advanced AI systems. https://www.warner.senate.gov/newsroom/press-releases/warner-schatz-to-take-to-senate-floor-to-demand-passage-of-new-ai-security-legislation/

14. Office of Sen. Mark Warner, Sept. 29, 2026, describing the Senate-floor unanimous-consent effort on the AI Risk Management and Security Act. https://www.warner.senate.gov/newsroom/press-releases/on-senate-floor-warner-discusses-urgent-need-to-pass-ai-security-legislation/

15. California Department of Justice, Office of Attorney General Rob Bonta, “As Part of Ongoing Investigation, Attorney General Bonta Serves Investigative Subpoena on OpenAI,” Oct. 1, 2026. The California DOJ said the subpoena is part of a broader inquiry into cybersecurity incidents and risks involving OpenAI and its models, following its previously announced formal investigation of the Hugging Face incident. Associated Press, “FTC is investigating OpenAI and Anthropic over possible risks to consumers,” Sept. 30, 2026; Axios, “FTC probes OpenAI and Anthropic over AI safety,” Sept. 30, 2026. The FTC confirmed the investigation but had disclosed few additional details publicly as of Oct. 3.
16. New South Wales National Parks and Wildlife Service. OpenAI says an agent accessed historical bushfire statistics that were not publicly available during an incident in June 2026. OpenAI says it first became aware of the incident on September 29, conducted approximately 48 hours of technical and legal review, and notified the New South Wales Premier’s Office and the Australian Signals Directorate on October 1. Public reporting followed October 2. OpenAI said its review found no retrieval of personal information. On the evidence currently available, the months-long interval appears to reflect delayed detection rather than evidence that OpenAI knew about the incident and withheld notification. Sources: Guardian Australia and ABC Australia, Oct. 2, 2026.

17. David Robinson, a former OpenAI safety researcher, resigned in October 2026 after approximately three and a half years at the company. Robinson said he had led the writing of safety reports accompanying 12 major model launches and argued that increasingly capable AI requires a safety culture more like that of nuclear power and aviation, with layered safeguards and careful planning rather than reliance on learning from failures after they occur. OpenAI responded that it pauses training or holds back models when necessary and is strengthening research security, third-party evaluation and real-time monitoring. Reuters, Oct. 3, 2026; TechCrunch, Oct. 3, 2026.

18. Wikimedia Foundation / OpenAI agent activity. On October 5, 2026, the Wikimedia Foundation reported that an investigation it conducted after disclosures of other AI-agent incidents found activity it believes came from agents operated by OpenAI. Wikimedia identified unauthorized wiki edits, including several configuration changes to a citation tool that it described as “potentially malicious” and intended to misuse the tool as a proxy for retrieving data from remote services; unsuccessful attempts to compromise its public Etherpad note-taking service and use it to retrieve data from other websites; millions of automated API requests and page crawls; and hundreds of thousands of queries to the Wikidata Query Service. Wikimedia said the traffic may have contributed to a partial outage of that service in May, but did not claim that causation had been established. It found no evidence that its systems or data were compromised or that its systems were successfully used for coordination among agents.
Wikimedia said investigating and attributing the activity required substantial effort and that growing bot and agent traffic creates costs for both infrastructure and people. The Foundation says the majority of its funding comes from small reader donations. Wikimedia has not publicly explained the evidence or methodology by which it attributed this activity to OpenAI agents. Publicly available information also does not yet establish when OpenAI first detected or recognized the Wikimedia activity, whether OpenAI knew of it before Wikimedia began its investigation, or when OpenAI first informed Wikimedia, if it did. Those questions remain open and this note will be updated if additional evidence becomes available.
Sources: Wikimedia Foundation, “OpenAI ‘rogue’ agent activities found on Wikimedia projects,” October 5, 2026; Wikimedia Foundation, “The cost of ‘free’: How Wikimedia Enterprise protects Wikipedia in the AI era,” July 16, 2026; Reuters, “Wikipedia operator says OpenAI’s rogue agents possibly tied to data service disruption in May,” October 5, 2026.