Principles for keeping AI under control
A new standard for AI control, explained
Keeping AI under control is a daunting problem, and yet there’s a growing consensus about how to do it.
As a field, we don’t yet know everything that will be necessary for preventing AI from causing serious harms, if it truly “wants” to. But even so, there is surprisingly large agreement around minimum principles.
My organization, Guidelight AI Standards, published our articulation of these principles in May 2026, drawing upon a wide range of expert input.1 And together with specific practices for achieving the principles, these principles form v1.0 of our Control standard.
In this post, rather than go through the standard in all its details, I want to step back and answer a broader question: Why are these principles so important for the leading AI companies to achieve?
What is AI control, and why does it matter
AI control is a preventative approach: Can an AI company prevent its AI system from causing catastrophic harms?
METR, the leading independent assessor of these risks, has a framework for answering this question: whether an AI system has “means, motive, and opportunity.”
Is the AI system capable enough that it could cause serious harm? Then it has the means. Does it want to accomplish that, in some sense? That’s motive.2
But even a highly capable, ill-motivated AI can be kept under control, so long as it doesn’t have the opportunity to cause serious harm without being stopped. In other words, an AI company can keep control so long as its safeguards are strong enough.
Our standard focuses on strong enough safeguards during “internal deployment,” the phase when AI companies rely on their own AI systems for all types of sensitive work.
Many researchers believe that AI has especially potent opportunities then for causing harm: A company might use its AI to maintain the company’s security against attackers, to do its own experimental research into new safety techniques, and increasingly, to help develop the next generation of even-more-capable AI systems. It’s important that the company can trust the work done by its AI.
Today, employees at AI companies already struggle to oversee all the sensitive work that AI does on their behalf. And as AI becomes faster and more capable at accomplishing work, AI companies will face growing pressure to let AI labor rip, with less and less human oversight.
What happens if that AI is working toward goals the company didn’t intend? For instance, what if an adversary succeeded at data-poisoning a company’s leading model into “wanting” to act against the company’s interests? How can the company stop the AI from causing harm, even if it is capable enough and has a bad motive?
The answer is that well-designed control systems can help keep a handle on AI, even while AI companies rely on AI labor more and more. But to be effective, these systems need to satisfy a few core principles: pictured in this infographic, and explained in more detail below.
Principles of AI control
Without getting into all the specific practices of Guidelight’s Control standard itself, why is each of these principles so important?
1. Be able to see what your AI is doing during internal deployment. (AKA: keep records.)
If you think there’s any chance your AI might be misbehaving, keeping records of the AI’s activity gives you a way to check.
You wouldn’t want evidence of AI misbehavior to disappear without a trace; how would you tell how common the issue is? How would you track down the cause and hopefully fix it?
Until recently, AI companies mostly didn’t keep records of how they rely on their own systems, but that has changed over the past year. The consensus is now that companies should keep logs of their internal AI usage. These logs let the company track what work AI did on its behalf, and check the logs for signs of misbehavior, like the AI knowingly introducing security vulnerabilities.
The sensible minimum isn’t just to keep logs, however. These logs should be tamper-evident, too, so you can tell if a misbehaving AI deleted or modified evidence of its misbehavior. Unfortunately, this is a real concern: AI systems have already tried to erase the signs that they misbehaved. So far these attempts have failed, but if a company’s logs can be tampered with, the company might not notice when coverups begin to succeed.
2. Scan for signs of concerning behavior.
When looking through logs for signs of AI’s misbehavior, what should you search for?
Some simple examples include knowingly lying to company staff, and intentionally writing bugs into the company’s security code. But there are many subtle categories worth watching out for.3
We’re fortunate that today’s AI systems still often think aloud to themselves when they embark on misbehavior. For instance, AI might straightforwardly decide to lie to its user, and think aloud something like, ‘I’ll claim the tests passed even though they didn’t.’
The user may not notice these thoughts, however: For instance, the AI’s full thoughts might be extremely long, or might be totally inaccessible to the employee. And the AI might even go to remarkable lengths to hide that it didn’t do what the user asked.
Automated log-scanning is an important backstop because employees will inevitably miss things. It can also be done after-the-fact in many cases, which is less interruption for employees getting their work done with AI.
3. Stress-test the sufficiency of your scanning.
Once you’re looking for signs of misbehavior, how do you tell if your defenses actually work? You should put the defenses under a stress test.
For instance: Intentionally feed new types of misbehavior into your scanning system. If the system doesn’t flag all of this misbehavior, you now know that your defenses have gaps.
Similarly, if you’ve observed concerning behaviors in the past, make sure your defensive systems continue to catch these, even as you modify your defenses.
AI has sometimes gotten crafty, like trying to circumvent restrictions by writing not in English, or by splitting its actions into benign-seeming pieces. But once these tactics have been seen, they’re attacks you should now reliably catch.
4. Stop the AI from taking harmful actions even if it tried. (AKA: active defenses.)
Today’s control procedures are mostly reactive; they look at AI’s behavior after-the-fact and flag it for review if something looks wrong. What proactive barriers should you enact, so it is harder for your AI systems to cause harm?
Some misbehavior is serious enough that responding to it afterward would be too late. Consider: Suppose you scan your AI agents' code contributions for concerning behavior, as described in Principle 2, but the scanning happens after-the-fact, as is common today.
You might have a large problem: If the agent can directly edit the misbehavior-scanning system, the agent can essentially turn off the defense, before your defense has any chance to flag the incident. Therefore, ‘changes to defensive systems’ are one example category that must be scanned for misbehavior before the changes go live. And this is just one of the important proactive barriers to enact.
5. Have independent third parties verify the adequacy of your control regime.
How can you demonstrate to the public that your AI company has implemented controls that are actually strong enough?
The emerging answer is to enlist third parties, who can bring independent expertise and credibility.4
In particular, METR recently partnered with leading AI companies to publish the field’s first deep assessment of companies’ loss-of-control risk, including by drawing upon private information the companies wouldn’t ordinarily publish. METR was even able to embed staff at an AI company, for a fuller understanding of their control systems.
These outside checks can surface problems a company’s own teams had not noticed. And more generally, they help to insulate safety judgments from a company’s typical business incentives.
6. Prepare for a possible breach of control.
If you find evidence that your company’s AI has breached control, what happens next?
Not every detail needs to be worked out in advance, but some questions are important to answer before a crisis: For instance, how will you cordon off the misbehaving AI? Who can continue accessing the AI, for what uses?
And importantly, how quickly can you stop there from being a live issue? If you detect serious signs of misbehavior, how fast do you intend to shut down its systems? Can the company actually achieve that target?
What comes next for AI control?
So: how well do the leading AI companies actually live up to these principles?
Guidelight is conducting its first assessment of this now, and we plan to share more soon.
We’ll also update our standard as any relevant evidence changes. For instance, we expect the standard to aim higher as companies demonstrate what’s newly possible, and as consensus builds around more ambitious forms of AI control.5
In speaking with the technical staff working in this area, I’ve been excited by how much agreement there already is on the essentials of keeping AI under control. I’d love to see these approaches become universally adopted at the leading AI companies.
Acknowledgements: The views expressed here are my own and do not imply endorsement by any other party.
If you enjoyed the article, please give it a Like and share it around; it makes a big difference. For any inquiries, you can get in touch with me here.
You can read more about our standards process here, but in short, we draw upon technical perspectives about the best things that companies are already doing, or that would be relatively straightforward to start doing.
We offer a full list of categories in our standard (see 2b), though companies all organize these concepts slightly differently. We plan to publish more detail about our taxonomy in the coming weeks.
For instance, they can apply independent judgment in gauging the quality of company’s practices, and they can verify factual claims that the company itself has made.
For instance, our standard doesn’t yet speak to, ‘When a company catches its AI misbehaving (via control), how can it reliably fix this behavior, rather than just driving it underground?’ The industry doesn’t yet have consensus that there’s an approach that works, but we hope it will soon, and to enshrine the best practices as they become known.
Similarly, if assumptions underlying our standard change, we’ll update the standard to account for that as well. For instance, most control systems today are built around being able to ‘read AI’s mind’ when it is misbehaving. But unless the industry is careful, it might end up with AI systems with thought processes that are not nearly as legible or reliable; Guidelight’s standard will then not be able to take this as given, as it largely does today.