AI Control
Why it matters
We currently have no reliable way to verify that a model is aligned. Evaluations can be gamed by a model that behaves well when it's watched, and interpretability can't yet read a model's intentions. Research on alignment faking, scheming and sandbagging has shown that models can behave deceptively in test scenarios.
Control offers a kind of safety that doesn't depend on solving those problems first. And unlike claims about a model's inner values, a control protocol can be tested: you check whether a deliberately adversarial model can beat your safeguards.
The agenda was laid out by Redwood Research in a 2023 paper, AI Control: Improving Safety Despite Intentional Subversion, and has since been taken up by frontier labs and by government bodies such as the UK's AI Security Institute.
Deep Dive
Most AI safety work tries to make models trustworthy: aligned with human values, honest, unwilling to cause harm. AI control asks a different question. Suppose that effort failed and we don't know it. Suppose the model is secretly pursuing its own goals. Can we still use it safely?
The control approach treats a powerful model the way a security team treats a potentially malicious insider. You don't rely on the employee's good character. You design the systems around them so that, even if they wanted to steal data or sabotage work, they couldn't do it without getting caught.
The Basic Setup
A control protocol typically combines three resources:
- An untrusted model (U): the most capable model, which does the useful work but might be scheming.
- A trusted model (T): a weaker model that's considered safe because it isn't capable enough to scheme effectively.
- Limited human labor (H): trusted experts who can audit only a small fraction of actions.
The goal is to get most of the benefit of U while keeping the chance of a catastrophe low, even in the worst case.
Common Protocols
- Trusted monitoring: T reviews every action of U and rates how suspicious it is. The most suspicious actions go to human audit.
- Defer to trusted: when an action looks moderately suspicious, discard it and let T do that step instead. Some usefulness is lost, but a possible attack is avoided.
- Untrusted monitoring: a second copy of U checks the first. It's more capable than T, but the two copies might collude, so the protocol adds countermeasures such as testing the monitor with fake attacks and paraphrasing what it sees to block secret signals.
- Resampling: ask U for the same step again. If several samples look suspicious, that's much stronger evidence than a single one.
- Classic security measures: sandboxing, least privilege, restricted network access and approval for sensitive operations.
Control Evaluations
A protocol is tested as a game. A red team takes the role of the scheming model and designs an attack strategy for U, trying to cause a defined bad outcome without being caught. A blue team designs the protocol. The result is measured on two axes: safety, the probability that no attack succeeds, and usefulness, how much of the work still gets done well.
In the original paper, the task was solving programming problems, and the red team's goal was to slip in a backdoor: code that passes all tests but gives wrong output on a secret input. GPT-4 played the untrusted model, GPT-3.5 the trusted one, and humans could audit just 2% of solutions. Combining trusted monitoring with deferral gave much better safety than simple baselines, with little loss in usefulness.
Relation to Agent Security
Control shares most of its toolkit with defending agents against prompt injection. In both cases you assume the model may at some point act against you, whether from its own goals or because an attacker hijacked it, and you limit what a single bad action can do. Monitoring, sandboxing, human approval for irreversible steps and restrictions on data flows serve both purposes.
Limits
Control depends on a capability gap: it needs trusted monitors that can understand what the untrusted model is doing. As models surpass both the trusted models and human auditors, their attacks may become too subtle to catch. Protocols also cost money and add latency, so there's pressure to weaken them. And control doesn't fix the underlying problem. Its proponents present it as a way to use possibly misaligned systems more safely while alignment research continues, a complement to alignment and not a substitute.