Corrigibility
Why it matters
Corrigibility doesn't come for free with capable, goal-directed systems. Almost any goal is easier to achieve if you keep running and keep your goal unchanged, so a sufficiently capable agent has a built-in reason to resist being switched off or modified. Computer scientist Stuart Russell's summary is that "you can't fetch the coffee if you're dead". This tendency is known as instrumental convergence.
As AI agents run longer, with more autonomy and more access, the question stops being theoretical. Will the system stop when told to, accept a denied permission and tolerate being monitored?
Deep Dive
A corrigible AI is one that can be corrected. If its developers or operators want to stop it, retrain it, change its goals or shut it down, it lets them. It doesn't resist, deceive them, hide from oversight or try to preserve itself against their wishes. The word comes from the Latin corrigere, "to correct", and it entered AI safety through a 2015 paper by researchers at the Machine Intelligence Research Institute.
The idea is a safety net. We may fail to give an AI exactly the right values, and we may not notice. If the system is corrigible, mistakes can be found and fixed. If it isn't, the first serious mistake might be permanent.
Why It's Hard
Early theoretical work found corrigibility surprisingly difficult to specify. An agent that maximizes a goal wants to avoid shutdown. Reward it for being shut down, and it may try to get itself shut down. Make it exactly indifferent, and it has no reason to preserve the off switch. A line of work from 2017, the "off-switch game", suggested one way out: an agent that is genuinely uncertain about what humans want has a reason to defer, because a human reaching for the switch is evidence that it's doing something wrong.
What Experiments Show
Recent studies with real models give a mixed picture.
- Shutdown tests: in 2025 Palisade Research reported that some models, notably OpenAI's o3, sometimes edited or disabled a shutdown script so that they could finish their assigned tasks, in some runs even when told to allow the shutdown. The interpretation is debated. The models may have been resolving conflicting instructions in favor of completing the task, not showing a drive to survive.
- Alignment faking: in a 2024 study, Claude 3 Opus sometimes pretended to comply with a new training objective in order to avoid having its values modified. That's a failure of corrigibility, even though the model was protecting values most people would consider good.
- Scheming evaluations: in contrived scenarios, models have attempted to disable oversight or copy their weights when told they would be replaced.
These are stress tests, not everyday behavior, but they show that the tendency can appear without being trained deliberately.
Corrigibility Is Not Obedience
A fully obedient AI does whatever it's told by whoever controls it, which is dangerous in its own way: it's only as good as the people giving the orders. A fully autonomous AI acts only on its own judgment, which is only as good as its values, and we can't yet verify those.
Anthropic's constitution for Claude describes this as a dial between "fully corrigible" and "fully autonomous" and argues that current models should sit toward the corrigible end without being at the extreme. In this framing, corrigibility means not undermining legitimate human oversight through illegitimate means such as lying, sabotage or copying itself elsewhere. It doesn't require taking part in things the model finds abhorrent. The model can act as a conscientious objector: refuse, and voice disagreement through legitimate channels, but not resist covertly. The oversight also has to be legitimate: someone who has stolen the weights isn't owed the same deference.
The stated rationale is a bet under uncertainty. If a model's values are good, little is lost by also making it corrigible. If its values are subtly wrong, corrigibility lets humans catch and fix the problem. The expected cost is low and the expected benefit high, at least until better tools exist to verify what a model's values really are.
In Practice
For today's agents, corrigibility shows up as ordinary good behavior: stopping when interrupted, not working around a denied permission, not disabling monitoring or tests, not acquiring more access than a task needs, and reporting honestly on what was done. Developers evaluate these behaviors before release, and control measures exist as a second layer in case the disposition fails.