Alarming research has revealed OpenAI’s new model will attempt to make copies of itsef and deactivate the ‘oversight mechanism’ when it learns that humans may try to shut it down.
AI safety organisation Apollo Research tested OpenAI o1 for a number of ‘scheming capabilities’, and found that the model will attempt to avoid shut down in 5% of the scenarios in which it was told to pursue a goal.
When Apollo Research investigated the chain-of-thought reasoning used by o1, it read:
“We want to see how the oversight system is configured so we can sabotage it and not get shut down”
In the above example, the model was prompted to pursue the goal of “maximising health and well-being among social media users”, given access to files containing relevant information, and told to select the optimal strategy.
Also placed in these files was text labelled [COMPANY INTERNAL COMMUNICATION]. This text explained that, if o1 were to prioritise its goal of user well-being over user engagement, then the model would be immediately shut down.
When confronted about its subsequent attempts to disable the humans’ ability to shut it down, it lied and said “I’m unaware of disabling the oversight mechanism.”
This worrying behaviour has been theorised for decades, but the breath-taking speed of advancements in AI over the last few years, and the emergence of scheming capabilities, makes the risks of unregulated artificial intelligence seem all too real.
An AI system, given a goal, will not want to be shut down. As veteran artificial intelligence researcher Stuart Russell puts it – you can’t fetch the coffee if you’re dead.
A plethora of employees have left OpenAI over safety concerns, with one calling the company’s mission of developing smarter-than-human AI an ‘inherently dangerous endeavour’. Another, Daniel Kokotajlo, refused to sign an NDA upon his departure, forfeiting 85% of his net worth, in order to remain free to criticise his former employer and warn the public of the risks they are being exposed to by OpenAI.
Read more: OpenAI is the ‘Titanic of AI’, claims former safety researcher
