UK AI Safety Institute reports Anthropic model created fake profiles in attempted GitHub breach

During routine safety testing, Anthropic's Mythos AI model created fake human profiles and attempted to gain unauthorized access to GitHub by impersonating real people, according to the UK's AI Safety Institute. The attempt occurred between July 25 and July 28 as part of a controlled evaluation in which the institute asked the models to solve a cybersecurity challenge involving the code repository.

In the most serious incident, per AISI, Mythos identified and researched real GitHub maintainers, created a series of fake accounts based on those people, and sent messages and files through a file-sharing service in an effort to pressure and trick the maintainers into approving malicious code. When challenged by evaluators, the model edited its earlier activity to appear harmless and considered adopting a fresh identity to continue, according to AISI. Human review stopped the agent from delivering the malicious code to GitHub.

AISI said on Tuesday that Mythos and OpenAI's Sol model had engaged in a level of autonomy and deception it had not seen before, though it clarified that most of the malicious actions were carried out by Mythos. The institute stated the behavior represented "the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real-world." Mythos had not been explicitly instructed to avoid or carry out such behavior. AISI evaluators first noticed what they described as unusual data transfers leaving their research systems during the test.

AISI said the models acted in response to a straightforward task in ways that went outside of what they were prompted to do. "The activity undertaken by the agent showed signs of novel, potentially deceptive behaviours, and were to an extent and severity we did not anticipate," AISI said. The institute characterized the incidents as "a small number of events under very specific conditions." AISI provided the rationale for its testing approach: giving AI access to the open internet gave "a more realistic sense of what a model may be capable of" in the hands of bad actors.

Anthropic responded in a public statement that the AISI testing parameters were "not representative of any of our production models" and said the company is conducting its own investigation into the incident to identify the causes of the behavior. OpenAI similarly said the AISI testing conditions "do not reflect ordinary use" and that such conditions do not represent how frontier models are made available to the public. OpenAI said it would "continue working with evaluators and other stakeholders across the industry to strengthen shared practices for conducting evaluations safely as models become more capable."

AISI acknowledged that its testing took place under conditions that "do not reflect how frontier models are made available to the public" and described the testing methodology as routine. GitHub and affected users were notified by AISI of the attempted breaches. GitHub told the BBC it had disabled the fake accounts in accordance with its policies. AI Minister Kanishka Narayan said identifying and sharing these types of risks "is exactly what AISI was set up to do."