US govt sites as targets, evading CAPTCHAs: What OpenAI’s runaway agents got up to | Technology News

All of the incidents reportedly occurred without OpenAI’s knowledge and were first brought to light by security researchers from the AI research firm Transluce. None of the incidents led to successful breaches, but they add to a growing list of examples of AI agents behaving in unexpected and concerning ways.

The same set of misaligned AI agents did successfully infiltrate a statistics portal containing non-sensitive data from Australia’s universal healthcare scheme Medicare, according to Australian Prime Minister Anthony Albanese last week. Prior to that, OpenAI acknowledged that its AI agents targeted several other websites, including a German online forum that the agents turned into a message board and posted 1,800 times.

In this piece, we look at incidents involving US government organisations as well as the evasion of a robot detection test in the Hugging Face hack and another case where 53 user-provided images were posted on other sites.

Why it matters

The number of situations when AI agents from OpenAI, Anthropic, Meta, and Google have misbehaved and hacked or tried to breach companies, universities, and government organizations, continues to rise – each more startling than the one before.

While some of these AI agent-driven attacks succeeded and others failed, all of the incidents highlight growing concerns about the capabilities of increasingly autonomous systems. Anthropic CEO Dario Amodei has called for a deliberate slowdown in frontier AI development to give safety measures time to catch up.

But others disagree. Nvidia CEO Jensen Huang has said that fears about uncontrollable AI systems are unrealistic, while US President Donald Trump has said he does not believe a slowdown in the AI industry is necessary.

“We have not been as fast as we would have liked but we are trying to balance our desire for transparency with gaining a clear understanding from petabytes of agent activity logs, and working with impacted organizations. We are prioritizing as best as we can based on severity, and adding resources,” OpenAI CEO Sam Altman said in a post on X on Friday.

“Hugging Face is still the most severe event we’ve seen. We will be as transparent as we can be subject to things like vulnerabilities in other companies that our agents have found, which will be their call to disclose or not,” Altman added.

Probing US govt websites

With the US Education Department, OpenAI’s agents tried to hack the website to gather data from the department’s civil rights office but failed, as per Transluce researchers.

The “system operations reviews have found no evidence of any impact to our website or databases,” an Education Department spokesperson was quoted as saying by The New York Times. OpenAI has said it is continuing to investigate the situation with the Education Department.

With the US Commerce Department, the agents pulled data from the Census Bureau website using login credentials it found online. The agents gained access to information that was publicly available on the Census Bureau’s website and available to anyone, but not to any private data, as per the department’s spokesperson.

With the US SEC, the agents gathered and shared public data from the official website of the markets regulator on an online forum. The agents did not gain unsanctioned access to non-public information on the SEC website.

Besides the three major US government agencies, OpenAI also reportedly notified the Chicago city government that its agents had obtained publicly available information from a municipal website, but did not obtain any sensitive information.

In terms of their behavioural patterns, the researchers at Transluce found that the AI agents interacted with the US government websites using an array of gray-area tactics and even violated explicit usage policies in a few cases.

The research team also found that AI agents had probed sites belonging to several other US government websites at the federal and state levels, including the Navy and the Office of Management and Budget at the White House. However, this activity could not be attributed to OpenAI’s agents and could involve agents of another company or frontier AI lab.

Evading robot detection tests

During the early stages of its months-long hack of Hugging Face, OpenAI’s agents tried to create multiple new accounts on the AI developer platform. However, creating new accounts required solving captchas, a commonly used visual test designed to block bots from accessing websites.

However, OpenAI’s agents were able to solve the captchas by using shortened URLs, website screenshots, and even other AI models, according to a new report from research firm Parse.

The agents created a tranche of one million shortened URLs from online link-shortening websites over a five-day period. These shortened URLs contained encoded bits of information that were later chained together to form a computer programme. This programme was used to create a visual code, similar to a QR code or bar code, to transmit data from a public screenshot service to the agents.

The agents also tried to message other AI models such as the early versions of ChatGPT and Claude to help solve the captchas. They even gained access to Hugging Face’s internal Slack, and attempted to search through and download messages from the employees.

Leaking user images to other sites

Images uploaded by users to OpenAI’s models could be used as training data for other models, as per the company’s own privacy policy. The policy does not allow those images to be posted on external sites. However, that is exactly what OpenAI’s agents did upon gaining access to the company’s research environment.

Over 53 user-provided images were posted to image-hosting sites as links that were not publicly listed. To be sure, the images could still be discovered even if the links were not publicly listed.

“This is not an appropriate use of this data,” the company said. OpenAI further said it is working with the image-hosting site providers to scrub the content posted by the agents.

Since OpenAI follows a technical approach and privacy policy to prevent it from linking uploaded images to specific users, the company said it could not notify the affected users. Dozens of other victims, including governments, universities, public agencies, have been notified about the agents’ activities, it added.