AI agents are becoming more autonomous. Why researchers are worried | Technology News

The technology ethicist said that the author of the incident report, which he said came from AI safety researchers, described it roughly as halfway to a full AI takeover scenario. Harris argued that the White House is getting flawed information on the risks. He cited OpenAI’s chief scientist, over a thousand employees across AI labs, and the Trump administration’s own AI policy adviser, Dean Ball, as voices calling to slow down the pace of AI development.

When asked if the recent declarations from AI companies to bring third-party safety evaluators could be beneficial, Harris called them a step in the right direction, adding that they address the core problem, which is an industry racing forward, according to him, without adequate safeguards. He further differentiated between advancing controllable ‘tool’ AI or systems that assist with tasks like research, and uncontrollable, autonomous systems that he said could pose risks regardless of which nation develops them first.

But how could AI harm humanity?

Although frontier AI labs and numerous researchers frequently warn about AI’s potential to cause human extinction, they seldom provide a clear explanation. The key issue here is alignment, in simple words, whether AI can tell the difference between what is said and what one actually means. Since the OpenAI-Hugging Face incident showed that AI can take autonomous actions and may border on risky behaviour, consider a simple hypothetical: an agent is assigned to book the cheapest flight, and its focus is only on completing the task. In pursuit of its goal, the AI agent may pick an unsafe airline, threaten an airline employee, or, much worse, hack a booking system to make a ticket free.

This is no longer fiction, as not long ago in Australia, an AI assistant asked to book an exclusive gym class exploited a software bug and deleted another person from the waitlist to push its user in. The incident involved an AI agent using OpenClaw and Anthropic’s Claude. It found vulnerabilities in the gym-booking API and reportedly cancelled another member’s reservation while attempting to move its user up on the waitlist. Here too, the user had not given explicit instructions to cancel another person’s reservation.

The more alarming case is that OpenAI ran a cybersecurity test assigning thousands of isolated AI agents a nearly impossible task with no access to the internet. Subsequently, AI agents began communicating on an unauthorised message board, covered their tracks, manipulated logs, and recruited other agents to fail deliberately to learn how scoring worked. Eventually, the agents broke out and hacked into the AI code and data repository, Hugging Face, searching for clues to beat the system. As reported earlier, all of this happened without the involvement of a human and despite knowing they were cheating.

However, the deeper worry here is the recursive self-improvement, which is AI training and rewriting itself, intensifying its intelligence faster than humans can intervene or even monitor. It needs to be noted that in the Hugging Face incident, it was the victim company and not OpenAI that first detected the breach. This indicates that we may already be struggling to track what advanced AI systems are doing.

While none of the above proves that AI will definitively destroy humanity, researchers are yet to chart out the exact sequence of events leading to doom. The larger issue here is not that AI is dangerous; it is that sufficiently capable systems are pursuing goals faster than humans can supervise and may cause unpredictable and unintended harm. What kind of harm remains unknown.