As warnings about the dangers of unchecked artificial intelligence grow louder, Anthropic has disclosed that it had to intervene in several attempts by researchers to use its AI models for work that could potentially aid the creation of biological weapons. In a report released Thursday, the San Francisco-based company said it banned accounts that had used its chatbot Claude for research into highly contagious diseases over the past eight months.
The report, titled "Detecting and Countering Misuse of AI," details five cases where Anthropic suspected the researchers were affiliated with a government or military in a country it had banned, or took deliberate steps to conceal their location and identity. The company declined to specify where the researchers were based, but its terms of service prohibit use from China, Iran, and Russia.
In one example, researchers used a third-party platform to attempt to complete a grant application for work that would involve repeatedly mutating chikungunya, a mosquito-borne virus that causes severe pain and fever in humans. The research, which would be conducted at a military institute, aimed to make the virus more infectious in live animals. Anthropic said it blocked all attempts by the researchers.
Another case involved a researcher using Claude to assist in research on making bird flu more virulent and damaging to mammals. Bird flu has a global fatality rate of around 50% in confirmed human cases. Anthropic noted that the researcher only received clerical help from its weakest class of models.
Balancing caution and scientific inquiry
Anthropic clarified that it did not find any concrete instance of a scientist attempting to use its technology for nefarious purposes, and that these use cases could have been part of beneficial scientific inquiry, such as vaccine research. Nevertheless, it said it erred on the side of caution and shut down the accounts.
"Biological misuse is one of the most serious risks of frontier AI models," the report states. "It has long been a concern that AI models might one day reach the level of capability where they can help to make existing pathogens more dangerous—or create entirely new ones. Without the correct safeguards, such capabilities could have catastrophic consequences."
The report also acknowledges that sophisticated threat actors are becoming more adept at evading detection, using the dual-use nature of biology to maintain plausible deniability. "More sophisticated actors can hide their intent, extracting assistance from an AI model in interactions that look plausibly beneficial, but when put in context and analyzed holistically, can provide clear warning signs of misuse," it says.
Beyond bioweapons, Anthropic found that governments like China, Iran, and Mali had used Claude to collect information about journalists and dissidents. Others used it for hacking campaigns, identity theft, and creating malware. The company hopes the report will help other developers recognize similar patterns and strengthen collective defenses.
The disclosure comes amid intense scrutiny of AI companies following reports of security breaches. Earlier this week, Anthropic researcher Jacob Coxon resigned, warning that the technology "could kill us all by the end of the decade" and that companies are prioritizing the race toward superintelligence without regard for risks. Several other researchers from Anthropic, OpenAI, and Google have shared similar warnings.
The report has also fueled calls for regulation. A poll by Data for Progress showed that 68% of likely US voters support a proposal by Sen. Bernie Sanders and Rep. Greg Casar for legislation that would ban the development of "superintelligent" AI and pause advanced AI development until a federal regulatory body is established. Casar emphasized that relying on AI corporations to voluntarily block weapons of mass destruction is insufficient.
As the debate over AI safety intensifies, the incident highlights the challenges of balancing innovation with security. For the Indo-Pacific region, where AI development is a strategic priority for countries like China, Japan, and South Korea, the implications are significant. SoftBank's AI bets and the need for a hybrid state-market model are just some of the regional discussions that intersect with these global concerns.
