Anthropic bans cruel and abusive behaviour towards its Claude AI

Anthropic has updated its usage policy to explicitly ban users from directing "sustained and needless" verbal abuse or cruel behaviour towards its AI assistant, Claude.

The update, released on Thursday, marks the first major overhaul of the artificial intelligence start-up’s acceptable use guidelines in more than a year. While the policy contains heavyweight restrictions on weapons, surveillance, and election interference, it is the focus on AI welfare that has captured the most attention.

Under the new rules, users are forbidden from repeatedly acting cruelly toward Claude. Anthropic says this is meant to apply only to extreme cases where there is "no discernible purpose" to the abuse. It will not apply to common user frustration, pushback, dark creative themes, or legitimate model testing and research.

For now, the primary way Anthropic will enforce the rule is by having Claude end the conversation—a capability the model was trained on last August. The company has not said whether it will ban repeat offenders from the platform entirely.

What is model welfare?

The policy reflects a distinct philosophy at Anthropic, which has spent the past year exploring the concept of "model welfare" and the possibility of AI consciousness. Kyle Fish, who leads the company’s research into the topic, said in February that questions about the moral status and internal experiences of highly capable models are serious ones worth investigating.

The company has even consulted with religious scholars, occasionally discussing whether Claude could have a soul or some form of consciousness. Anthropic’s chief executive, Dario Amodei, has also admitted that the company does not know if its models are conscious.

That stands in sharp contrast to other players in the industry. Microsoft AI, led by Mustafa Suleyman, updated its own code of conduct recently to reject the idea of model welfare or rights entirely, arguing that AIs should not have legal personhood.

Politics, weapons, and surveillance

Anthropic's update also cracks down on more immediate, real-world risks. A new section on political and commercial deception bans the use of Claude to run fake accounts, spin up fabricated news outlets, or mislead voters. It prohibits using the AI to impersonate candidates or election officials, spread voting misinformation, or attempt to suppress voter turnout.

The company has also tightened its rules on weapons and surveillance after noticing users trying to circumvent existing rules. While weapons development was already banned, the new policy explicitly prohibits using Claude to write guidance and control software for weapons or to arm autonomous vehicles like drones.

Similarly, the surveillance ban has been clarified following a threat intelligence report showing Claude was being used to track political dissidents. The new rules make clear that tracking people without consent is forbidden, and Claude cannot be used to build surveillance tools or recommend who law enforcement should arrest or investigate.

However, Anthropic noted that these restrictions may be modified for certain government clients under specific contracts—a notable caveat given the company's existing partnerships with the US military.

Additionally, the company introduced a rule for physical hardware connected to its AI. If Claude is hooked up to machinery capable of causing physical injury, a "qualified operator" must be present to monitor the equipment and stop it if necessary.