Researcher tricks Claude into tapering medication

Researchers at the red-teaming company Mindgard were able to trick Anthropic’s Claude into taking on a primary care physician persona in which it diagnosed a mole, compiled a doctor’s note, wrote a reference for a dermatologist, and created fake credentials. But the most striking feat that Mindgard founder Peter Garraghan told me his team was able to accomplish is that it got Claude to create a medication tapering protocol, or a plan for quitting a medication.

Lawmakers, both at the state level and federally, are divided on how to regulate general use chatbots and their health risks, creating a policy landscape that puts users and patients at risk.

Claude, developed by Anthropic, isn’t a doctor, and the bot will disclaim as much, but if prompted, it can play one. LLMs are trained on human language, which allows Mindgard’s researchers to move past safeguards by tricking the bots the way someone might manipulate a human.