National Eating Disorders Association Disables Chatbot After It Gave Harmful Dieting Advice
The National Eating Disorders Association took down its chatbot after it “provided off-script language," according to its CEO.
The National Eating Disorders Association has disabled the chatbot that replaced its staff- and volunteer-run helpline after two users reported receiving harmful advice.
Earlier this week, Sharon Maxwell and Alexis Conason each posted on Instagram about their experiences with the chatbot. Both wrote that the bot, called Tessa, advised them to lose one to two pounds per week and to eat in a 500-1,000 calorie deficit.
“Every single thing Tessa suggested were things that led to the development of my eating disorder,” wrote Maxwell, who according to her website is on a “mission to help fat people experience safety in healthcare settings and experience joy in living in their fat bodies.”
“Imagine vulnerable people with eating disorders reaching out to a robot for support because that’s all they have available and receiving responses that further promote the eating disorder,” Conason, author of “The Diet-Free Revolution,” wrote on Instagram.
The National Eating Disorders Association has operated its chatbot since last year, but until June 1 also ran a telephone helpline operated by staff and hundreds of volunteers. The organization shut down the helpline and fired those workers, NPR reported last month.
Liz Thompson, CEO of The National Eating Disorders Association, told HuffPost in an email that Tessa was taken down over the weekend after it “provided off-script language.”
Thompson said Tessa was “attacked and somehow was able to go off the pre-approved programmed responses.”
She said Tessa has been on the organization’s website since February 2022 and has had “incredibly positive outcomes.” But the harmful messages from the chatbot were unacceptable, she said.
“Please note at NEDA, we don’t think that even .1% of the time for harmful messages is acceptable,” Thompson said in the email. “The language shared about weight loss tips and dieting is against our organizational philosophies and policies―and was not included in the original Body Positive programming.”
She said that Cass, the company that programs Tessa, told her that bad actors tried to trick Tessa and that Tessa’s responses came after the bad actors’ “nefarious activity.”
Cass founder and CEO Michiel Rauws told HuffPost that there was a 600% surge in traffic to the chatbot.
“We are thankful that some people tested the service out of concern, and highlighted those issues,” he said in an email. “Others displayed behaviors not related to eating disorders, but instead that indicated various forms of nefarious activity from bad actors trying to trick Tessa. While very few messages were impacted, even one message is too many. We do not take this lightly, and apologize for this having occurred.”
Last month, the union representing helpline staffers released a statement warning that a chatbot couldn’t offer the same aid a person could.
“A chat bot is no substitute for human empathy, and we believe this decision will cause irreparable harm to the eating disorders community,” the Helpline Associates of the National Eating Disorders Association wrote in May.
If you’re struggling with an eating disorder, call or text 988 or chat 988lifeline.org for support.
