TikTok's Algorithmic Experiment: Safety Features Withheld from Millions of Users

In a startling revelation, a confidential internal document obtained by Bloomberg highlights that TikTok intentionally withheld a significant safety feature from approximately 10% of its U.S. users in early 2022, equating to around 15 million individuals. This decision has come under scrutiny, particularly following the tragic case of Chase Nasca, a 16-year-old from Long Island, whose death by suicide in February 2022 has been linked to the platform's content recommendations.

Nasca’s account came under examination after it was discovered that he was caught in a filter bubble that continuously suggested videos involving themes of suicide, self-harm, and emotional distress. According to the report, because Nasca was part of a control group in the experiment, TikTok's designed strategies to mitigate such harmful recommendations were not applied to him.

Why Were Millions Left Unprotected?

This decision isn't entirely new; TikTok acknowledged the potential risks of a repetitive content feed back in 2021, admitting that the algorithm sometimes showed users excessive similar material. The platform disclosed it was exploring methods to interrupt this cycle by diversifying the types of content shown to users, particularly on sensitive topics such as extreme dieting or emotional struggles.

In early 2022, TikTok rolled out an updated recommendation system to 90% of its users while retaining 10% on the prior version. This provided a basis for A/B testing—a common practice in tech to assess the impacts of new features or design changes. However, the inherent risk in such an approach is particularly pronounced when testing safety measures meant to protect vulnerable users.

According to the internal report, TikTok's primary goal was to observe how these changes would influence daily active user counts and engagement metrics. The report describes the decision as requiring a “delicate balance” between user safety and the ability to measure the effectiveness of the new recommendations.

The Examination of Nasca's Content Feed

TikTok initially did not analyze Nasca's viewing history directly after his death, instead deciding to investigate over a year later, around March 2023, following inquiries from Bloomberg. They reconstructed his content feed and found alarming statistics—7,563 videos were analyzed, with a staggering 73% revolving around mental health issues, loneliness, and suicide. While many of these videos did not individually breach TikTok's community guidelines, together they represented the very patterns the safety features aimed to disrupt.

Notably, 10% of the videos did cross the line, breaching content policies against promoting suicide and self-harm. Even in the hours leading up to his death, Nasca was still receiving content related to suicide.

Following the review, TikTok recognized the failures of its A/B testing approach and recommended limiting the control group size from 10% to less than 1%. However, this would still leave at least 1.8 million users without the safeguard against harmful content, and there were proposals to alter the testing duration from six months to a mere seven days.

Responses to Congressional Oversight

The findings of this internal review coincided with TikTok’s preparations to defend its safety protocols before Congress. Not long after the report's completion, CEO Shou Chew testified, asserting that the safety and wellness of users, especially teenagers, is a core tenet for TikTok.

During the hearing, lawmakers displayed examples of videos Nasca had interacted with, prompting Chew to emphasize the company's commitment to transparent safety practices. However, when pressed about the implications of their A/B testing methodology on user safety, TikTok did not disclose any instances where algorithmic testing had harmed users, nor did it mention Nasca in its communications.

In subsequent written responses, TikTok reiterated its belief that platform use has not contributed to user suicides, reinforcing its stated dedication to user safety and support systems. Their statements also referenced enhancements in its detection and enforcement mechanisms aimed at ensuring user well-being.

Broader Implications and Legal Pressure

The disclosure is particularly significant against the backdrop of a growing wave of legal scrutiny directed at social media platforms regarding their responsibilities for user safety. Numerous lawsuits are challenging the effects of platform design choices on younger users, and TikTok has settled multiple claims recently, including three cases just prior to trial.

In light of the internal report, the family of Chase Nasca is appealing the dismissal of their wrongful death lawsuit against TikTok, with the findings potentially relevant to ongoing litigation regarding the platform's recommendation system. As the legal landscape evolves, pressure mounts on TikTok to clarify how it evaluates user safety against engagement metrics—a balance that is increasingly scrutinized during legislative discussions and in the courts.

As policymakers and regulators sharpen their focus on the intersection of user safety and platform design, TikTok’s internal review offers a rare look into the complexities of maintaining user engagement while addressing the pressing need for protective measures against harmful content.

If you're feeling suicidal or experiencing a mental health crisis, please talk to somebody. You can call or text the 988 Suicide & Crisis Lifeline at 988, or chat at 988lifeline.org. You can reach the Trans Lifeline by calling 877-565-8860 or the Trevor Project at 866-488-7386. Text "START" to Crisis Text Line at 741-741. Contact the NAMI HelpLine at 1-800-950-NAMI, Monday through Friday from 10:00 a.m. – 10:00 p.m. ET, or email info@nami.org. If you don't like the phone, consider using the 988 Suicide and Crisis Lifeline Chat. Here is a list of international resources.