The Profitability of Predation: Apple and Google's App Store Moderation Failure

OPINION: For years, Apple and Google have positioned themselves as the benevolent gatekeepers of the mobile ecosystem, promising a curated, safe experience for users. But as the frontier of generative AI expands, these companies are treating their app stores like lawless territories, profiting from predatory tools while offering the bare minimum in moderation.

According to reporting from Wired, San Francisco City Attorney David Chiu recently sent cease-and-desist letters to Apple and Google demanding the immediate removal of 13 "face-swap" apps. These tools, which allow users to create nonconsensual nude images via AI, are overwhelmingly used to target women and girls. Chiu's office argues that by hosting these apps and taking a cut of in-app payments, the tech giants are effectively "aiding and abetting" the sale of explicit deepfake images.

This is not a case of a few bad actors slipping through the cracks; it is a systemic failure of oversight. Wired reports that researchers have repeatedly flagged apps in both the App Store and Play Store that facilitate the generation of sexual images, some of which were even rated as suitable for children. The scale of the problem is staggering. The Tech Transparency Project (TTP), an independent watchdog, uncovered approximately 100 such apps across both platforms in January and April of this year. According to TTP director Katie Paul, the issue persisted—and potentially worsened—even after an initial report was filed. TTP estimates these apps were collectively downloaded around 480 million times and may have generated approximately $120 million in combined revenues.

While Apple and Google maintain developer policies that prohibit harassment, abuse, and pornography, their enforcement is reactive rather than proactive. They remove apps after journalists or researchers sound the alarm, yet the tools continue to reappear. David Chiu told Wired that Apple and Google have likely "made millions of dollars in fees" from these nudification apps, suggesting a financial incentive to maintain a porous moderation system.

Google has attempted to frame this as a victory of its internal processes. Google spokesperson Dan Jackson told Wired that the company has deleted "hundreds" of apps with nudifying features and has restricted search terms like "nudify" on the Play Store. Jackson noted that Google has already removed the five Android apps flagged by Chiu’s office. Apple, however, declined to comment on the matter.

But the human cost of this negligence is devastating. The technology is designed to strip autonomy and destroy lives. Chiu noted that these images are used to bully, humiliate, and threaten victims, leading to severe mental health crises and, in some cases, suicidal ideation. The real-world impact extends to the most vulnerable; Wired and Indicator Media have reported incidents in at least 90 schools where deepfake sexual abuse images were created of minors.

The deceptive nature of these apps is part of the strategy. Many of the 13 apps targeted by the City Attorney’s Office advertise themselves as benign "face-swapping" tools, only revealing their capacity for sexual deepfakes once they are downloaded. One such app, which boasts over 1 million downloads, lists styles such as "cinematic intimacy," "calm busty," and "bikini queen curvy" on its website. Another app explicitly promises "free and uncensored" videos.

California law prohibits supporting services that create deepfake pornography, yet the world's most powerful technology companies continue to facilitate this industry. Whether it is through the App Store or the Play Store, the mechanism is the same: allow the predatory software to proliferate, collect the commission on the transactions, and then play the part of the surprised moderator when a government official sends a legal notice.

Apple and Google cannot claim they are unaware of the harm. When the very tools used to sexually abuse minors and women are available for download with a few clicks, the "moderation" priority is a myth. Until these companies prioritize human safety over the frictionless growth of their app economies, they remain complicit in the digital violence their platforms enable.