San Francisco’s city attorney has ordered Apple and Google to remove 13 apps from their platforms that allow users to create AI-generated nonconsensual nude images. The legal notices, sent Thursday by City Attorney David Chiu, demand that the companies stop ‘aiding and abetting’ the creation of explicit deepfake images and cut ties with the developers behind these tools.
As reported by Wired, the cease-and-desist letters target eight apps on Apple’s App Store and five on Google’s Play Store. These apps market themselves as ‘face-swapping’ tools, but their ability to generate sexual deepfakes becomes apparent once users engage with them. Chiu’s office says Apple and Google have likely made millions of dollars in fees from these apps through in-app purchases, a cut of which goes directly to both companies.
‘Generating non-consensual intimate images is illegal, harmful, and completely unacceptable,’ Chiu told Wired. ‘These images are used to bully, humiliate, and threaten women and girls. There have been victims who’ve been suicidal.’
Google confirmed it has already removed the five Android apps flagged in the letters. A spokesperson said the company has deleted ‘hundreds’ of apps with nudifying features for policy violations and has restricted related search terms like ‘nudify’ on its platform. Apple did not comment ahead of publication.
The problem is not new, and neither is the pressure on these platforms to act. Over the past year, the Tech Transparency Project, an independent watchdog group, identified around 100 apps across both stores capable of generating nonconsensual nude images. Those apps had been collectively downloaded roughly 480 million times and may have generated around $120 million in combined revenue. ‘Apple and Google make a lot of promises in their marketing about how trusted and safe their app stores are,’ said TTP director Katie Paul. ‘And that is just not what is playing out in reality.’
A preprint research paper published in May by Cornell and Georgetown University researchers adds more weight to the concern. They identified 420 apps offering general face-swap features across both stores, then tested 155 of them. In 70 percent of cases, the apps could be used to create face swaps with nude images, with no safety measures in place to stop it. The researchers noted that none of these apps explicitly advertised themselves as nudification tools, which helps them slip through content moderation. They called this a ‘dual-use’ problem: apps that appear harmless on the surface but can easily be used to create harmful content.
This is part of a much bigger and worsening trend. Over the last five years, a wave of deepfake nudification tools, including apps, websites, and bots, has spread across the internet. Earlier this year, xAI’s Grok was used to generate millions of sexualized AI images. Creating this content often requires nothing more than a single reference photo and a few clicks, with some results delivered in seconds. Previous reporting by Wired and Indicator Media found deepfake sexual abuse images of minors had been created in at least 90 schools.
Both Apple and Google have developer policies that ban pornography, abuse, and harassment. They have removed nudify apps before in response to reports from researchers and journalists. But critics argue these removals are reactive rather than systematic, and that the scale of the problem demands much stronger screening before apps ever reach users.
Chiu’s office previously took legal action against 16 deepfake websites and says it will keep pursuing this issue. ‘My hope is that Apple and Google will immediately remove these apps and strengthen their screening systems,’ he said. ‘But if they don’t, we will have to consider all of our legal options.’