San Francisco’s Legal Battle Against AI-Generated Non-Consensual Intimate Imagery

Editor’s Note: In a move that could shape the future of digital governance and online safety, San Francisco has launched a lawsuit against 16 websites and applications accused of producing AI-generated non-consensual intimate imagery (NCII). This case, led by City Attorney David Chiu, addresses the disturbing rise of platforms that exploit generative AI to create explicit images without consent, often victimizing women and girls. As AI technology advances, the ethical and legal frameworks surrounding its use are increasingly challenged, making this lawsuit a crucial step in combating digital exploitation. The outcome could set significant legal precedents, influencing global efforts to regulate AI-driven abuses and protect vulnerable individuals from online harm.

Industry News – Artificial Intelligence Beat

San Francisco’s Legal Battle Against AI-Generated Non-Consensual Intimate Imagery

ComplexDiscovery Staff

In recent legal developments, San Francisco has instituted a groundbreaking lawsuit targeting 18 websites and applications responsible for generating unauthorized, AI-created explicit images of women and girls. These platforms employ advanced artificial intelligence to undress or nudify photos uploaded by users, producing highly realistic but non-consensual intimate imagery (NCII). This case, initiated by David Chiu, the elected city attorney of San Francisco, has gained international attention for its potential to set a significant legal precedent. Chiu stated, “The proliferation of these images has exploited a shocking number of women and girls across the globe,” underscoring the widespread nature of the issue.

The lawsuit focuses on websites primarily managed outside the U.S. and in countries such as Estonia, Serbia, and the United Kingdom. These sites have remained accessible by avoiding inclusion in app stores but can be easily found online. The platforms are often designed to lure users by allowing the insertion of victims’ faces onto AI-generated explicit images without their consent. One service claimed their CEO operates within the USA but declined to clarify further, highlighting the clandestine nature of these operations.

The harmful consequences of these images are profound, impacting victims’ mental health, reputations, and autonomy, frequently leading to severe psychological distress and even suicidal tendencies. Chiu remarked, “These images are used to bully, humiliate, and threaten women and girls,” thus conveying the grave repercussions of such digital exploitation. The lawsuit, now filed on behalf of Californians, asserts that these platforms violate multiple state laws, including those against fraudulent business practices and child sexual abuse.

Despite challenges in pinpointing the exact operatives behind these sites, Chiu remains resolute. Leveraging investigative tools and subpoena authority, the city attorney’s office aims to uncover and dismantle these networks. Stanford’s Riana Pfefferkorn emphasized the difficulty of bringing non-U.S. defendants to justice but acknowledged the potential to shutter these sites if domain-name registrars, web hosts, and payment processors comply with the court’s orders.

The issue is not isolated to California. A significant case in Almendralejo, Spain, saw a juvenile court sentencing 15 students to probation for using similar AI tools to create and distribute deepfake nudes of their peers. The incident, which garnered wide attention, highlighted the international scope of this growing problem. Dr. Miriam al Adib Mandiri, whose daughter was among the victims, stressed the responsibility of both society and digital giants in addressing these abuses. “It is not only the responsibility of society, of education, of parents and schools but also the responsibility of the digital giants that profit from all this garbage,” she said.

The European Union, however, has noted that smaller platforms like those used in Almendralejo fall outside the jurisdiction of its new online safety regulations. This underscores the regulatory gaps that exist, allowing such platforms to operate with relative impunity. Organizations like Thorn and The Internet Watch Foundation are actively monitoring these developments, hoping that San Francisco’s legal actions will catalyze broader regulatory reforms.

Victims of AI-generated NCII often face insurmountable challenges in removing these images from the internet, leading to long-lasting psychological, emotional, and economic damage. The FBI and other law enforcement agencies are increasingly overwhelmed with reports of AI-generated child sexual abuse material, further complicating their efforts to address physical abuse cases. Chiu’s lawsuit seeks not only to impose fines of $2,500 per violation but also to force these sites to cease operations entirely, preventing future misconduct.

Emily Slifer, director of policy at Thorn, views the lawsuit as a potential turning point. “The lawsuit has the potential to set legal precedent in this area,” she remarked, signaling its importance in influencing future policies. At the same time, Chiu’s initiative aims to sound a broader alarm about the misuse of generative AI, emphasizing the technology’s capability for both immense benefit and profound harm.

Generative AI tools like those used to create NCII represent a critical challenge in modern digital governance. While they offer substantial benefits in creative and professional fields, their misuse in generating explicit, non-consensual imagery necessitates stringent regulatory and legal actions. As Chiu concluded, “Generative AI has enormous promise, but as with all new technologies, there are unanticipated consequences and criminals seeking to exploit them. We must be clear that this is not innovation. This is sexual abuse.”

News Sources

• San Francisco goes after websites that make AI deepfake nudes of women and girls
• San Francisco files first-of-its-kind lawsuit to tackle AI deepfake nudes
• Boys are using AI to deepfake nude photos – a lawsuit could stop it
• San Francisco Takes on Websites That Allow Deepfake Nudes
• Popular AI “nudify” sites sued amid shocking rise in victims globally

Assisted by GAI and LLM Technologies

Additional Reading

• AI Risks and Ethics: Insights from MIT, Deloitte, and CSA
• eDiscovery Review in Transition: Manual Review, TAR, and the Role of AI

Source: ComplexDiscovery OÜ