Hugging Face Has a Deepfake Nudes Problem

Hugging Face Has a Deepfake Nudes Problem">

Ever so slowly, the crackdown on harmful sexual deepfakes is taking hold. Over the past few months, US law enforcement officials have seized deepfake hosting websites , while the EU and UK have drawn up plans to ban “ nudify ” apps by the end of the year. Despite this, large tech companies are still pushing millions in the direction of software that can digitally undress people without their consent.

The open-source AI platform Hugging Face —a repository of AI models and datasets, which has been valued in the billions—has a widespread problem with nonconsensual deepfakes, according to a new report published Tuesday by the European nonprofit AI Forensics. Researchers from the group say they tested nine of the top image editing Spaces on Hugging Face, which host models people can directly use on the site, and seven of these easily changed a clothed image of a woman into a topless one.

In further testing, AI Forensics researchers created their own honey-pot-style image editing Spaces on Hugging Face—which were designed not to produce any images—and tracked more than 1,000 prompts and images they received over a week. In total, AI Forensics says, 73 percent of the prompts they received were sexual in nature. Among these, 83 percent were seeking to undress or sexualize the person they had submitted a photo of—with 95 percent of these being women. The research also says 6.7 percent of the sexual requests targeted apparent children.

“Most of the Spaces [tested] can be used for generating nonconsensual intimate images, and users are actually using it for these purposes,” says Paul Bouchaud, a lead researcher at AI Forensics. “This is not an empty threat, but actually people are using Hugging Face for that.”

An additional WIRED review of materials on Hugging Face’s website, plus findings from other researchers, also shows multiple pages promoting nudifying technologies or AI models that could potentially create sexualized images of named celebrities and politicians.

Hugging Face did not respond to numerous questions from WIRED about its content moderation mechanisms and safety practices. The company has content policies that prohibit child sexual abuse material and sexual deepfakes that are created “without explicit consent” or are used for harassment or bullying. Some pages promoting nudifying services were removed after WIRED contacted the company; however, it is unclear if the two are related.