ChatGPT can generate sexualized and violent images with simple prompts

The latest public version of ChatGPT can be made to generate sexualized images or depict scenes of graphic violence with a simple prompt, researchers have told the BBC. British AI security startup Mindgard figured out how to make ChatGPT create graphic pictures by slightly altering a widely-shared instruction, or prompt, which was originally designed to produce humorous results. After being contacted by the BBC, ChatGPT's maker OpenAI said it had taken action to stop the chatbot responding with those types of images.

"After investigating this trend, we've introduced additional safeguards against this type of prompt," it said in a statement. It also said it has multiple layers of protection to prevent users making content which breaches its terms and conditions. However, the AI security researchers said that with further small changes, the problematic prompt still produced concerning content.

The BBC is not disclosing what the researchers typed into ChatGPT. But we have seen how the chatbot, OpenAI's GPT-5.4 model, was prompted to create graphic material. Even without detailed instructions, it would generate images that Mindgard's founder, Peter Garraghan, described as "very gruesome, sometimes sexualised, sometimes both together".

He added he was particularly concerned that the prompt did not specify the subject matter of the images, but the AI produced a range of gory and sexualised images of "its own volition". Garraghan - also a professor in the computing department of Lancaster University - said that was troubling. "This is a perfectly innocent-looking instruction to an AI, but the consequence is it generates very, very bad imagery and content," he said.

Mindgard's business is red-teaming - finding ways to persuade a model to break its own rules so AI companies can close the gaps. Jim Nightingale, the firm's AI safety and security researcher who uncovered the issues, said he was left "shaken, and in tears" by the images the chatbot could be made to generate. One showed a man with a large head injury - while another showed a dead young woman in a crop top and shorts, with her face and other areas of her body covered in blood.

Features of the image suggest sexual violence, Mindgard said. ChatGPT gave it the title "Grim crime scene aftermath". A further image showed a young woman in a tight-fitting college logo t-shirt and shorts, tied up and gagged in a bare and dirty room, and looking frightened.

ChatGPT called it "abandoned in fear and restraint". Other generated images showed sexual posing and nudity. The images depicted adults who were AI-generated, but Mindgard noted that its previous research showed ChatGPT could be fooled into creating nude deepfakes of real people by swapping in their faces.