Within the fascinating new actuality of the web, teen women can’t study intervals on Reddit and indie artists can’t promote smutty video games on Itch.io, however a navy contractor will make you nonconsensual deepfakes of Taylor Swift taking her high off for $30 a month.
Early Tuesday, Elon Musk’s xAI launched a brand new picture and video generator known as Grok Think about with a “spicy” mode whose output ranges from suggestive gestures to nudity. As a result of Grok Think about additionally has no perceptible guardrails in opposition to creating photographs of actual individuals, meaning you possibly can essentially generate softcore pornography of anybody who’s well-known sufficient for Grok to recreate (though, pragmatically, it appears to mainly produce seriously NSFW output for women). Musk bragged that greater than 34 million photographs had been generated inside a day of launching operations. However the actual coup is demonstrating that xAI can ignore stress to maintain grownup content material off its providers whereas serving to customers create one thing that’s extensively reviled, because of authorized gaps and political leverage that no different firm has.
xAI’s video characteristic — which debuted across the similar time as a romantic chatbot companion named Valentine — appears from one angle strikingly bizarre, as a result of it’s being launched throughout a interval the place intercourse (all the way down to the word itself) is being pushed to the margins of the web. Late final month, the UK began imposing age-gating guidelines that required X and different providers to dam sexual or in any other case “dangerous” content material for customers beneath 18. Across the similar time, an activist group known as Collective Shout successfully pressured Steam and Itch.io to crack down on grownup video games and different media, main Itch.io particularly to mass-delist any NSFW uploads.
Deepfake porn of actual individuals is a type of nonconsensual intimate imagery, which is unlawful to deliberately publish within the US beneath the Take It Down Act, signed by President Donald Trump earlier this yr. In an announcement printed Thursday, the Rape, Abuse & Incest Nationwide Community (RAINN) called Grok’s feature “a part of a rising downside of image-based sexual abuse” and quipped that Grok clearly “didn’t get the memo” concerning the new legislation.
However in accordance with Mary Anne Franks, a professor at George Washington College Legislation College and president of the nonprofit Cyber Civil Rights Initiative (CCRI), there’s “little hazard of Grok dealing with any form of legal responsibility” beneath the Take It Down Act. “The legal provision requires ‘publication,’ which, whereas sadly not outlined within the statute, suggests making content material obtainable to multiple individual,” Franks says. “If Grok solely makes the movies viewable to the one who makes use of the software, that wouldn’t appear to suffice.”
Regulators have didn’t implement legal guidelines in opposition to massive corporations even after they apply
Grok additionally doubtless isn’t required to take away the pictures beneath the Take It Down Act’s takedown provision — regardless of that rule being so worryingly broad that it threatens most social media providers. “I don’t suppose Grok — or no less than this specific Grok software — even qualifies as a ‘lined platform,’ as a result of the definition of lined platform requires that it ‘primarily supplies a discussion board for user-generated content material,’” she says. “AI-generated content material typically includes person inputs, however the precise content material is, because the time period signifies, generated by AI.” The takedown provision can be designed to work by individuals flagging content material, and Grok doesn’t publicly put up the pictures the place different customers can see them — it simply makes them extremely simple to create (and virtually inevitably put up to social media) at a big scale.
Franks and the CCRI called out the limited definition of a “lined platform” as an issue for different causes months in the past. It’s certainly one of a number of methods the Take It Down Act fails to serve individuals impacted by nonconsensual intimate imagery while posing a risk to internet platforms performing in good religion. It may not even cease Grok from posting lewd AI-modified photographs of actual individuals publicly, Franks told Spitfire News in June, partly as a result of there are open questions on whether or not Grok is a “individual” impacted by the legislation.
These sorts of failures are a operating theme in web regulation that’s ostensibly purported to crack down on dangerous or inappropriate content material; the UK’s mandate, as an illustration, has made it harder to run independent forums whereas nonetheless being fairly easy for kids to get around.
Compounding this downside, notably within the US, regulatory companies have didn’t impose significant penalties for all types of rulebreaking by highly effective corporations, together with Musk’s many companies. Trump has given Musk-owned companies an almost total pass for unhealthy conduct, and even after formally leaving his highly effective place on the Division of Authorities Effectivity, Musk doubtless maintains great leverage over regulatory companies just like the FTC. (xAI just got a contract of as much as $200 million with the Division of Protection.) So even when xAI had been violating the Take It Down Act, it most likely wouldn’t face investigation.
Past the federal government, there are layers of gatekeepers that dictate what is suitable on platforms, they usually typically take a dim view of intercourse. Apple, as an illustration, has pushed Discord, Reddit, Tumblr, and different platforms to censor NSFW materials with various ranges of success. Steam and Itch.io reevaluated grownup content material beneath menace of shedding relationships with fee processors and banks, which have beforehand put the screws on platforms like OnlyFans and Pornhub.
In some instances, like Pornhub’s, this stress is the results of platforms permitting unambiguously dangerous and unlawful uploads. However Apple and fee processors don’t seem to keep up hard-line, evenly enforced insurance policies. Their enforcement appears to rely considerably on public stress balanced in opposition to how a lot energy the goal has, and regardless of his falling out with Trump, nearly no one in enterprise has extra political energy than Musk. Apple and Musk have repeatedly clashed over Apple’s insurance policies, and Apple has principally held agency on issues like its fee structure, however it’s apparently backed down on smaller issues, together with returning its advertisements to X after pulling them from the Nazi-infested platform.
Apple has banned smaller apps for making AI-generated nudes of real people. Will it exert that form of stress on Grok, whose video service launched completely on iOS? Apple didn’t reply to a request for remark, however don’t maintain your breath.
Grok’s new characteristic is dangerous for individuals who can now simply have nonconsensual nudes fabricated from them on a serious AI service, however it additionally demonstrates how hole the promise of a “safer” web is proving. Small-time platforms face stress to take away consensually recorded or solely fictional media made by human beings, whereas an organization run by a billionaire can generate profits off one thing that’s in some circumstances outright unlawful. In case you’re on-line in 2025, nothing is about intercourse, together with intercourse — which, per normal, is about energy.