When “Banned” Isn’t Enough: Enforcement Gaps in AI-Generated Abuse
After Grok controversy that happened in January, multiple countries started to investigate the situation. While governments are focusing on Grok, it is not the only available source to create explicit images by attackers.
Researchers found apps that are available to everyone on both App Store and Google Play, while another danger is actively going on Telegram.
1. Updates on Grok Case
The European Commission, the UK, Canada, France and many more countries are currently investigating xAI’s Grok due to creation of non-consensual explicit images, involvement of minors and digital safety.
Although xAI claims that the creation of such images has been blocked and is against its guidelines, a CBS News investigation published on January 26 found that, despite the ban, reporters were still able to generate “bikini-fied” images using both X and the Grok app.
According to an EU spokesperson, the current investigation focuses on Grok’s integration into X rather than the standalone application. While CBS reporters were able to generate such images on both platforms, the Digital Services Act (DSA) applies to designated online platforms, not individual applications.
As a result, the investigation is examining whether X’s recommender systems, including features powered or influenced by Grok, recommended, or failed to adequately limit the spread of content involving non-consensual explicit images.
The platform faces scrutiny under the EU Digital Services Act (DSA), which allows fines of up to 6% of a company’s global annual turnover, as well as an investigation under the UK Online Safety Act. Potential penalties under the UK law include £18 million or 10% of the company’s qualifying worldwide revenue, whichever is greater.
Grok is not the only platform where such images are created or shared. The Tech Transparency Project investigated apps across the application stores and identified 55 apps on Google Play, as well as 47 on the App Store that enable the creation of non-consensual explicit images.
Telegram is another platform where this content is both generated and distributed.
2. Over 150 Active Channels on Telegram
Although explicit images are prohibited on Telegram, The Guardian identified at least 150 active channels where non-consensual explicit images of celebrities, influencers, and ordinary women are being shared.
These channels attract users worldwide, spanning countries such as India, Brazil, the UK, and Russia. Some users sell these images for a fee, allowing customers to upload media of any woman and receive a manipulated video depicting explicit acts. Others focus on generating content involving celebrities and influencers. Many of these channels also exchange tips on deepfake tools.
Telegram stated to The Guardian that such content violates its policies and is removed whenever detected. In 2025 alone, the platform reported removing 952,000 pieces of flagged content related to non-consensual explicit imagery.
3. Lack of Regulations and Gray Areas
While the Grok case is getting attention, it’s only part of a much bigger issue. Non-consensual explicit images are not only generated on illegal websites, they’re also accessible through trusted channels such as app stores and well-known social media platforms. While platform restrictions are important, they aren’t enough on their own.
Recent incidents show that simple content removal or account bans don’t stop offenders, who often return by creating new accounts. Stronger regulation is needed to prevent the abuse at its source, rather than relying on temporary, reactive fixes.
Because this form of abuse is still relatively new, many countries are operating in legal gray areas with incomplete or outdated legislation. While lawmakers are taking action, especially after the Grok situation, policy is not moving at the same pace as generative AI. And removal after the fact is rarely sufficient as by the time content is taken down, victims may already have experienced distress, fear, harassment, and serious reputational harm.
The Grok case also highlighted how many gaps malicious actors can exploit and why “it’s banned on our platform” is not a sufficient defense if it doesn’t work in practice. To close these loopholes, governments and platforms need to strengthen the full chain of accountability with clearer laws that explicitly cover AI-generated abuse, meaningful penalties and enforcement, stronger app store review and takedown standards, and faster detection on distribution channels. Without coordinated action across creation, access, and sharing, offenders will simply keep moving to the next easiest place to operate.
How Cyberette supports organizations
Cyberette supports social media monitoring teams by detecting deepfakes and gives forensic-grade analysis they can act on.
Have questions? Reach out at info@cyberette.ai