In recent years, the abuse of intimate imagery has taken a grim turn with the rise of new technologies, particularly generative AI. On July 30, 2024, Microsoft updated its approach to addressing non-consensual intimate imagery (NCII) and released a policy whitepaper calling for policy modernization to counter AI-generated content abuse. The company's new partnerships and technologies reflect an evolving strategy that blends technical tools and partnerships with advocacy for better legislative frameworks.
Courtney Gregoire, Chief Digital Safety Officer at Microsoft, highlighted that this issue has become more complex as AI deepfake technology makes the creation and distribution of hyper-realistic synthetic intimate images easier. The threat particularly targets women and girls, both in public and private spaces, with real-world consequences that extend beyond digital harassment to significant emotional and reputational damage.
The Growing Problem of AI-Generated Intimate Image Abuse
Intimate image abuse isn’t new. Since 2015, Microsoft has acknowledged the severe consequences of NCII, including emotional distress, reputational damage, and financial extortion. What’s changed, however, is the nature and scale of the abuse. The rise of generative AI has made it possible to create highly realistic “deepfake” images and videos, which can be used maliciously to harass or extort individuals. As AI-generated content becomes more sophisticated, so do the challenges for those tasked with policing this digital abuse.
The proliferation of AI-generated NCII is happening at a rate faster than regulatory or legislative frameworks can evolve. While the technology offers incredible opportunities for creativity and innovation, it has also enabled bad actors to exploit the vulnerability of individuals, particularly women. Microsoft’s update underlines a stark reality: the challenges of controlling intimate image abuse in the digital age have become far more severe.
Microsoft’s Partnership with StopNCII: A Tech-Driven Approach to Protection
As part of its ongoing efforts to combat NCII, Microsoft has partnered with StopNCII, an initiative that allows victims to take greater control over their online presence. StopNCII, a platform run by SWGfL, provides a mechanism for individuals to create digital fingerprints or “hashes” of their images, without the actual images leaving the victim’s device. These hashes are then used to detect and prevent the sharing of intimate images across participating platforms.
This partnership, along with Microsoft's deployment of the PhotoDNA technology, is particularly aimed at preventing NCII from appearing in Bing's search results. According to Gregoire, this victim-centered approach has already led to action being taken on 268,899 images by the end of August 2024.
This initiative is critical not just for curbing NCII abuse but also for protecting individuals from the evolving threat of AI-generated content. Synthetic intimate images created without a person’s consent can be just as harmful as real images, making it essential to expand detection tools like StopNCII to cover synthetic content as well.
A Comprehensive Approach to Online Safety
Microsoft's approach to combating intimate image abuse is holistic. The company enforces a zero-tolerance policy on NCII across its services, from search results in Bing to shared content on its cloud platforms. This includes banning not just the sharing of real or synthetic NCII but also any threats to extort or blackmail individuals with such content. Microsoft is committed to quickly removing abusive content flagged through its various reporting tools, whether it’s identified by users, NGOs, or other partners.
A cornerstone of this strategy is empowering victims with reporting tools. Microsoft’s centralized reporting portal allows individuals to request the removal of NCII shared without consent. Victims can use a simple three-step process to flag inappropriate content, and Microsoft commits to removing such content from Bing search results or hosted services like Xbox and OneDrive.
In addition to these efforts, Microsoft tailors its approach to AI-based content creation, ensuring that its platforms like Microsoft Store and its Generative AI services prohibit the creation of sexually explicit material. The company is also actively involved in efforts to demote low-quality content and boost the visibility of authoritative sources in its search engine.
The Need for Collaborative Action and Legislative Support
As Microsoft notes, the challenge of tackling NCII is a societal issue that requires a “whole-of-society” approach. That’s why the company isn’t just focused on building internal tools. Microsoft is advocating for legislative reforms, particularly in the United States, to modernize laws that protect victims of AI-driven abuse. This advocacy is aligned with President Biden’s Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, which Microsoft commends for its focus on curbing the risks associated with synthetic NCII.
Microsoft’s policy whitepaper released in July underscores its call for clearer legislation around deepfakes and AI-generated content, which currently falls into a legal gray area. Legislative action is necessary to hold perpetrators accountable, deter future abuse, and ensure justice for victims.
Beyond legislation, Microsoft is working with industry leaders and NGOs, including Thorn and All Tech is Human, to introduce new safety-by-design principles in the development of AI tools. By making safety a priority from the outset, Microsoft aims to minimize the potential for harm while also ensuring that bad actors cannot exploit AI for malicious purposes.
The Role of AI in Safety and the Future of Online Protection
The increasing use of AI in online safety is a double-edged sword. On the one hand, AI-driven tools can be incredibly effective at detecting and preventing NCII and other forms of abuse. On the other hand, the same technology is being leveraged to create the very content it’s meant to stop.
Microsoft’s approach acknowledges this complexity. By using technologies like PhotoDNA and AI-driven hash detection, the company is leading the charge in using AI to protect individuals. But as Gregoire pointed out, these efforts are just one part of a larger solution. Ultimately, collaboration between technology companies, governments, and civil society will be essential in tackling the ongoing and evolving threats posed by AI-generated NCII.
DXP Opinion: A Step in the Right Direction, But Is It Enough?
Microsoft's update represents a significant step forward in the fight against NCII, especially in an era where AI-generated content is growing more difficult to manage. Their partnership with StopNCII, along with the use of AI-driven detection tools, is commendable, and the company’s advocacy for legislative changes is crucial.
However, a broader conversation is needed about the ethical use of AI and how companies like Microsoft can balance innovation with responsibility. While tools like PhotoDNA are essential, they’re only part of the solution. What happens when AI becomes even more sophisticated, or when bad actors find ways to circumvent detection systems?
For now, Microsoft is leading by example, but this is a challenge that will require continued vigilance, stronger international collaboration, and proactive regulation. More tech companies need to follow Microsoft’s lead, but it’s also up to governments to ensure that laws evolve quickly enough to keep pace with the rapid changes in technology. At DigitalExperience.Live, we believe that the future of online safety lies not just in technological solutions but in a coordinated global effort to address these issues before they escalate further.
As the digital world continues to evolve, the responsibility of protecting individuals from harm will only grow more urgent. Microsoft's work, while admirable, is just the beginning. The battle against AI-driven exploitation has only just begun.
Read more of our tech-driven news and editorial opinion here.
