Remember that viral image of the Pope in a surprisingly stylish white puffer jacket? Or maybe the video of the queen saying something completely out of character? It’s moments like these that highlight a growing challenge we’re facing with the rise of generative AI and deepfakes. While sometimes humorous, these AI-generated or manipulated media can spread misinformation like wildfire. Now, imagine the stakes if your job relies on discerning the absolute truth from digital media.
This is the reality for intelligence analysts, who sift through vast amounts of information, including images, videos, and audio, to provide critical insights for national security. The increasing sophistication of deepfakes presents an unprecedented challenge to their work. How can we equip them with tools that are not just powerful, but also reliable, understandable, and truly useful in their high-stakes environment?
To answer this, our research group embarked on a Requirements Study, the first phase of a larger work that was published to appear at ACM CHI2025 aimed at understanding and empowering these crucial users. We wanted to dive deep into how analysts perceive deepfakes and what they truly need from detection technology.
Listening to the Experts: Our Methodology
Our primary goal was to hear directly from the practitioners. We conducted semi-structured interviews with 30 professionals from the United States Intelligence Community, including analysts, researchers, and technical support specialists. Their roles involved everything from verifying media provenance and identities to writing detailed intelligence reports.
The interviews explored their current workflows, challenges, familiarity with deepfakes, and expectations for detection tools. To ground the discussion, we used prototype screens based on an existing deepfake detection tool (originally designed for journalists but adapted based on preliminary knowledge of analysts’ needs) to elicit specific feedback on interface design and features. We also explored preferences for different ways of explaining how a tool arrived at its conclusion – a critical aspect known as explainability.
Conducting research with intelligence analysts presents unique challenges. Given that they often work with classified data, interviewing them required careful coordination, sometimes through government proxies or with supervision. All study materials, notes, and transcripts had to undergo a formal review process by a federal body to ensure no sensitive or classified information was inadvertently disclosed and to redact identifying details. Throughout the interviews, we constantly reminded participants to speak only at an unclassified level, and we framed our questions carefully to avoid pushing them toward revealing confidential information. This rigorous process ensured we could gather valuable insights while upholding strict security and ethical standards.
Key Findings: What Analysts Told Us
Our conversations revealed several key themes about the analysts’ needs and frustrations:
- Deepfakes are a Clear and Present Danger:
- Analysts are acutely aware of and concerned about deepfakes, viewing them as an immediate and a long-term threat.
- While definitions varied slightly, they understood the synthetic nature and the role of AI in the creation of deepfake. Some emphasized that the intent behind manipulation is as important as the manipulation itself.
- Video was cited most often as the most concerning modality due to its potential for rapid, widespread dissemination and high believability.
- Professionally, concerns included hindering subject identification and the severe consequences of fake evidence in legal or high-stakes scenarios. The rapid advancement of deepfake tech creates an “arms race” feeling.
“A lie makes it three times around the world and becomes a conspiracy theory before the truth has time to put its shoes on.” – P29
- Fragmented Tools and the Need for Integration:
- A major pain point is the fragmented nature of current toolsets. Analysts often have to juggle multiple, disconnected tools, making it hard to consolidate findings and maintain workflow efficiency.
- There’s a strong desire for a comprehensive, integrated “one-stop-shop” that handles diverse media types (images, video, audio, text).
- This ideal tool should incorporate traditional forensic methods (like metadata viewers, hex editors) alongside modern deepfake analytics. Features like batch processing and potentially media comparison databases (though data compartmentalization is a challenge) were requested.
- Standardization is key. Consistent processes and terminology (aligned with standards like ICD 203) are needed for clear communication and reporting.
- Built-in report generation capabilities, allowing easy export of results and explanations, were highly sought after to streamline this crucial output of their work.
- Explainability is Non-Negotiable:
- Simply getting a “fake” or “real” verdict isn’t enough. Analysts need to understand why the tool reached its conclusion to trust it, justify their findings, and use the information effectively in reports.
- Current explainability methods, like heatmaps often used in commercial tools, were frequently deemed insufficient, too technical, or difficult to interpret reliably.
- Clear, intuitive textual explanations were strongly preferred, as they are easier to understand and integrate directly into reports.
- Analysts also desire features that aid understanding, like tooltips explaining specific functions or scores, and dedicated training sessions.
- When prioritizing features, accuracy (especially minimizing false positives) and explainability are generally ranked higher than raw speed, although the ideal balance can depend on the specific task (e.g., real-time vs. offline analysis). Good explainability can even help mitigate concerns about occasional inaccuracies.
The Way Forward: Building Better Tools
This Requirements Study provided invaluable insights into the world of intelligence analysts grappling with deepfakes. It’s clear that while technology is advancing, the tools available often fall short of meeting their specific, complex needs. The frustrations with fragmented systems and the critical demand for trustworthy explainability underscore the gap between current offerings and what’s truly required.
Understanding these requirements is the crucial first step. Our findings highlight the need for a shift towards user-centered design, focusing on integration, usability, and transparent, understandable results.
But understanding the problems is only half the battle. How can we translate these needs into tangible solutions? Our next steps involved proposing and evaluating potential answers, including a redesigned user interface tailored for analysts and an innovative Digital Media Forensics Ontology designed to structure the complex world of detection analytics and enhance explainability from the ground up.
Stay tuned for our next blog post, where we’ll delve into these proposed solutions and share findings from the second phase of our research!