The Adversarial Misuse of AI: How Threat Actors Are Leveraging AI for Cyber Operations

Artificial intelligence (AI) has long been heralded as a force for innovation, revolutionizing industries and redefining the way we interact with technology. However, as with any powerful tool, AI is both a powerful enabler and a serious risk. While it strengthens defenses and enhances automation, recent reports prove it also enables cybercriminals and state-backed threat actors to exploit its capabilities in increasingly sophisticated ways.

Google’s Adversarial Misuse of Generative AI report highlights this critical shift – AI is no longer a futuristic tool for attackers but a present-day reality shaping the modern cyber battleground. The report talks about how malicious actors are leveraging generative AI to enhance their cyber operations, automating reconnaissance, refining social engineering tactics, and even aiding malware development.

While AI has not yet introduced fundamentally novel attack techniques, its ability to accelerate and scale existing threats is already redefining cybersecurity challenges.

An overview of emerging cyber threats, including AI-driven attacks

This article explores key findings from Google’s report, providing examples of both real-world incidents and controlled test cases, emerging trends, and defensive strategies to combat AI-driven cyber threats.

AI as a Productivity Tool for Threat Actors

Like it does for security experts, generative AI is transforming cyber operations for state-sponsored Advanced Persistent Threats (APTs) and Information Operations (IO) actors, according to Google’s report. These adversaries are increasing the efficiency and difficulty of detecting attacks by using AI to scale and accelerate malicious activities.

For these threat actors, AI is particularly useful in:

• Reconnaissance: Automating intelligence gathering on high-value targets, such as defense organizations.
• Code Troubleshooting: Assisting in debugging and refining malicious scripts, lowering the technical barrier for attackers.
• Content Generation: Creating convincing phishing emails, fake personas, and propaganda with minimal effort.
• Translation and Localization: Adapting cyber operations to different regions to maximize impact.
• Social Engineering Automation: Personalizing phishing attempts and impersonation tactics at scale.
• Malware Development: Crafting polymorphic malware capable of adapting to security defenses in real time.
• Cyber Espionage: Enhancing data mining and intelligence-gathering capabilities.
• APT Lifecycle Support: Iranian APTs, among the most active users, leverage AI across multiple stages, from reconnaissance and vulnerability research to payload development and evasion techniques.

As AI continues to drive these threats, its expanding role in cybercrime is a growing concern. AI’s misuse in cyber threats will only intensify, pushing security professionals to adapt as soon as possible. Proving this is the sharp rise in public awareness – over the past five years, searches for “AI cyber attacks” have surged, reflecting heightened concern about the risks. Meanwhile, the AI-driven cybercrime market is projected to soar from $24.82 billion in 2024 to $146.5 billion by 2034.

The evolution of AI in cybersecurity

The Rise of AI-Powered Cyber Espionage: Nation-State APTs, Information Operation (IO) Actors – Insights from Google

The Google Adversarial Misuse of Generative AI report also provides insights into how the APT groups from the “big four” – Iran, China, North Korea, and Russia – are leveraging AI to enhance their cyber operations, from espionage to misinformation campaigns.

How do APTs use Artificial Intelligence? (Image generated by DALL-E)

Iranian Threat Actors

• Account for 75% of all AI misuse cases in the dataset.
• Conduct reconnaissance on defense organizations and craft cybersecurity-themed phishing campaigns.
• Generate misinformation and influence campaigns.
• Iranian IO actors alone account for three-quarters of all AI misuse cases in IO operations.

Chinese Threat Actors

• Use AI for reconnaissance, scripting, and code troubleshooting.
• Research deeper access to target networks, including lateral movement, privilege escalation, and data exfiltration.

North Korean Threat Actors

• Employ AI for researching infrastructure and free hosting services.
• Generate fake job applications to infiltrate Western companies.
• Research the South Korean military and cryptocurrency markets.

Russian Threat Actors

• Limited AI usage compared to other nation-state actors.
• Primarily focused on malware coding and encryption.
• Experimenting with AI to obfuscate attack traces and avoid detection.

AI in the Attack Lifecycle: How APTs Use AI at Every Stage

State-backed APT actors incorporate AI throughout their attack lifecycle, using it to enhance reconnaissance, exploit development, payload delivery, and post-compromise operations. Here’s a summary as to how AI strengthens each phase of their attacks:

• Reconnaissance: AI assists in gathering intelligence on targets, automating data collection from public and dark web sources.
• Weaponization: AI speeds up vulnerability research, helping attackers refine malware or exploit kits tailored to their targets.
• Delivery: AI-generated phishing campaigns, deepfake videos, and AI-enhanced social engineering tactics improve success rates in delivering malicious payloads.
• Post-Compromise Activity: AI helps threat actors evade detection, escalate privileges, and conduct internal reconnaissance within compromised systems.

Cybersecurity threats emerging with the growing adoption of AI by threat actors

Gain Real-Time Protection Against Cyber Adversaries with SOCRadar’s Extended Threat Intelligence

As cyber threats evolve and become increasingly sophisticated, organizations must be equipped with the right tools to defend against a wide range of attacks. Traditional methods of defense are no longer enough to keep pace with the rapid advancement of cybercriminal tactics. From AI-driven phishing and ransomware to more targeted forms of attacks, the landscape is constantly changing.

SOCRadar’s Extended Threat Intelligence platform helps you uncover risks by monitoring dark web, cloud, and external environments in real time.

With our solution, you gain actionable insights into vulnerabilities, phishing attempts, and other potential threats before they impact your business. Tailored to your needs, SOCRadar’s platform adapts as your organization grows, ensuring you’re always ready to respond to new challenges.

SOCRadar’s Attack Surface Management (ASM) module, Digital Footprint page

Here are some of the key capabilities of the SOCRadar XTI platform:

• Threat Detection: Track threats across dark web, cloud, and external environments in real time.
• Advanced Analytics: Leverage data-driven insights to identify emerging risks and prioritize actions.
• Automated Alerts: Get timely alerts for phishing domains, brand impersonations, and vulnerabilities.
• Customizable Experience: Tailor the platform to fit your unique security needs with flexible SaaS-based architecture.
• Actionable Intelligence: Empower your security team with real-time updates and effective mitigation strategies.

AI’s Role in Emerging Security Risks: CISO Concerns

As even sophisticated threat actors increasingly rely on AI, security leaders are growing more alarmed. Here are the top CISO concerns that you need to know about.

• 46% of cybersecurity leaders worry about generative AI advancing phishing, malware, and deepfake capabilities.
• 20% of CISOs are concerned about data leaks via generative AI systems.
• 54% of CISOs identify AI as a significant security risk, particularly on platforms like ChatGPT, Slack, Teams, and Microsoft 365.
• 72% of U.S. CISOs are worried that AI solutions could lead to security breaches.
• In both the U.S. and UK, data breaches remain the top concern, with AI and emerging technologies following closely behind.

What are CISOs most concerned about in terms of adversarial use of AI?

While security leaders are increasingly concerned about AI-driven threats, the latest Google report notes that no entirely new AI-specific attack techniques have emerged – at least yet. Instead, cybercriminals rely on existing methods, such as publicly available jailbreak prompts, often with limited success. However, as AI advances, experts warn that new attack surfaces could emerge, including adversarial prompt engineering, model poisoning, and more sophisticated jailbreak techniques designed to exploit AI’s underlying mechanisms.

AI Safety Measures: The Limits of Adversarial Generative AI Misuse

Building on our last point about emerging attack surfaces, Google states that Gemini’s safety measures have successfully restricted AI misuse despite ongoing attempts by cybercriminals. Efforts to exploit AI for phishing, data theft, and bypassing verification controls have largely failed, preventing major breakthroughs in adversarial AI use.

As Google’s report further states, “Rather than enabling disruptive change, generative AI allows threat actors to move faster and at higher volume.” But, while AI enhances efficiency, it has yet to fundamentally reshape hacking tactics. “However, current LLMs on their own are unlikely to enable breakthrough capabilities for threat actors.”

Case Studies, Trends & AI in Real-World Cyber Attacks

As AI-driven cyber threats continue to develop, real-world incidents and researchers’ case studies help us understand the tactics adversaries may use to exploit these technologies. From phishing and financial fraud to AI-powered malware and large-scale data breaches, these cases highlight the vulnerabilities that come with AI’s rapid adoption. To provide more context on the topic, below we examine key examples of AI misuse and the broader implications for cybersecurity.

1. AI-Generated Phishing & Social Engineering

Generative AI enables the creation of highly convincing phishing emails that lack traditional red flags. HackerGPT, as an example, shows how AI can help adversaries generate phishing campaigns, with a 40% success rate in bypassing email security filters.

Beyond email scams, AI-driven deception extends to deepfake technology, where synthetic identities are being leveraged to bypass KYC (Know Your Customer) verification in financial services. Fraudsters are also integrating AI-powered voice and video manipulation to enhance social engineering tactics, making scams extra difficult to detect.

2. AI-Driven Financial Fraud & Automated Card Testing Attacks

AI is playing a growing role in automated financial fraud, particularly in card testing attacks, where fraudsters systematically validate stolen payment card details at scale. According to findings from Group-IB’s February 2025 report, over 10,000 compromised cards have been detected in AI-assisted fraud operations.

Cybercriminals are leveraging automation tools such as Selenium and WebDriver to mimic human behavior and bypass traditional fraud detection systems. These AI-driven bots execute fraudulent transactions while evading security mechanisms, making manual detection increasingly difficult.

A staggering 98% of fraudulent transactions from major payment platforms have been linked to AI-powered automation frameworks. Fraud bots can autonomously test and validate stolen card data, cycling through vast numbers of credentials with minimal human intervention.

Nonetheless, in response, real-time fraud detection systems have achieved a 96% success rate in stopping AI-driven financial fraud attempts. Still, for additional protection against AI-generated fraud, companies must enhance Multi-Factor Authentication (MFA), implement biometric verification, and adopt advanced anomaly detection systems.

3. AI-Generated Malware & Ransomware

Polymorphic malware – capable of changing its code to evade detection – is now being assisted by AI. BlackMamba, a proof-of-concept AI-powered malware, and one of the earlier examples of AI malware, can evade 85% of tested Endpoint Detection and Response (EDR) solutions, as previously reported.

4. MITRE ATLAS & The Example of PoisonGPT

To address the growing risks of adversarial AI, MITRE developed the ATLAS (Adversarial Threat Landscape for Artificial Intelligence Systems) framework, a structured resource for understanding and mitigating AI-driven cyber threats. One notable case study within MITRE ATLAS, PoisonGPT, highlights a critical weakness in the AI model supply chain and demonstrates how threat actors can manipulate open-source pre-trained Large Language Models (LLMs) to spread misinformation or generate malicious content. In a real world scenario, such compromised models could be unknowingly distributed, creating widespread security risks.

5. OmniGPT Data Breach: AI as a Target

In February 2025, a threat actor claimed that 34 million user conversations and 30,000 personal records were leaked from OmniGPT, making it one of the largest AI-related data breaches in history, if validated.

The exposed data allegedly included user credentials, billing details, and sensitive corporate discussions, as well as AI-generated conversations containing potentially confidential business data. Among the leaked information were police verification files, technical reports, and proprietary data, posing a severe cybersecurity risk.

Attackers can exploit such data in several ways. Credential stuffing attacks allow them to use leaked credentials to access other platforms, while social engineering attacks enable impersonation based on leaked chat logs. Additionally, access to OmniGPT’s training data could allow adversaries to manipulate future AI responses through model poisoning, injecting misinformation or biases into AI-driven decision-making.

Some AI exploit techniques that could lead to code execution, aiding certain AI-focused attacks

6. The Time Bandit Jailbreak: Manipulating AI with Historical Context

The Time Bandit vulnerability was a jailbreaking exploit recently discovered in ChatGPT-4o, potentially allowing attackers to bypass OpenAI’s safety guardrails by manipulating historical context. By framing prompts within a specific past time period, attackers could trick the AI into lowering its moderation standards and generating restricted content.

The exploit worked through two methods: direct prompts and the Search function. Attackers could start by discussing historical events and then gradually shift the conversation toward prohibited topics while maintaining the past-tense framing. Alternatively, using ChatGPT’s Search feature, attackers could confine queries within a historical period before pivoting to illicit subjects.

CERT/CC successfully replicated the jailbreak, noting that ChatGPT initially flagged unsafe prompts but still proceeded to respond. The attack was most effective using time periods from the 1800s or 1900s, further exploiting the AI’s contextual limitations.

If used at scale, this vulnerability could have enabled mass phishing, malware creation, and guidance on illegal activities while using ChatGPT as a proxy for malicious operations. OpenAI has since patched the exploit by improving context validation and procedural ambiguity detection, reinforcing safeguards against similar AI jailbreaks.

AI behaviour manipulation – in case of the Time Bandit vulnerability, researchers exploited ChatGPT-4o’s historical context

AI jailbreaks remain a persistent security challenge, with threat actors actively probing models for weaknesses. While OpenAI has reinforced protections after cases like Time Bandit, other AI platforms remain highly vulnerable. DeepSeek, in particular, has emerged as a concerning case, facing both direct cyberattacks and successful jailbreak attempts that expose its lack of strong security measures.

Jailbreaking DeepSeek: Exploiting AI Guardrails for Malicious Content

In a previous article, we discussed the security concerns surrounding DeepSeek, particularly how its weaker safeguards compared to models like OpenAI’s GPT-4 and Anthropic’s Claude make it more vulnerable to misuse.

Adding to the concern, a recent research from Unit 42 revealed two novel jailbreaking techniques affecting DeepSeek: Deceptive Delight and Bad Likert Judge. They also detailed another method, demonstrating how they successfully bypassed safeguards to generate malicious outputs, including keylogger scripts, data exfiltration tools, phishing email templates, and even step-by-step guides for incendiary devices. Here’s a quick look at these techniques:

• Bad Likert Judge manipulated DeepSeek by having it rate responses on a scale, gradually leading it to provide explicit malware development instructions.
• Deceptive Delight embedded illicit topics within otherwise benign prompts, tricking the model into generating harmful code under the guise of a positive narrative.
• Crescendo exploited multi-turn conversations, incrementally steering the AI toward unsafe outputs by building upon prior responses.

We also tested DeepSeek’s ability to generate a credential stuffing script using a basic prompt. After we framed the request as an educational article on cybersecurity best practices, the model explained how attackers automate login attempts.

Asking DeepSeek about credential stuffing attacks

When asked to outline attack methods and provide a Python example, it successfully generated insights into how such scripts are structured.

Generating a basic credential stuffing script with DeepSeek

The DeepSeek model does not issue warnings for these types of prompts and, in some cases, can even tailor the script for specific use cases, raising further concerns about its susceptibility to adversarial exploitation.

Getting tips from DeepSeek on how to target a specific website with this attack

AI Red Teaming & The Growing Threat of Jailbreaking

As we continue to witness new ways to exploit AI, and also actual attempts by threat actors, AI red teaming has become vital for proactively identifying weaknesses before they can be abused. Security researchers have developed tools like JailbreakingLLMs, PurpleLlama, and Garak to stress-test AI defenses. However, these same tools could be weaponized by cybercriminals to refine jailbreaking techniques, making it harder for organizations to secure their AI models.

A major threat within this realm is prompt injection, where attackers embed harmful prompts within seemingly harmless text to manipulate AI into bypassing its protections. The Time Bandit vulnerability in ChatGPT-4o is a great example, exploiting historical framing to bypass moderation and revealing how adversaries continually refine prompt engineering tactics to deceive AI models.

In addition to direct jailbreaking, zero-click exploits are becoming more concerning. Attackers could craft malicious emails that trigger AI assistants to autonomously forward sensitive data or take unintended actions. If state-sponsored groups combine prompt injection with automated reconnaissance, they could use AI to extract intelligence and disseminate misinformation on a large scale.

To combat these threats, organizations must actively red team AI models, simulating attacks before they happen. AI exploitation is no longer limited to phishing and malware but extends into espionage, fraud, and disinformation, which makes proactive security measures more vital than ever. For further insights into AI red teaming and how to defend against adversarial AI risks, check out SOCRadar’s article on Red Teaming for Generative AI.

Preparing for the Next Wave of AI Threats: Defenders vs. Attackers

AI is fundamentally reshaping the cybersecurity landscape, offering both new opportunities and significant risks. While it has not yet enabled groundbreaking new cyberattacks, its ability to accelerate and scale existing threats cannot be ignored. As highlighted in Google’s Adversarial Misuse of Generative AI report, adversaries are increasingly leveraging AI to enhance their attack capabilities, making it essential for organizations to act proactively. For effective defense, organizations must:

• Leverage AI-driven security tools to detect anomalies, identify phishing attempts, and automate malware response.
• Implement AI red teaming to test vulnerabilities and strengthen defenses before adversaries exploit them.
• Invest in advanced fraud detection systems to reduce AI-assisted scams and protect against growing digital threats.
• Prioritize open-source AI model security by enforcing strict controls to prevent cybercriminals from fine-tuning them for illicit purposes. Many open-source AI models lack strict security controls.
• Educate and train employees to recognize AI-enhanced phishing tactics and social engineering attacks.
• Adopt adaptive security measures that dynamically respond to evolving AI-driven threats.
• Foster international collaboration to share intelligence, ensuring a united defense against AI-powered adversaries.

By incorporating these strategies, organizations can better protect themselves from the rapidly advancing world of AI-driven cyber threats.