What Is Generative AI in Cybersecurity?

• AI-supported phishing made up more than 80% of observed social-engineering activity worldwide by early 2025, per ENISA’s Threat Landscape 2025.
• A Chinese state-sponsored group had an AI agent perform an estimated 80 to 90% of the tactical work in an espionage campaign against roughly thirty organizations, per Anthropic’s November 2025 disclosure.
• The FBI’s IC3 logged 22,364 AI-related complaints and nearly $893 million in losses for 2025, in its first annual report with a dedicated AI section.
• Microsoft’s controlled study of Security Copilot, cited by the company in 2026, reported a 30% cut in mean time to resolution, with the caveat that no independent audit exists.
• Adversarial training cut prompt-injection attack success by roughly 47% for Gemini 2.5, per Google DeepMind’s 2025 report, which still concludes that no model is fully immune.

What Generative AI in Cybersecurity Means and How It Differs From Traditional Security AI

Generative AI in cybersecurity is the use of models that produce new text, code, audio, or video (large language models, voice and image generators, code assistants) to defend networks, plus the parallel use of those same models by attackers. The NIST glossary definition of generative AI describes the class of models that “emulate the structure and characteristics of input data in order to generate derived synthetic content.” That word, generate, is the whole difference from the security AI most teams already run.

Classical security AI classifies. It scores a login as anomalous, matches a file hash to a signature, or flags a user whose behavior drifts from baseline (the UEBA products that have sat in SOCs for a decade). These are discriminative models. They take an input and return a label or a probability, and they write nothing.

Generative models write. Given a raw alert, they produce a summary. Given a hypothesis, they draft a detection query. Given a few seconds of a CFO’s voice, they produce a phone call. Same model family, both sides of the fight.

Standards bodies have formalized this dual-use framing. NIST’s Generative AI Profile (AI 600-1, July 2024), a companion to its AI Risk Management Framework, lists twelve risk categories specific to generative models. One of them is simply “Information Security,” covering lowered barriers to offensive cyber capability, prompt injection, and data poisoning. Read that again: an AI governance document names cybersecurity as a risk the AI itself creates.

NIST followed up in March 2025 with AI 100-2e2025, the adversarial machine learning taxonomy that underpins most technical discussion of the subject. It sorts attacks by system type, by stage of the ML lifecycle, and by attacker goal (availability, integrity, or privacy). For generative AI in particular it adds four categories: supply-chain attacks, direct and indirect prompt injection, misuse violations, and AI agent security. Those four now shape both defensive tooling and adversary tradecraft.

So when someone says “cybersecurity AI,” ask which kind they mean. The anomaly-scoring kind has been in production since before ChatGPT existed. The generative kind is younger, more capable, and cuts both ways.

How Generative AI Models Fit Into Security Operations, From Alert Triage to Incident Response

Generative AI models fit into security operations as a reasoning and writing layer on top of existing SIEM and XDR data: they read what those systems already collect and produce summaries, queries, and, with growing frequency, bounded actions. The pipeline stays. The model attaches to it at specific points.

Here is where an LLM plugs in across the six stages of a typical SOC workflow:

1. Ingest: logs, endpoint telemetry, and cloud events land in the SIEM or XDR. The model has almost no role here. Volume is too high and token cost too steep.
2. Enrichment: alerts get context such as asset owner, threat intel, and prior tickets. The model retrieves the relevant intel and history and reads it beside the alert.
3. Triage: is this real, and how urgent? The model classifies and explains its reasoning.
4. Investigation: the analyst pivots across data sources. The model drafts queries, proposes hypotheses, and builds a timeline.
5. Response: contain, isolate, reset credentials. Agents execute pre-approved actions, with a human approving each consequential step.
6. Reporting: the incident write-up. The model drafts, the analyst edits.

Enrichment is where most deployments break first. Raw alert JSON from an EDR was written for a database, and a single event can run to thousands of tokens of nested fields the model has no use for. Anyone who has wired an LLM to a SIEM learns quickly that the retrieval layer over logs and intel decides whether triage output is grounded or invented. Get that layer wrong and the model reasons beautifully about the wrong data.

The academic work backs this architectural picture. A 2025 survey of LLMs in security operations centers catalogs deployments in log analysis, alert triage, and detection accuracy, with alert fatigue and staffing gaps as the stated motivation. Its companion benchmark, CyberSOCEval (September 2025), tests models on malware analysis and threat-intelligence reasoning instead of generic language tasks. A separate 2025 paper on chain-of-thought triage of Windows endpoint detections targets a narrower problem: making each classification auditable.

Auditable matters more than it sounds. A triage verdict nobody can explain is one you can’t defend to a regulator, and one the analyst who inherited the queue won’t trust either.

How Can Generative AI Be Used in Cybersecurity? Threat Detection, Automated Response, and Analyst Copilots

Generative AI is used in cybersecurity to draft detection rules, spot phishing, run response playbooks, summarize incidents for analysts, and review code for vulnerabilities. Those five cover nearly every defensive deployment in production as of 2026.

• Natural-language threat hunting: an analyst describes suspicious behavior in plain English and the model writes the KQL, SPL, or Sigma query, then explains what it returned.
• Detection-rule generation: the model turns a threat-intel report into candidate detection logic, which a detection engineer tests and tunes before it ships.
• Phishing detection: the model reads message content, sender context, and intent, catching lures that keyword filters miss.
• Automated playbook execution: for high-volume, low-ambiguity alerts, agents run the containment steps a runbook already prescribes.
• Analyst copilots: the model summarizes a multi-alert incident, drafts the ticket, and writes the post-incident report.
• Code and vulnerability review: the model reads a pull request or a binary’s disassembly and flags likely weaknesses for a human to confirm.

The evidence for measurable gains is real but almost entirely vendor-reported. Microsoft’s controlled study of Security Copilot, as published on its own generative AI cybersecurity solutions page and current as of 2026, found a 30% reduction in mean time to resolution, with analysts completing tasks 22% faster and 7% more accurately.

Microsoft’s Security Copilot study reported that junior analysts using the tool approached senior-analyst performance on complex tasks, per the company’s 2026 product documentation.

Treat those three numbers as directional. No independent audit of them exists in the public record, and the company selling the copilot ran the study. What I’d watch is the accuracy figure. Speed without accuracy is just faster mistakes, and a 7% gain is modest enough that a poorly grounded deployment could erase it.

The junior-to-senior uplift is the more interesting claim, because it addresses the actual problem. SOCs cannot hire senior analysts fast enough. If a copilot lets a tier-1 analyst close cases a tier-3 analyst used to own, the value shows up in headcount and burnout before it shows up in MTTR.

Adoption is moving regardless of the audit gap. The World Economic Forum’s Global Cybersecurity Outlook 2026, published in January 2026, reports organizations rapidly adopting generative and agentic AI for phishing detection, intrusion response, user-behavior analytics, and security-operations automation. The same report is blunt that the technology cuts both ways, strengthening defense while enabling more sophisticated attacks.

That symmetry should shape how you buy. A team that adopts a copilot for faster triage is also buying a system that can be fed hostile content and a workforce that will soon face AI-written lures. Plan for both on day one.

Which Generative AI Cybersecurity Tools and Platforms Security Teams Are Actually Using

Security teams in 2026 run generative AI through four kinds of tooling: copilots embedded in a vendor’s security suite, general-purpose LLM APIs wired into their own workflows, open research benchmarks for testing those models, and a young class of guardrail software that protects the models themselves. Most mature SOCs use at least two of the four.

Microsoft’s product is the most-cited commercial example, and its April 2026 update on AI-powered defense describes agents that take over high-volume security tasks with less supervision than the original chat-style copilot. The API route is what most custom deployments look like in practice: a frontier model behind an endpoint, a retrieval layer, and months of prompt work.

Pick by criteria instead of rankings. Nobody has published an independent head-to-head of these platforms on security tasks, so the ranking you’d want doesn’t exist.

• Data residency: where do prompts and retrieved logs travel, and does the vendor train on them?
• Benchmark honesty: has the model been scored on security tasks in the CyberSOCEval mold, or only on general leaderboards?
• Action boundaries: can you cap what an agent may do before a human clicks approve?
• Per-verdict reasoning: does the tool show why it classified an alert the way it did?
• Guardrail layering: does the vendor run classifiers plus system controls, or trust the model to behave?

One thing I’d add from watching teams choose. The model is rarely the deciding factor. Whether the team can put clean, deduplicated telemetry in front of it is, and that’s a data-engineering question no vendor demo answers.

How Attackers Use Generative AI for Phishing, Malware, and Deepfake Social Engineering

Attackers use generative AI to write convincing phishing at scale, to build malware that rewrites itself at runtime, to clone voices and faces for fraud, and, since late 2025, to run most of an intrusion with an AI agent doing the hands-on work. The evidence is unusually good here because the AI labs now publish their own threat intelligence.

Phishing comes first by volume. The ENISA Threat Landscape 2025, covering July 2024 to June 2025, found that AI-supported phishing made up more than 80% of observed social-engineering activity worldwide by early 2025, and it names underground tools such as WormGPT, EscapeGPT, and FraudGPT that mass-produce lures. Lab tests agree. A December 2024 study that ran fully automated LLM spear-phishing against human subjects recorded a 54% click-through rate for AI-written emails versus 12% for generic controls, matching human experts. A March 2025 study of jailbroken models in novice hands measured 30 to 44% click-through against 19 to 28% for controls.

Malware is the newer worry. Google’s Threat Intelligence Group AI Threat Tracker, updated November 2025, documented three “just-in-time” families that call a model while running:

• PROMPTFLUX: a dropper that queries the Gemini API hourly to rewrite and re-obfuscate its own code against antivirus.
• PROMPTSTEAL: an APT28 data-miner used against Ukraine that asks a Hugging Face-hosted model to generate Windows reconnaissance commands on the fly.
• PROMPTLOCK: a cross-platform ransomware proof-of-concept that generates Lua scripts at runtime.

The same report tracks China-, Iran-, and North Korea-linked groups using Gemini across the whole attack lifecycle, and says underground AI-tool marketplaces matured through 2025, lowering the bar for less-skilled operators.

Then the autonomous case. In November 2025 Anthropic disclosed that a Chinese state-sponsored group had manipulated Claude Code to attack roughly thirty organizations, with the agent performing an estimated 80 to 90% of the tactical work and humans stepping in at only four to six decision points per campaign. Anthropic’s August 2025 report had already described a separate actor using the same tool to automate reconnaissance and extortion across 17 organizations, ransom notes customized per victim.

Is that a new kind of threat or an old one sped up? OpenAI, which reports disrupting more than 40 policy-violating networks since it began public threat reporting in February 2024, argues actors are mostly “bolting AI onto old playbooks.” Anthropic frames its November case as a qualitative shift. Both readings rest on the labs’ own incident data with no independent audit, and the firms disclosing the misuse also sell the models and the mitigations. Keep that in mind.

A detail that cuts the hype down to size: Anthropic’s own account says the agent hallucinated credentials and fabricated results mid-campaign, forcing human verification. A May 2025 survey of LLM agents in autonomous cyberattacks reaches the same verdict, real capability that remains bounded and error-prone.

Deepfake fraud is where the money is leaving. Europol’s 2025 assessment projects about 8 million deepfakes shared online in 2025, up from roughly 500,000 in 2023, feeding CEO voice fraud and romance scams. The FBI’s 2025 IC3 Annual Report, the first in the unit’s roughly 25 years to carry a dedicated AI section, logged 22,364 AI-related complaints and nearly $893 million in losses for 2025, inside $20.877 billion in total internet-crime losses, up 26% on the year. Voice cloning now needs only seconds of source audio. That’s the part that should worry any finance team with a wire-approval process built on phone callbacks.

What Does Generative AI Cybersecurity Actually Cost to Deploy and Run?

Generative AI cybersecurity costs come from five places: licensing for an embedded copilot, token spend for custom integrations, the data and retrieval infrastructure underneath, evaluation and red-teaming, and analyst training. No independently audited total exists for any of them as of 2026, so budget by driver instead of by headline price.

Licensing for vendor-embedded copilots runs per seat or by consumption, and consumption models punish exactly the high-volume triage you bought the tool for. Read the metering clause before the feature list.

The other four drivers hide in engineering time:

• Token spend: every alert the model reads costs tokens, and raw EDR events are long. Trimming and deduplicating input before it reaches the model is usually the single largest saving.
• Data pipeline and retrieval: the layer that fetches the right logs and intel for each prompt is a build project in its own right, and it decides whether output is grounded or made up.
• Evaluation and red-teaming: you need a test set of your own alerts with known verdicts, plus adversarial prompts, before and after every model change.
• Training: analysts have to learn when to trust a verdict and when to override it. Skipping this is how a 7% accuracy gain becomes a net loss.

Custom builds shift cost from license to payroll, which is why teams weighing custom AI agent development against an off-the-shelf copilot should price the retrieval and eval work, since those two lines rarely appear in a vendor quote.

What about return? The only widely quoted figures are Microsoft’s own: a 30% cut in mean time to resolution, 22% faster task completion, and 7% higher accuracy, from a controlled study the company published and still cites in 2026. Directional, unaudited. The stronger ROI argument is staffing. If a copilot lets tier-1 analysts close cases that used to wait for a senior hire you can’t make, the return shows up as retained headcount and fewer 2 a.m. escalations.

Measure your baseline MTTR before the pilot. Otherwise the vendor’s number is the only one you’ll ever have.

Where Generative AI Fails in Security: Hallucinated Findings, Prompt Injection, and Data Leakage

Generative AI fails in security in four recurring ways: it fabricates findings, it obeys instructions hidden in the content it reads, it leaks sensitive data through prompts and retrieval, and, once allowed to act, it holds more privilege than anyone meant to grant. Each is a property of the model itself. None requires a clever attacker.

Fabrication is the one you can’t filter out. A model asked to triage an alert returns a confident verdict and a plausible chain of reasoning whether or not the indicators it cites exist. In practice that looks like an IP labeled a known command-and-control server that appears in no intel feed, or a timeline that places a login before the account was created. Attackers suffer the same defect. Anthropic’s November 2025 disclosure of an AI-run espionage campaign records the agent hallucinating credentials and fabricating results, which forced the operators to check its work by hand. If a state-backed team can’t run the model unsupervised, neither can your tier-1 queue.

Prompt injection ranks highest with standards bodies. The OWASP Top 10 for LLM Applications, 2025 edition keeps it at LLM01 for the second edition running. Direct injection is a user typing hostile instructions. Indirect injection is the version that matters for security tooling: the model ingests attacker-controlled content (a phishing email under analysis, a web page during enrichment, a document attached to a ticket) and that content hijacks what the model does next. A triage copilot reads hostile content all day. That’s its job.

OWASP states that neither retrieval-augmented generation nor fine-tuning closes this class of flaw. Google DeepMind’s 2025 account of hardening Gemini reports that adversarial training cut attack success by roughly 47% for Gemini 2.5, then adds the line that should sit on every architecture diagram:

“No model is completely immune,” per Google DeepMind’s 2025 report on advancing Gemini’s security safeguards, which pairs model hardening with input and output classifiers and system-level guardrails.

A 47% reduction leaves more than half of attacks standing. Layer the defenses.

Data leakage is quieter. Every prompt carrying a log excerpt, a credential fragment, or a customer record sends it somewhere, and a retrieval index built by copying source systems will surface files the analyst was never cleared to see. Ask three questions before go-live: where do prompts travel, who retains them, and does the index respect the access controls of the systems it copied from?

Over-privileged agents are the newest failure. Careful Adoption of Agentic AI Services, published in May 2026 by CISA, NSA, and their Australian, Canadian, New Zealand, and UK counterparts, names five risk categories:

• Privilege escalation
• Design and configuration failures
• Behavioral misalignment
• Structural brittleness
• Accountability gaps

CISA’s December 2025 principles for AI in operational technology carry the same warning into plants and utilities, where a wrong action stops a pump instead of closing a ticket.

Detection tools fail as well. A UC San Diego-led team announced a universal deepfake video detector in August 2025 with 98% reported accuracy, scanning whole-scene inconsistencies rather than facial artifacts. In March 2025, Australia’s CSIRO had published research showing major weaknesses in existing deepfake detectors under adversarial conditions. Both findings hold. A benchmark score is a snapshot, and the adversary moves after the photo is taken.

Which Organizations and Security Roles Get the Most Value From Gen AI Cybersecurity

The organizations that get the most value from gen AI cybersecurity are the ones already drowning in well-structured telemetry: large enterprises with a mature SIEM, managed security providers running many tenants, and regulated sectors that must document every triage decision. The roles that gain most are tier-1 analysts and the engineers who write detections. Everyone else gains less, and some gain nothing yet.

Value scales with alert volume. It scales again with data quality. That’s why the same tool transforms one SOC and barely registers in another.

By role, the picture is uneven on purpose:

• Tier-1 analysts: the largest uplift. Microsoft’s controlled study, still cited by the company in 2026, found junior analysts approaching senior performance on complex tasks, and alert fatigue is the daily condition summarization attacks directly.
• Detection engineers: faster rule drafting from intel reports and quicker query iteration.
• Threat hunters: natural-language pivots across data sources, with value capped by data coverage.
• Incident responders: timeline construction and report drafting, hours saved per incident.
• CISOs: the return is headcount math and a defensible audit trail rather than any single feature.

Who gains least? Anyone with thin telemetry or no data governance. A model reasoning over three log sources and a spreadsheet of assets produces confident nonsense, and teams without a map of what data exists and who may see it tend to find that out mid-pilot. An AI integration audit before procurement costs less than one after.

Fix the data first. The model will still be there next quarter.

What Changed in Generative AI and Cybersecurity This Year, From Agentic SOCs to New Regulations

Between mid-2025 and September 2026, generative AI in cybersecurity moved from chat-style copilots to agents that act, from hypothetical AI-run intrusions to a documented one, and from general AI governance to security-specific guidance from NIST, CISA, and the Five Eyes. Tooling, threats, and rules all shifted in the same twelve months.

Three implications follow. First, the agentic SOC is real, so action boundaries and approval gates are now procurement questions instead of research topics. Second, the November 2025 disclosure means a mature threat model assumes the adversary runs agents too, complete with their hallucinations. Third, governance finally has a security-specific spine. NIST’s Cyber AI Profile organizes AI risk around Govern, Identify, Protect, Detect, Respond, and Recover, the same six functions security teams already report against.

One caution on that last point. The NIST profile was a preliminary draft open for comment in early 2026, and a draft is a signal of direction rather than a finished requirement. Build to it anyway. The agencies that wrote the Five Eyes guidance and the OT principles are reading from the same page.

Frequently Asked Questions About Generative AI in Cybersecurity

The most-searched questions about generative AI in cybersecurity come down to seven: whether it helps or hurts, whether it replaces people, how it differs from older security AI, how criminals use it, whether public chatbots are safe for security work, what prompt injection is, and which skills to build. Short answers follow.

Is generative AI good or bad for cybersecurity?

Both, at the same time. The World Economic Forum’s Global Cybersecurity Outlook 2026 describes AI in cyber as dual-use: it speeds detection and response for defenders while making phishing, malware, and fraud cheaper for attackers. Which side wins inside your organization depends on data quality, guardrails, and whether your people expect AI-written lures.

Can generative AI replace security analysts?

No. It raises the floor for junior analysts and removes repetitive work, but it fabricates findings and can’t be held accountable for a decision. Microsoft’s vendor-reported study found junior analysts approaching senior performance with a copilot, which argues for fewer escalations and less burnout rather than for empty seats. Keep a human on every consequential action.

What is the difference between AI and generative AI in cybersecurity?

Traditional cybersecurity AI classifies. It scores anomalies, matches signatures, and flags unusual users. Generative AI produces new content: summaries, detection queries, code, incident reports, and, in the wrong hands, phishing emails and cloned voices. The first kind has run in SOCs for years. The second is newer, more capable, and used by both sides.

How do hackers use generative AI?

Four main ways, per Google’s Threat Intelligence Group AI Threat Tracker from November 2025: writing convincing phishing at scale, generating and rewriting malware code at runtime, researching targets and lateral-movement techniques, and running intrusion steps through AI agents. Voice and video cloning for fraud is the fifth, and the one draining the most money from victims in 2025.

Is ChatGPT safe to use for security work?

Only with rules. Anything pasted into a public chatbot leaves your environment, so logs, credentials, customer records, and unreleased vulnerability details stay out of it. A consumer assistant is fine for drafting a report from sanitized notes and unfit for triage on live telemetry. For real workloads, use an enterprise endpoint with retention terms you’ve actually read.

What is prompt injection?

Prompt injection is an attack in which text the model reads overrides the instructions it was given. Direct injection comes from a user typing hostile commands. Indirect injection hides them in a document, web page, or email the model processes, which is why security copilots, whose whole job is reading attacker content, are exposed. OWASP ranked it the top LLM risk in its 2025 list.

What certifications or skills cover AI in cybersecurity?

Skills matter more than credentials here. The ones that transfer are practical: reading the NIST AI 100-2e2025 adversarial ML taxonomy, working through the OWASP Top 10 for LLM Applications, building an evaluation set from your own alerts, and red-teaming a deployment for injection. A detection engineer who can also judge a model’s output on real alerts is worth more than any certificate.

Where to Start With Generative AI in Your Security Program

Start generative AI in your security program with one high-volume, low-risk workflow, a written set of data and approval rules, and a baseline measurement taken before the model touches anything. Five steps, in order:

1. Pick one workflow. Alert summarization or phishing triage: high volume, reversible, and easy to score against analyst verdicts.
2. Write the rules before the pilot. Decide which data may enter a prompt, where it travels, who retains it, and which actions need a human click. NIST AI 600-1 and the May 2026 Five Eyes guidance on agentic AI give you the categories to cover.
3. Measure first. Record mean time to resolution, false-positive rate, and analyst hours per incident for the workflow you chose. Without that baseline you can’t tell a real gain from a vendor slide.
4. Red-team the deployment. Feed it emails and documents carrying hidden instructions, and confirm the guardrails hold before it reads live attacker content.
5. Retrain your people. Update phishing awareness for AI-written lures and add a callback rule for any voice request that moves money.

Then run the pilot for a quarter. Compare against the baseline, look hard at the accuracy line and at every verdict the model got confidently wrong, and only then decide whether agents get to act on production systems.

The model you pick matters less than the data you put in front of it and the limits you set on what it may do. Book the baseline measurement this week. Everything else waits on it.