What Is Generative AI in Cybersecurity?

In early 2024, a finance worker at the engineering firm Arup joined a video call with people who looked and sounded like the company's UK chief financial officer and several colleagues. Every face on the call was a deepfake. The worker, reassured by familiar voices, approved fifteen transfers totaling about 200 million Hong Kong dollars, roughly 25 million US dollars, to accounts the attackers controlled. No malware ran. No system was breached. A generative model produced convincing video and audio, and a normal business process did the rest.

That is the shape of generative AI in security. It does not hack a firewall. It manufactures the content, the text, voices, faces, and code, that makes the rest of an attack work. The same capability, pointed the other way, drafts incident summaries, explains alerts in plain language, and writes detection queries for analysts who would otherwise hand-build them.

This guide defines generative AI, explains how the models actually produce output, and then splits the security picture in two: how defenders use GenAI in a SOC, and how attackers use it against you. It is written for blue teams who now have to account for a technology that sits on both sides of the fight.

What is generative AI (GenAI)?

Generative AI is a class of machine learning that produces new content, text, images, audio, video, or code, by learning the statistical patterns in a large body of existing data and then sampling from what it learned. Where a traditional classifier answers a closed question (is this file malicious, yes or no), a generative model answers an open one (write a phishing email, generate a face, complete this code). The output is novel, not retrieved.

The distinction that matters for security is generation versus discrimination. A discriminative model draws a boundary between known categories. A generative model learns the underlying distribution well enough to create fresh samples that fit it. That is why GenAI is the engine behind chatbots that write, image tools that paint, and voice clones that speak: all of them sample new data that looks like the data they trained on.

Large language models (LLMs) are the form most security teams meet first. An LLM is a generative model trained on text that predicts the next token over and over to produce fluent language. GPT, the family behind many AI assistants, stands for Generative Pre-trained Transformer, and the three words name exactly what it is: it generates, it is pre-trained on a broad corpus, and it is built on the transformer architecture. The same generative principle, applied to pixels or audio waveforms instead of tokens, gives you image and voice generation.

How generative AI works

Three building blocks, stacked, get you from raw data to a model that writes a phishing email or a detection rule.

Deep learning and neural networks. A neural network is layers of simple math units, loosely inspired by neurons, that adjust their internal weights as they see training data. Stack enough layers and the network learns hierarchical patterns: characters into words, words into grammar, grammar into something that reads like a sentence. Training means showing the network enormous volumes of examples and nudging the weights until its output matches the patterns in the data.

The transformer. Modern GenAI runs on the transformer, the architecture introduced in the 2017 paper *Attention Is All You Need* by Vaswani and colleagues at Google. Its core idea, self-attention, lets the model weigh how much every part of the input relates to every other part, in parallel, rather than reading strictly left to right. That parallelism is why models could be trained on internet-scale text and why output stays coherent across long passages. Almost every current LLM is a transformer.

Generation by sampling. Once trained, an LLM generates by predicting the next token, appending it, and repeating. A "token" is a chunk of text, roughly a word or word-piece. The model outputs a probability distribution over all possible next tokens and samples one; that choice feeds back in, and the loop continues until the response is complete. The same loop, run on a model trained for images, drives diffusion models, which start from random noise and denoise it step by step into a coherent picture. Generative adversarial networks (GANs), introduced by Ian Goodfellow and colleagues in 2014, were an earlier approach that pits a generator against a discriminator, and they remain a backbone of deepfake media.

The practical takeaways for a defender: the model only knows what was in its training data and its current context, it produces plausible output rather than verified-true output (the source of "hallucination"), and it reads instructions and data through the same channel, which is the structural flaw attackers exploit.

Generative AI in cybersecurity: defensive uses

GenAI is most useful in the SOC where the bottleneck is language and volume, not judgment. Four uses are real today.

Alert and detection summarization. The first hours of an investigation are spent reading: log lines, process trees, alert payloads, threat-intel reports. An LLM that ingests that material and produces a plain-language summary, what fired, on which host, what the indicators are, compresses triage time. The analyst still decides; the model does the reading and drafting.

Detection and query authoring. Writing a SIEM query, a YARA rule, or a regex is exactly the kind of structured-language task GenAI does well. An analyst describes the behavior in English and the model drafts the query, which the analyst reviews and tunes. It lowers the barrier for junior staff and speeds up senior ones. The review step is not optional: a model will confidently produce a query that is subtly wrong.

Knowledge and report drafting. Incident reports, runbook steps, and post-incident write-ups are language-heavy deliverables that pull analysts off live work. GenAI drafts the first version from the case data, and a human edits for accuracy. The gain is speed on the boilerplate, not the analysis.

Phishing and synthetic-data generation for testing. Blue teams use GenAI to generate realistic phishing lures for awareness training and to synthesize log or malware-behavior data for testing detections, the defensive mirror of how attackers use the same tools. This is also where agentic AI enters the SOC: assistants that do not just answer but take steps, pulling context and chaining actions under human supervision.

The honest limit on all four: GenAI is a force multiplier on tasks that involve reading and drafting, not a replacement for analysis. It hallucinates, it has no ground truth, and it widens your attack surface (see the risks below). Every output it produces in a security workflow needs a human gate before it drives an action.

Generative AI in cybersecurity: how attackers use it

The same capabilities lower the cost and raise the quality of attacks. This is the side a defender has to plan for.

Phishing at scale, without the tells. The old reliable signal, broken grammar and awkward phrasing in a phishing email, is gone. An LLM writes fluent, targeted, personalized lures in any language, at volume, for near-zero cost. It can mimic a specific person's writing style from a few samples. This raises the floor on every phishing campaign and is the most immediate GenAI threat to most organizations.

Deepfakes and voice cloning. Generative video and audio produce convincing impersonations of real people, the technique behind the Arup fraud. A cloned voice on a phone call or a deepfake executive on video defeats the "I recognize them" check that a lot of social engineering relies on. This is the engine behind a fast-growing class of AI social engineering, where the impersonation itself is synthetic.

Malware and tooling assistance. GenAI helps attackers write, obfuscate, and vary malicious code, and lowers the skill needed to assemble an attack chain. Underground "uncensored" LLM services market themselves on stripping the safety guardrails commercial models enforce. The realistic read is that GenAI is an accelerant and a skill-leveler for malware development more than a source of novel, never-seen techniques, but accelerant is enough to matter.

Attacks on AI systems themselves. As organizations deploy LLMs, the models become targets. Prompt injection, hiding instructions inside the data a model reads so it obeys the attacker, is ranked LLM01, the top entry in the OWASP Top 10 for LLM Applications. The broader discipline of attacking and defending these models is adversarial AI and machine learning, and it is now a required competency for any team running GenAI in production.

Pros and cons of generative AI in cybersecurity

GenAI is neither a silver bullet nor a net negative. It shifts work and changes the threat model. The table sets the two sides against each other.

Two costs deserve a closer look. First, hallucination: a generative model optimizes for plausible output, not correct output, so it will produce a confident, wrong answer with no signal that it is wrong. In a security workflow that means every model output, a query, a summary, an attribution, is a draft to verify, never a fact to act on. Second, data and privacy exposure: training and operating these models can pull in sensitive data, and feeding internal incident data into a third-party model is itself a disclosure decision. The governance question, what data the model sees and where it goes, is as important as the detection question.

Frequently Asked Questions

What is generative AI in cybersecurity?

Generative AI in cybersecurity is the use of models that create new content, text, code, images, audio, on both sides of the security fight. Defenders use it to summarize alerts, draft detection queries, and write reports. Attackers use it to generate phishing lures, deepfakes, and malware. It is a force multiplier, not a standalone control, and its output always needs human verification.

How is generative AI different from traditional AI?

Traditional, discriminative AI classifies or predicts within known categories, such as labeling a file malicious or benign. Generative AI learns the underlying distribution of its training data and produces new samples that fit it, such as writing an email or generating an image. The first draws boundaries between things that exist; the second creates things that did not exist before.

How do attackers use generative AI?

Attackers use generative AI to write fluent, personalized phishing emails at scale, to produce deepfake video and cloned voices for fraud and social engineering, and to help write and obfuscate malware. They also attack the AI systems defenders deploy, most commonly through prompt injection, ranked the top risk in the OWASP Top 10 for LLM Applications.

Can generative AI help a SOC?

Yes, on language-heavy and high-volume work. GenAI compresses triage by summarizing alerts and logs, drafts SIEM and detection queries from plain-language descriptions, and writes first-draft incident reports. It does not replace analysis: outputs can be subtly wrong, so a human reviews every result before it drives an action.

What is prompt injection?

Prompt injection is an attack on large language models in which an attacker hides instructions inside the data the model reads, exploiting the fact that an LLM processes instructions and data through the same channel. It is ranked LLM01, the top risk in the OWASP Top 10 for LLM Applications. It is the main way attackers subvert GenAI systems that organizations deploy.

What is a deepfake?

A deepfake is synthetic audio, video, or imagery generated by AI to convincingly impersonate a real person. Generative adversarial networks and diffusion models produce the media. In security, deepfakes power fraud and social engineering by defeating voice and face recognition, as in the 2024 Arup case where a deepfaked video call led to a roughly 25 million US dollar transfer.

Does generative AI hallucinate, and why does that matter?

Yes. A generative model optimizes for plausible output, not verified-true output, so it can produce a confident answer that is wrong with no signal that it is wrong. In a security context that means treating every GenAI output, a detection query, a summary, an attribution, as a draft to verify against ground truth, never as a fact to act on directly.

The bottom line

Generative AI creates content; in security that content cuts both ways. Attackers use it to manufacture phishing, deepfakes, and malware at a quality and scale that erase the old warning signs, and they target the models defenders deploy through prompt injection. Defenders use the same models to read faster and draft faster: summarizing alerts, authoring detection queries, and writing reports, with a human checking every output.

The models are transformers that learn patterns from huge datasets and generate by sampling, which is precisely why they are fluent, why they hallucinate, and why instruction-data confusion is exploitable. For a blue team the task is not to adopt or reject GenAI but to use it where reading and drafting are the bottleneck, verify everything it produces, govern what data it touches, and defend the models you run as the new attack surface they are.