Fileless Malware Detection: How SOC Teams Hunt In-Memory Attacks

Fileless Malware Detection: How SOC Teams Hunt In-Memory Attacks

Traditional malware detection relies on a simple principle: malware writes files to disk, antivirus scans those files, and signatures match. Fileless malware breaks that assumption entirely. It executes entirely in memory, hijacks legitimate system tools, and leaves behind little or no disk artifacts, making conventional malware detection strategies largely ineffective.

For SOC teams, this represents one of the most difficult detection challenges in modern cybersecurity. This guide covers exactly what fileless malware is, how it operates at the process and memory level, and most importantly, how to detect it through behavioral telemetry, memory forensics, and threat hunting.

⚠ Critical SOC Challenge

Fileless attacks are involved in a significant portion of modern breaches. Traditional endpoint security tools with signature-based detection miss the majority of fileless attacks because there is no file to scan.

What Is Fileless Malware?

Fileless malware (also called memory-resident malware or living-off-the-land malware) is a class of attack that operates entirely within the memory of legitimate processes without ever writing a malicious executable to disk. Rather than deploying a standalone binary, attackers abuse built-in system tools and trusted scripting environments already present on the target machine.

The term "fileless" is somewhat misleading. Many attacks begin with a file (a malicious document, a phishing link). What makes them fileless is the payload execution stage: the actual malicious code lives only in RAM, injected into running processes, written to the Windows Registry, or reflectively loaded in ways that never touch the filesystem in a scannable form.

Fileless vs. Traditional Malware: Key Differences

Property	Traditional Malware	Fileless Malware
Disk presence	Drops .exe, .dll, .bat, scripts	Minimal or none after execution
Execution mechanism	Launches its own process	Injects into or hijacks legitimate processes
Detection method	File hash, signature scan	Behavioral, telemetry, and memory analysis
AV/EDR evasion	Obfuscation, packing	Inherits the trust of the host process
Persistence	Startup folders, services, registry run keys	WMI subscriptions, scheduled tasks, registry payloads
Forensic evidence	File artifacts, PE headers on disk	Memory dumps, event logs, PowerShell history

Common Fileless Malware Techniques

Understanding the attack surface is the foundation of effective malware detection. These are the primary execution methods used by modern fileless threats in the wild.

T1059.001 - PowerShell In-Memory Execution: Attackers use PowerShell's Invoke-Expression (IEX) or -EncodedCommand to download and execute shellcode entirely in memory. The payload never touches disk. Commonly paired with DownloadString to pull code from remote URLs.

T1055 - Process Injection: Malicious code is injected into a trusted process (svchost.exe, explorer.exe, lsass.exe) using Windows APIs like VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread. The malicious code runs under the identity and privileges of the host process.

T1055.002 - Reflective DLL Injection: A DLL is loaded directly from memory without calling Windows loader APIs, bypassing the standard DLL load path and therefore most application allow-listing. Used heavily by frameworks like Cobalt Strike and Metasploit.

T1547.001 - Registry-Based Persistence: Malicious scripts or shellcode are stored directly inside Windows Registry values (e.g., HKCU\Software\Microsoft\Windows\CurrentVersion\Run). A legitimate binary like mshta.exe or wscript.exe reads and executes the payload at startup.

T1546.003 - WMI Event Subscription: Windows Management Instrumentation (WMI) subscriptions are used to execute a payload when specific conditions are met (system boot, user logon, scheduled time). The execution persists across reboots without a single file dropped.

T1218 - LOLBins / Signed Binary Proxy: Legitimate, Microsoft-signed binaries like mshta.exe, regsvr32.exe, certutil.exe, wmic.exe, and rundll32.exe are abused to download and execute arbitrary code. Because these are trusted OS tools, they bypass most application control policies.

The Typical Fileless Attack Chain

Fileless attacks follow a recognizable lifecycle despite their evasive nature. Knowing each stage allows SOC analysts to identify detection opportunities at multiple points rather than relying on a single control.

1. Initial Access

A user opens a malicious Office document with a macro, clicks a phishing link, or exploits a vulnerable browser plugin. The initial stage may write a small loader to disk, but this is often just a stager, not the final payload.

2. In-Memory Payload Download

PowerShell, mshta, or wscript reaches out to an attacker-controlled URL and downloads the second-stage shellcode or script entirely into memory using Net.WebClient.DownloadString() or similar APIs.

3. Execution in Legitimate Process

The payload is injected into a trusted process (explorer.exe, svchost.exe) or executed via a LOLBIN. From this point, all malicious activity appears to originate from a legitimate Windows binary.

4. Persistence Without Files

The attack establishes persistence through WMI event subscriptions, scheduled tasks pointing to registry-embedded scripts, or COM object hijacking mechanisms that survive reboots without traditional file artifacts.

5. C2 and Lateral Movement

The attacker establishes command-and-control over the injected process. Lateral movement proceeds through wmic, PsExec, or pass-the-hash attacks, all executed from memory, often mimicking legitimate admin operations.

Malware Detection Strategies for Fileless Attacks

Because fileless malware defeats file-based detection, your malware detection strategy must shift entirely to behavioral signals, telemetry analysis, and memory inspection. Below are the primary detection layers every SOC should have active.

1. PowerShell Script Block Logging and Transcription

PowerShell is the most commonly abused tool in fileless attacks. Windows provides native PowerShell logging capabilities that, when enabled, capture exactly what code was executed, including decoded content from EncodedCommand invocations.

Required GPO Settings:

Enable via Group Policy: Computer Configuration → Administrative Templates → Windows Components → Windows PowerShell. Enable both "Turn on Module Logging" and "Turn on PowerShell Script Block Logging". Set logging for all modules ("*").

KQL: Microsoft Sentinel / Defender Detect encoded PowerShell with network download

2. Process Injection Detection via API Call Monitoring

Process injection leaves distinctive traces in API telemetry. EDR solutions that support kernel-level event capture (CrowdStrike Falcon, Microsoft Defender for Endpoint, SentinelOne) expose these as process events that can be correlated in your SIEM.

Look for these API call sequences in process telemetry. Individually, they may be legitimate, but in sequence, they constitute a classic injection chain:

3. Memory Forensics for In-Memory Attack Detection

When a live process is suspected of hosting injected code, memory forensics becomes essential. The goal is to identify memory regions that contain executable code but lack a corresponding file on disk a hallmark of reflective injection.

As covered in depth in the CyberDefenders guide "Why Memory Forensics Matters in Modern Cybersecurity," memory analysis exposes the hidden layer that attackers rely on to remain invisible.

Volatility 3: Memory Analysis: Detect injected code in process memory

Key Indicator:

RWX memory regions with no backing file are the single most reliable indicator of reflective injection or shellcode execution. Volatility's malfind plugin specifically looks for private committed memory with execute permissions that doesn't map to a file on disk; these should always be investigated.

4. WMI Activity Monitoring

WMI-based persistence is notoriously difficult to detect because it operates entirely within the Windows Management Instrumentation framework. The key events to monitor are WMI subscription creation events, specifically EventFilter, EventConsumer, and FilterToConsumerBinding object creation in WMI.

KQL - Detect WMI Persistence via Sysmon Requires Sysmon Event ID 19, 20, 21

5. AMSI (Antimalware Scan Interface) Telemetry

Starting with Windows 10 / Server 2016, Microsoft's AMSI provides a bridge between scripting engines (PowerShell, JScript, VBScript, .NET) and security products. Every script invocation passes through AMSI before execution, giving AV/EDR a chance to inspect the decoded runtime content and bypass obfuscation that would otherwise fool static analysis.

In your SIEM, AMSI detection events (Event ID 4688 or equivalent EDR telemetry) where a malicious string was detected but execution proceeded (AMSI bypass was attempted) are particularly high-value alerts. Common AMSI bypass strings to hunt for include AmsiInitFailed, [Ref].Assembly.GetType, and various obfuscated AMSI patch attempts.

Detection Rules: Sigma Signatures for Fileless Activity

Sigma rules provide a platform-agnostic way to describe detection logic. The following rules cover core fileless attack patterns and can be converted to Splunk SPL, Microsoft Sentinel KQL, Elastic DSL, or any other SIEM query language.

sigma_rule_01 - Suspicious PowerShell Cradle with Web Download T1059.001

sigma_rule_02 - LOLBin Spawning Unusual Child Process T1218

Threat Hunting Hypotheses for In-Memory Attacks

Proactive threat hunting, rather than waiting for alert fires is the most effective way to find fileless attackers who have slipped past automated detection. Use the following hunting hypotheses as starting points.

Each hypothesis pairs with the threat hunting approach covered in our guide on Hacker Mindset: How Attackers Think, understanding attacker intent is inseparable from building effective hunts.

Hunting Hypothesis 1

"Legitimate processes are making outbound network connections that they never normally make." Hunt for svchost.exe, explorer.exe, or lsass.exe initiating direct internet connections particularly to non-Microsoft IP ranges or uncommon ports.

Hunting Hypothesis 2

"A trusted scripting binary ran in an unusual context." PowerShell is spawned by Word, Excel, or a browser. mshta.exe with a remote URL argument. certutil.exe downloading a file with the -decode flag. Any of these in your environment deserves review.

Hunting Hypothesis 3

"A process is running code from memory regions that have no associated file on disk." Requires memory forensics capability (live response via your EDR, or a full memory dump). Hunt for private RWX regions in high-value processes like lsass.exe, svchost.exe, explorer.exe.

Hunting Hypothesis 4

"WMI or scheduled task persistence was created in a non-standard way." New WMI event subscriptions created by a non-admin account, or at an unusual hour, are a strong signal. Baseline your environment's WMI subscriptions and alert on any deviation.

Incident Response for Fileless Malware

When a fileless malware detection fires, the IR workflow differs significantly from handling a traditional file-based infection. Speed is critical the evidence exists only in memory and will be lost on system restart.

1. Isolate But Do Not Restart

Isolate the endpoint from the network immediately. Do not restart or shut down this will destroy all in-memory evidence. Use your EDR's network isolation feature to cut off C2 while preserving the memory state.

2. Capture a Full Memory Image

Use tools like WinPmem, Magnet RAM Capture, or your EDR's live memory acquisition feature to capture a full physical memory image. This becomes the primary forensic artifact for the investigation.

3. Collect Volatile Data

Before any further action, collect: running process list with full command lines, open network connections (netstat -anob), loaded drivers, WMI subscriptions, active PowerShell sessions, and scheduled tasks. These exist only while the system is running.

4. Analyze Memory with Volatility

Run malfind, pslist, netscan, dlllist, and cmdline plugins against the memory image. Look for hidden processes, injected modules, suspicious parent-child relationships, and memory regions with no mapped file.

5. Trace Lateral Movement

Cross-reference with your SIEM for authentication events, remote PowerShell sessions (Event ID 4688, 4624 Type 3), and WMI remote activity (Event ID 4648). Fileless attackers rarely stay on one machine hunt the lateral movement path.

For a deeper dive into log analysis during incident response, see our guide on the Alert Triage Process: The Complete SOC Analyst's Guide, which walks through systematic evidence prioritization.

MITRE ATT&CK Mapping Summary

Mapping your detections to MITRE ATT&CK ensures coverage visibility and helps identify gaps. The following table maps the primary fileless attack techniques to the detections discussed in this guide.

MITRE ID	Technique	Detection Method	Coverage
T1059.001	PowerShell	Script Block Logging, AMSI, KQL hunt	High
T1055	Process Injection	EDR API telemetry, memory forensics	Medium
T1055.002	Reflective DLL Injection	Memory forensics (malfind), RWX hunt	Medium
T1218	Signed Binary Proxy (LOLBins)	Process spawn rules, Sigma detection	High
T1546.003	WMI Event Subscription	Sysmon EIDs 19/20/21, WMI logging	High
T1547.001	Registry Run Keys	Registry change monitoring (Sysmon EID 13)	High

Preventive Hardening to Reduce the Attack Surface

Detection must be paired with hardening. The following controls directly reduce the attack surface for fileless malware; they don't eliminate risk, but they make each technique significantly harder to execute at scale.

Control	Target Technique	Implementation
Constrained Language Mode (PowerShell)	PowerShell cradles	Group Policy + AppLocker/WDAC enforcement
Attack Surface Reduction rules	Office macros, LOLBins	Microsoft Defender ASR policies
Disable WMI remote access	WMI lateral movement	Firewall rules, disable WINRM where unused
Credential Guard	Credential theft from lsass	Hyper-V HVCI + Credential Guard GPO
Just Enough Administration (JEA)	Privilege abuse via PowerShell	PowerShell JEA role definitions
Application Control (WDAC)	LOLBin abuse, unsigned code	Windows Defender Application Control policies

Conclusion: Fileless Malware Detection Requires a New Mindset

Fileless malware and in-memory attacks represent the evolution of threat actor tradecraft specifically designed to defeat traditional malware detection. There are no files to hash. No binaries to sandbox. The attacker borrows the trust and identity of your own operating system.

Effective detection requires a layered approach: enabling comprehensive logging (PowerShell Script Block, Sysmon, AMSI), building behavioral detection rules tuned to your environment, investing in memory forensics capability, and training SOC analysts to hunt proactively rather than wait for signature alerts.

The disk forensics work covered in Disk Forensics: SOC Analyst Playbook complements this guide while fileless malware avoids disk artifacts, the initial access stage and persistence mechanisms frequently leave traces that disk forensics can surface.

For hands-on practice investigating real fileless attack scenarios, CyberDefenders' Blue Team Labs include dedicated labs covering PowerShell-based attacks, Cobalt Strike beacon analysis, and memory forensics with Volatility applied in realistic SOC simulation environments.

Related Blogs:

• SOC Detection: Encoded PowerShell Detection: How to Investigate Encoded PowerShell Commands Read article →
• SOC Operations: Alert Triage Process: The Complete SOC Analyst's Guide Read article →
• Disk Forensics: SOC Analyst Playbook: how disk artifacts inform SOC detection and threat hunting during incident investigations. Read article →