DEFENDER ENGAGEMENT // STRUCTURAL DETECTION RESEARCH

01 // SITUATION REPORT

The VADER rootkit kill chain has 8 components. 7 pass Defender's static scan cleanly after XOR key mutation. One component — inject_dll (signature set HOTEL) — is detected regardless of which XOR key is used. After 10 consecutive key rotations across the full 0x00-0xFF range, every compiled binary is flagged.

This document analyses WHY Defender detects inject_dll, maps the three-layer detection chain, and identifies documented gaps in each layer. This is research into detection mechanics — understanding how endpoint security works from the inside.

SKYWALKER (cold standby fork) has the same inject_dll pattern. Previously found a clean key after 8 attempts, but as of 18 June 2026, SKYWALKER inject_dll also fails 10/10 with high-bit keys. Defender has updated its signatures — both variants are now equally detected. The detection is purely structural.

02 // MUTATION PIPELINE RESULTS

The XOR key mutation pipeline (mutate.py) automates the compile → scan → rotate cycle. For each component, it generates a new random XOR key, re-encodes all string arrays, recompiles, and scans with MpCmdRun.exe -Scan -ScanType 3. Up to 10 attempts per component.

What XOR Mutation Does

XOR encoding transforms plaintext strings (API names, DLL names, file paths) into byte arrays that don't match Defender's string signatures. At runtime, a single-byte XOR key decodes them back. The encoding is its own inverse: A ^ K = E, E ^ K = A.

What XOR Mutation Does NOT Do

It changes the data in .rdata, not the code in .text. The API call sequence, the function structure, the import table, the PE layout — all remain identical regardless of key. If Defender's signature is on code structure rather than string patterns, XOR rotation has zero effect.

03 // DETECTION CHAIN ANALYSIS

Defender operates a three-layer detection pipeline. Each layer uses different signals. inject_dll likely triggers all three.

Layer 1 — Static Analysis (Import Table)

Before any code executes, Defender inspects the PE file's Import Address Table. Combinations of APIs associated with injection patterns raise a heuristic score:

SUSPICIOUS IMPORT COMBINATION (inject_dll):
  CreateToolhelp32Snapshot   ← thread/process enumeration
  Thread32First / Thread32Next
  OpenThread                 ← accessing other threads
  SuspendThread / ResumeThread
  GetThreadContext / SetThreadContext  ← context manipulation
  AddVectoredExceptionHandler         ← VEH registration
  GetProcAddress / LoadLibraryA       ← dynamic resolution

A DLL with this import combination and minimal legitimate exports is flagged with elevated confidence before the emulator even starts.

Layer 2 — Emulation (mpengine.dll)

Defender's ~14MB emulation engine translates machine code to an intermediate language and executes it in a sandbox. It emulates x86/x64 instructions, Windows API calls, kernel functions, filesystem, and registry. The emulator enters at DllMain(DLL_PROCESS_ATTACH) and follows code flow from there.

inject_dll does ALL suspicious work inside DllMain: resolve targets, register VEH, enumerate threads, set debug registers. This is the worst case for emulation detection — everything is reachable within the emulator's instruction budget in a single linear code path.

Layer 3 — Runtime Behavioral (ETW-TI)

The Microsoft-Windows-Threat-Intelligence ETW provider instruments thread manipulation from kernel space. This layer only fires at runtime (not during file scan), but Defender's cloud analysis and AMSI integration can correlate runtime signals back to the file.

04 // DEFENDER EMULATION ENGINE

The emulator is the second line. Understanding its capabilities and limits is essential for understanding detection.

Capabilities

• Full x86/x64 instruction set emulation
• Emulated Windows API (CreateFile, LoadLibrary, GetProcAddress, etc.)
• Emulated NT kernel calls (NtCreateFile, NtAllocateVirtualMemory, etc.)
• Emulated filesystem and registry
• DLL loading and dependency resolution
• Entry point analysis: DllMain for DLLs, main/WinMain for EXEs

Known Limits

• Finite instruction budget — terminates after N instructions or N API calls
• Time limit on total emulation
• Memory ceiling on emulated allocations
• Known environment artifacts: GetComputerName returns "HAL9TH", GetUserName returns "JohnDoe"
• Cannot fully emulate multi-threaded behavior
• Network calls return simulated responses

Reference

BlackHat 2018: Windows Offender — Reverse Engineering Windows Defender's Emulator

0xAlexei/WindowsDefenderTools (GitHub)

05 // ETW THREAT INTELLIGENCE (KERNEL TELEMETRY)

ETW-TI is the highest-fidelity detection layer. It operates at kernel level — usermode code cannot hook, intercept, or disable it without a kernel driver.

Relevant Events

ETW-TI Event           Trigger                        inject_dll fires?
———————           —————                        —————————
Task 5  (local)        NtSetContextThread (self)       YES (own thread)
Task 10 (remote)       NtSetContextThread (other)      YES (all threads)
Memory alloc events    VirtualAllocEx (remote)         NO
APC queue events       NtQueueApcThread                NO

Debug Register Inspection

When NtSetThreadContext fires with CONTEXT_DEBUG_REGISTERS flags, kernel-level telemetry can inspect the DR0-DR3 values being written. If those values resolve to addresses within amsi.dll (AmsiScanBuffer) or ntdll.dll (EtwEventWrite), that is a near-certain indicator of a patchless AMSI/ETW bypass.

06 // NtContinue — THE DOCUMENTED GAP

Praetorian Security documented that NtContinue updates thread context — including debug registers — WITHOUT traversing the kernel code path that emits EtwTiLogSetContextThread.

NtSetContextThread  →  fires ETW-TI  →  Defender sees DR manipulation  →  DETECTED
NtContinue          →  NO ETW-TI     →  kernel telemetry BLIND          →  ???

How NtContinue Works

NtContinue takes a CONTEXT structure and a boolean RaiseAlert. It restores the thread's context from the structure — including RIP, RSP, and debug registers. Because it's designed for the exception dispatcher path (which fires millions of times in normal operation), the kernel does NOT instrument it with ETW-TI telemetry the way it instruments NtSetThreadContext.

Implications

• A thread can set its OWN debug registers via NtContinue without generating kernel telemetry
• Combined with APC injection, each target thread could call NtContinue to set its own DRs
• This removes the cross-thread SetThreadContext pattern entirely
• The thread enumeration + suspend/resume pattern is also eliminated

Source: Praetorian — ETW Threat Intelligence and Hardware Breakpoints

07 // WHY dark_room PASSES BUT inject_dll FAILS

Both components use the same VEH + HWBP technique. Both set DR0 on AmsiScanBuffer and DR1 on EtwEventWrite. Both use the same VEH handler logic. But dark_room passes Defender's scan and inject_dll does not.

The difference is the cross-thread manipulation. Setting your own debug registers via GetCurrentThread() is a benign-looking operation — debuggers do this constantly. Enumerating every thread in a process, suspending each one, setting debug registers, and resuming is an injection pattern that Defender specifically recognizes.

08 // PIPELINE FIXES — DEFENDER QUARANTINE HANDLING

During mutation testing, a secondary problem emerged: when Defender detects and quarantines vader_inject.dll, it places an exclusive file lock. The mutation pipeline then fails because the compiler can't overwrite the locked binary.

The Quarantine Lock Problem

Defender detects vader_inject.dll
  → Places exclusive handle on file
  → os.remove()   → PermissionError [WinError 5]
  → os.rename()   → OSError [WinError 225]
  → cl.exe /Fe:   → LNK1104 "cannot open file"
  → Mutation pipeline CRASHES

The Fix — Temp-Dir Compile Fallback

When the target binary is locked, the pipeline now:

• Detects the lock via os.remove() try/except
• Creates a temporary build directory (tempfile.mkdtemp())
• Compiles to the temp directory instead
• Attempts os.replace() to swap the new binary in
• Falls back to .new suffix if replace fails
• Scans whichever file exists (prefers .new)
• On CLEAN result, promotes .new to expected filename

This fix was applied to 4 files across both VADER and SKYWALKER repositories:

• vader-rootkit/mutate.py — compile_component() + rotate_component()
• vader-rootkit/deploy.py — compile_component()
• skywalker/mutate.py — compile_component() + rotate_component()
• skywalker/deploy.py — compile_component()

09 // STRUCTURAL DIFF — VADER vs SKYWALKER

Priority research direction #1: diff the two inject_dll source files. VADER fails 10/10 key rotations. SKYWALKER originally found a clean key after 8 attempts (now also fails 10/10 as of 18 June 2026 — Defender updated signatures). What's different?

The Key Difference (Literally)

Why This Matters for Detection

VADER's byte distribution is a statistical fingerprint. Key 0x77 is itself an ASCII character (lowercase 'w'). XOR-encoding ASCII text with an ASCII-range key produces encoded values that cluster in the low half of the byte space (0x00-0x59). This narrow, low-range distribution screams "XOR-encoded ASCII strings" to entropy-based scanners.

VADER produces null bytes. Where plaintext contains 'w' (two occurrences in "Windows" in the canary path), the encoding produces 0x00. Null bytes in .rdata const arrays adjacent to non-null data is an unusual pattern that can serve as a heuristic anchor point.

SKYWALKER's bytes look like normal binary data. Key 0xE3 has its high bit set, which flips the high bit of every ASCII character during encoding. All encoded values land in the 0x80-0xD9 range — overlapping with common binary data patterns like relocations, padding, and encoded resources. This looks unremarkable in a PE .rdata section.

Detection Model

The evidence suggests Defender uses BOTH detection layers against inject_dll:

Layer A — Static byte pattern signature
  Key 0x77 → low-range clustering → ALWAYS MATCHES → DETECTED
  Key 0xE3 → high-range bytes     → MISSES          → passes static

Layer B — Behavioral/heuristic rule (API call sequence)
  Both keys → same code structure → SOMETIMES MATCHES → inconsistent

VADER:   Layer A hits 100%  →  always DETECTED (Layer B irrelevant)
SKYWALKER: Layer A misses    →  Layer B fires inconsistently → CLEAN after 8 tries

Implications for Mutation Pipeline

• The mutate.py key selection should prefer keys with high bit set (0x80-0xFF)
• Keys that produce null bytes when XOR'd with common ASCII ('a'-'z', 'A'-'Z', '.', '\') should be avoided
• Key selection alone does not guarantee evasion — it only defeats Layer A (static). Layer B (behavioral) still fires based on the code structure
• To defeat both layers, the code structure must change (see research directions below)

10 // RESEARCH DIRECTIONS

Prioritised by expected impact and feasibility. Each direction targets a specific layer of the detection chain.

11 // SOURCES

• Praetorian: ETW Threat Intelligence and Hardware Breakpoints
• CrowdStrike: Patchless AMSI Bypass Attacks
• FluxSec: Detecting VEH Abuse via Debug Registers
• BlackHat 2018: Reverse Engineering Defender Emulator
• 0xAlexei/WindowsDefenderTools (GitHub)
• Outflank: Early Cascade Injection
• Hackmosphere: Bypass Defender 2025
• InfoGuard Labs: Fuzzing mpengine.dll
• Elastic Security Labs: PHANTOMPULSE RAT
• CyberAsk: Detecting Process Injection via ETW-TI