TL;DR: Writing your own shellcode encoder/decoder pair is a rite of passage on the OSED path and a practical skill any time msfvenom output gets flagged or your target buffer rejects certain bytes. An encoder mangles your payload at build time so it survives the wire and the parser; a decoder stub (prepended to the payload) reverses the transform in memory and jumps into the decoded blob. The classic patterns are XOR, ADD/SUB, rolling-XOR, and alphanumeric-only (Skape). The hard parts are not the math: they are bad-character avoidance (bad-character-handling), stub size, register discipline, and getting self-modifying code to actually execute under modern Windows mitigations. Companion to custom-windows-shellcode-writing, osed-custom-exploit-walkthrough, egghunters, pe-backdooring-and-code-caves, and stack-bof-walkthrough-end-to-end.

Why it matters

Three operational reasons drive custom encoders:

1. Signature avoidance. shikata_ga_nai and friends have been in every vendor’s signature feed for over a decade. A 30-byte custom XOR decoder you wrote yesterday is, by definition, not in anyone’s YARA pack. This buys you minutes-to-hours, not weeks, but on a one-shot OSED exam machine that is plenty.
2. Charset restrictions. Real bugs filter input. strcpy kills \x00. lstrcpyW widens everything. HTTP headers reject \r\n. SQL injection vectors choke on quotes. The decoder lets the wire-visible payload live inside whatever subset of bytes the bug allows, while the decoded shellcode in memory uses the full 256-byte range.
3. Learning the substrate. Once you have written a SUB-encoded decoder by hand you will never again be confused about what shikata_ga_nai is doing, and you will read disassembly of other people’s loaders much faster.

Bringing this together with the rest of the OSED chain: the decoder sits between your overflow primitive (stack-buffer-overflow, seh-overwrite) and your final stager, often after a rop-chains sequence that disables dep-bypass on the buffer.

Classes, patterns, process

Single-byte XOR

The simplest viable decoder. Pick a key byte K that does not appear in the encoded blob; XOR every payload byte with K at build time; the stub XORs back at runtime.

Skeleton (x86, ~20 bytes):

    jmp short get_eip
decoder:
    pop esi              ; esi -> encoded payload
    xor ecx, ecx
    mov cl, payload_len
decode_loop:
    xor byte [esi], 0xAA ; key
    inc esi
    loop decode_loop
    jmp short payload
get_eip:
    call decoder
payload:
    db 0x?? , 0x?? , ...

Pros: tiny stub, trivial to write. Cons: any byte equal to K in the original becomes 0x00 after encoding, which kills string-copy bugs. You will frequently need to try several keys.

Rolling XOR

Use the previous encoded byte (or a running counter) as the next key. The decoder keeps state in a register:

    mov al, seed
decode_loop:
    xor [esi], al
    mov al, [esi]        ; next key = current decoded byte (or pre-XOR byte)
    inc esi
    loop decode_loop

This removes the “all K bytes collapse to null” problem and gives you a polymorphic-feeling output even though the stub is still tiny.

ADD / SUB additive

For bugs that ban most bytes but allow a clean alphanumeric range, encode by subtracting a per-byte delta and decode by adding it back. This is the basis of the “SUB encoder” trick where you build arbitrary 32-bit values by chaining SUB EAX, 0x???????? instructions using only printable bytes. OSED-style: zero EAX (via AND EAX, 0x554E4D4A; AND EAX, 0x2A313235), then three SUBs to reach the value you want, then PUSH EAX.

Alphanumeric-only (Skape)

When the bug accepts only [A-Za-z0-9] (classic Unicode-conversion bugs, some web parsers), you cannot use most x86 opcodes directly. The Skape “Writing IA32 Alphanumeric Shellcodes” paper (Phrack 57) gives a decoder stub built entirely from alphanumeric opcodes that decodes a payload also encoded with the alphanumeric alphabet. Practical reuse: msfvenom still implements x86/alpha_mixed and x86/alpha_upper; study its output as a reference, then write your own with a different stub shape so signatures miss.

Polymorphic decoders

“Polymorphic” here means: every build of the encoder emits a structurally different decoder stub that still produces the same decoded payload. Techniques:

• swap register pairs (use EDI as the pointer instead of ESI)
• swap loop construct (LOOP vs DEC ECX; JNZ)
• insert junk instructions that do not change state (XCHG EAX, EAX is NOP, but MOV EAX, EAX, PUSH/POP pairs, CLC work too)
• reorder commutative ops

This is what shikata_ga_nai does plus a self-modifying key-update. Worth implementing once for the experience; do not pretend it defeats modern static engines.

Decoder-stub size considerations

Your exploit budget is the buffer size minus the offset minus the ROP-to-RWX chain. A 4-byte ROP gadget here, a VirtualProtect call there, and suddenly you have ~80 bytes for stub+payload. A calc.exe payload is ~190 bytes; a reverse shell is ~340. So:

• prefer single-byte XOR (~20 bytes) over rolling-XOR (~30) over alphanumeric (~80+) unless the bug forces it
• consider using an egghunters stub + small staging decoder if the immediate buffer is tiny but a larger second buffer is reachable
• a pe-backdooring-and-code-caves approach can host the larger decoded payload in a code cave that the stub jumps to

Register-clobbering discipline

If you decode inside a callback, an exception handler, or anywhere the OS expects to resume normal execution, you must preserve the calling convention’s non-volatile registers. On x86 Windows that is EBX, EBP, ESI, EDI (and ESP of course). On x64 Windows that is RBX, RBP, RDI, RSI, R12-R15, XMM6-XMM15 plus the shadow space rules. Forgetting this is the most common reason “the decoder works in debugger, crashes under real flow.”

Practical recipe: PUSHAD at the top of the stub, POPAD before the jump into decoded payload. Costs 1 byte each, saves hours.

Self-modifying code under modern Windows

This is the part nobody on a 2015 tutorial warns you about.

• DEP / NX. Decoded bytes still live on a non-executable page unless the buffer was already RWX or you used a dep-bypass ROP chain (VirtualProtect, VirtualAlloc+copy, NtSetInformationProcess+MEM_EXECUTE_OPTION_DISABLE, or WriteProcessMemory to a remote RWX region). The stub itself must be in executable memory to run at all, so the typical pattern is: ROP -> VirtualProtect(buf, len, PAGE_EXECUTE_READWRITE) -> jump to stub -> stub decodes the payload that sits in the same now-RWX buffer.
• ACG (Arbitrary Code Guard). If the target process opts into ACG (Edge content processes, some Office surfaces), VirtualProtect with PAGE_EXECUTE_* is blocked outright; you cannot make a writable page executable. The decoder must live in already-executable memory (existing code cave) and decode into a separate already-executable region, or you must escape the ACG-protected process first.
• CFG (Control Flow Guard). CFG validates indirect call targets against a bitmap. A direct jmp into your decoded buffer from your own stub is not a CFG-protected indirect call, so CFG does not block the handoff; but if your stub uses a function-pointer-style indirect call to enter the payload, CFG will trip. Use a plain relative jmp short payload or call $+5; pop; jmp.
• CET / shadow stack. If the stub does call/ret tricks to get EIP, the shadow stack on CET-enabled CPUs will not match and the process dies. Prefer fnstenv or direct relative offsets for get-EIP.
• XFG. Adds type-signature checks to indirect calls; same mitigation as CFG above: avoid indirect calls to the decoded blob.

Comparison: why not just msfvenom shikata_ga_nai?

It still works as an encoder (the bytes-on-wire are random-looking) but it does not bypass modern AV/EDR because:

• the decoder-stub shape is fingerprinted in every commercial signature feed
• the FNSTENV get-EIP trick is itself a signature
• the decoder unpacks into memory that EDR is watching via NtProtectVirtualMemory hooks (edr-hooks-and-unhooking)

For OSED specifically, msfvenom payloads are allowed, but on real engagements you will mix a custom stub with a custom or custom-windows-shellcode-writing payload, then handle EDR separately via syscall-direct-and-indirect and amsi-bypass.

Embedding decoded payload as a resource

For full executables (not in-buffer exploits), drop the encoded payload into a .rsrc section of a benign-looking PE. The loader FindResource/LoadResource/LockResource, decodes in place into a fresh VirtualAlloc(PAGE_EXECUTE_READWRITE) region, and jumps. This is the pattern most “stage-0 loaders” use; pairs naturally with dll-side-loading and process-injection-techniques.

Defensive baseline

For blue team / detection engineering:

• Memory scans for short stubs ending in LOOP followed by high-entropy bytes catch the lazy case
• AMSI sees PowerShell-wrapped shellcode before decoding; amsi-bypass notes the gap, detection-engineering-pyramid-of-pain notes the durability
• Process-level mitigations to push for: ACG on browsers/Office, CFG/XFG on all binaries built in-house, CET where the CPU supports it
• ETW kernel provider Microsoft-Windows-Kernel-Memory surfaces VirtualProtect to RWX in interesting ways (etw-bypass explains the offensive countermove)

Workflow to study

1. Write the encoder in Python and the decoder in nasm. Assemble with nasm -f bin, hexdump, paste into a Python bytes literal.
2. Test the decoder standalone in a tiny C harness (VirtualAlloc(PAGE_EXECUTE_READWRITE), memcpy, cast-to-function-pointer, call) before plumbing it into an exploit.
3. Move to a debugger run inside a vulnerable target (the OSED labs are perfect for this). Single-step the decoder once to confirm the final jmp lands at payload: and the bytes there are clean.
4. Generate three keys for the encoder, pick the one whose encoded output has zero bad characters per bad-character-handling.
5. Drop into the real exploit chain after the ROP-to-RWX section of stack-bof-walkthrough-end-to-end or osed-custom-exploit-walkthrough.
6. For the OSED exam, time-box: 60 minutes to write a working stub from scratch is a realistic budget if you have done it twice before.

Related

• custom-windows-shellcode-writing
• osed-custom-exploit-walkthrough
• bad-character-handling
• egghunters
• pe-backdooring-and-code-caves
• stack-bof-walkthrough-end-to-end
• stack-buffer-overflow
• seh-overwrite
• rop-chains
• dep-bypass
• aslr-bypass
• safeseh-bypass
• mona-py
• syscall-direct-and-indirect
• edr-hooks-and-unhooking
• amsi-bypass
• process-injection-techniques
• detection-engineering-pyramid-of-pain

References

• Skape, “Writing IA32 Alphanumeric Shellcodes,” Phrack 57: https://phrack.org/issues/57/15
• Corelan, “Exploit writing tutorial part 4: From Exploit to Metasploit – the basics,” covers encoder/decoder concerns: https://www.corelan.be/index.php/2009/08/12/exploit-writing-tutorials-part-4-from-exploit-to-metasploit-the-basics/
• Offensive Security, OSED / EXP-301 syllabus (public outline): https://www.offsec.com/courses/exp-301/
• Microsoft, “Control Flow Guard” docs: https://learn.microsoft.com/en-us/windows/win32/secbp/control-flow-guard
• Microsoft, “Arbitrary Code Guard (ACG)” documentation: https://learn.microsoft.com/en-us/windows/security/threat-protection/microsoft-defender-atp/exploit-protection-reference
• Intel, “CET (Control-flow Enforcement Technology) Specification”: https://www.intel.com/content/www/us/en/developer/articles/technical/technical-look-control-flow-enforcement-technology.html