Linux kernel architecture

Linux kernel architecture

TL;DR: The kernel is one monolithic address space entered through syscalls, ioctls, netlink, and eBPF; bugs in any of those edges yield ring-0 primitives that exploit dev must understand structurally before fuzzing.

What it is

A short orientation map for kernel researchers: how user→kernel transitions work on x86_64, what subsystems present the largest attack surface, and which mitigations gate exploitation. This is scaffolding for the kernel-bug notes — not a replacement for source-tree reading.

Preconditions / where it applies

  • Modern x86_64 Linux (5.x/6.x); ARM64 has analogous structure with a different syscall ABI.
  • Researcher access: source tree (kernel.org), ftrace, kprobes, KASAN/KFENCE/KMSAN sanitisers, syzkaller for coverage feedback.

Technique

Entry surfaces worth mapping first:

  • syscallsarch/x86/entry/syscalls/syscall_64.tbl enumerates them; handlers live across fs/, net/, kernel/, mm/.
  • ioctl — character/block device drivers; unlocked_ioctl handler is the routing function; lookup with git grep "\.unlocked_ioctl".
  • netlinkAF_NETLINK sockets; genetlink families parse attributes per policy table — historically rich in bugs.
  • eBPF — verifier in kernel/bpf/verifier.c plus JIT; many CVEs (e.g. CVE-2021-3490, CVE-2022-23222) abuse verifier confusion.
  • filesystemsio_uring, fuse, overlayfs, ntfs3 are recent high-yield surfaces.
  • drm/gpu, USB, Bluetooth (HCI), WiFi (mac80211) — peripheral parsers with attacker-controlled buffers.

Memory model:

  • Per-CPU stacks (~16 KiB), separate IRQ stack; kernel heap is SLAB/SLUB with per-CPU freelists; allocations bucketed by size into kmalloc-N caches.
  • vmalloc for large non-contiguous; module_alloc lands within ±2 GiB of kernel text for short RIP-relative calls.

Mitigations gating exploitation:

  • KASLR — text + module base randomised; defeat with leak primitives.
  • SMEP/SMAP — see smep-smap-overview for the Windows mirror; Linux uses CR4 bits identically.
  • KPTI — page-table isolation against Meltdown; affects syscall entry path.
  • CFICONFIG_CFI_CLANG enforces type-tagged indirect calls (kernel CFI similar in spirit to xfg).
  • STACKLEAK, KFENCE, RANDSTRUCT, slab freelist hardening — kill classic spray techniques and uninit reads.

Privilege escalation pattern: corrupt cred struct, overwrite modprobe_path, hijack core_pattern, or call commit_creds(prepare_kernel_cred(NULL)) from a controlled gadget.

Detection and defence

  • Production hardening: lockdown, ftrace restrictions, unprivileged BPF off, KASLR + KPTI, module signing, SELinux/AppArmor profiles.
  • Detection: auditd on init_module, bpf(), suspicious setresuid(0,0,0) from non-root parents; kernel oops with KASAN messages indicate found bugs.
  • Vendor kernels add Yama ptrace scoping, restricted dmesg, kptr_restrict.

References

See also: ret2csu, ret2libc, heap-exploitation-linux.