Linux kernel syscall — source review

Linux kernel syscall — source review

TL;DR: Linux kernel syscalls are the user/kernel boundary. Audit risks: missing copy_from_user / copy_to_user (raw user-pointer deref), TOCTOU between check and use, integer overflow in length math, missing capable() / namespace checks, unchecked file-descriptor lookups, and reference-count errors in object lifecycle. This is the OSEE/Pwn2Own-adjacent Linux audit floor. Companion to linux-kernel-architecture and windows-driver-ioctl-audit.

Tooling for the read

  • Cross-reference: cscope, git grep, bootlin elixir, lxr, or VSCode + clangd.
  • Build with compile_commands.json so clangd resolves macros (kernel uses heavy macro work).
  • A local checkout of the kernel matching your target version.
1
2

make defconfig
make compile_commands.json

The syscall entry

1
2
3

SYSCALL_DEFINE3(my_op, int, fd, const void __user *, buf, size_t, count) {
    ...
}

The __user annotation marks user-space pointers; sparse / clang static analysis flags missing copies. Audit:

1

grep -rn '__user' fs/ kernel/ drivers/your-driver/

Bug class 1 — raw user-pointer deref

1
2
3
4

SYSCALL_DEFINE2(badsysc, void __user *, buf, size_t, len) {
    char tmp[64];
    memcpy(tmp, buf, len);          // BAD: buf is user pointer, len attacker-controlled
}

buf is a user-space address. memcpy will follow it as a kernel pointer — possibly reading kernel memory if the user supplied a kernel address; possibly crashing.

Correct:

1
2

if (copy_from_user(tmp, buf, min(len, sizeof(tmp))))
    return -EFAULT;

copy_from_user checks address validity and handles faults.

Bug class 2 — integer overflow

1
2
3

size_t total = count * sizeof(struct entry);
struct entry *e = kmalloc(total, GFP_KERNEL);
if (copy_from_user(e, ubuf, total)) ...

Same as user-mode pattern. Wrap-around total allocates small; the copy_from_user crashes with EFAULT — or succeeds against a smaller-than-expected region and corrupts adjacent slab.

Mitigation: kmalloc_array(count, sizeof(struct entry), GFP_KERNEL) checks the multiplication. Or check_mul_overflow.

Bug class 3 — signed/unsigned

1
2
3
4
5

SYSCALL_DEFINE2(bad, void __user *, p, int, sz) {
    if (sz > MAX) return -EINVAL;
    void *kbuf = kmalloc(sz, GFP_KERNEL);    // sz is int; negative passes check, becomes huge cast to size_t
    ...
}

Use size_t or bound sz >= 0.

Bug class 4 — TOCTOU between check and use

1
2

if (!user_can_access(file_path)) return -EACCES;
fd = open(file_path);              // path may have changed (symlink swap)

Inside the kernel this is rare in syscall code (most do the operation once) but appears in code that re-resolves names after permission decisions. Use file-descriptor-based APIs once the check has passed.

Bug class 5 — file-descriptor lookup races

1
2
3
4

struct fd f = fdget(fd);
if (!f.file) return -EBADF;
... // operations on f.file
fdput(f);

Patterns to flag:

  • fdget(fd) without matching fdput(f) — refcount leak.
  • Touching f.file after fdput — UAF.
  • Skipping fdget_pos for ops that need the position lock.

Bug class 6 — capable() in the wrong place

1
2
3
4

static int handle(...) {
    if (!capable(CAP_SYS_ADMIN)) return -EPERM;     // OK
    return do_thing(...);
}

vs.

1
2
3
4
5

static int handle(...) {
    int ret = do_thing(...);                         // BAD: side effect before check
    if (!capable(CAP_SYS_ADMIN)) return -EPERM;
    return ret;
}

The capability check must precede any side effect. In a user namespace, you often want ns_capable(user_ns, CAP_*), not the bare capable. Containers escalate via the difference.

Bug class 7 — namespace confusion

1
2
3

if (capable(CAP_NET_ADMIN)) {     // BAD: in a user namespace, this evaluates true for namespace owner
    do_root_thing();
}

In a user namespace, the namespace creator has all capabilities within the namespace. Operations that affect the host kernel should use ns_capable(&init_user_ns, ...). See user-namespace-attacks.

Bug class 8 — reference counting

kref, get_task_struct / put_task_struct, dget / dput. Any missing put is a leak; missing get + put-after-free is UAF.

1
2
3

get_task_struct(t);
... // critical section
put_task_struct(t);

Audit:

  • Every get paired with a put.
  • Cleanup paths (error returns) also put.
  • Locks held across allocations released before sleep paths.

Bug class 9 — sleeping under a spinlock

kmalloc with GFP_KERNEL can sleep. Under spin_lock, sleeping kills the system. Static checkers (sparse / smatch) flag; reviewers should too.

1
2

spin_lock(&l);
p = kmalloc(sz, GFP_KERNEL);       // BAD: can sleep

Use GFP_ATOMIC (smaller pool) or move alloc outside the lock.

Bug class 10 — uninitialised stack / kernel heap

1
2

struct ctx c;
copy_to_user(out, &c, sizeof(c));     // BAD: c is uninitialised → infoleak

Bound with memset(&c, 0, sizeof(c)). Same for kmalloc — prefer kzalloc when the buffer will be copied out.

Audit workflow

  1. Pick a subsystem (filesystem, net stack, ioctl path, BPF, etc.).
  2. List syscalls + ioctls exposed.
  3. For each, trace: user input → conversion → operation → return.
  4. Note every copy_from_user / get_user for length and target validation.
  5. Note every capable() for namespace correctness.
  6. Note every alloc/free for refcount/UAF.
  7. Note every length arithmetic for overflow.

Tools for finding bugs after the read

  • Smatch / sparse — static analysers tuned for kernel idioms.
  • CodeQL — increasingly popular for kernel audit; Anthropic / GP project ecosystems share queries.
  • syzkaller — coverage-guided fuzzer with syscall descriptions; finds bugs you missed at scale.
  • KASAN / UBSAN — runtime sanitisers for kernel; turn on in test builds.

OSEE relevance and beyond

Modern kernel exploitation requires bypassing SMEP, SMAP, KPTI, KASLR, and CFI. The audit phase is finding the bug; the exploitation phase is converting it into a primitive that survives mitigations. See linux-kernel-architecture and kernel-exploits-linux.

Source-audit checklist

  • Every user pointer accessed via copy_from_user/copy_to_user/get_user/put_user.
  • Length arithmetic uses overflow-checked helpers (check_mul_overflow, kmalloc_array).
  • Lengths held as size_t or bounded for int.
  • capable() precedes the side effect, namespace-aware variant where appropriate.
  • Every fd-get paired with put; refcount errors free of bugs.
  • Buffers copied out are zeroed before populate.
  • No sleeping under spinlocks.

References