Syscall Latency Tracer

Spring 2026

A Linux kernel module that measures system call latency under varying load, built as part of an OS course project.

Overview

This was an OS course project where I built a Linux kernel module from scratch to trace system call latency. The module hooks into four syscalls (read, mmap, futex, openat) using kretprobes, records nanosecond-precision timestamps on entry and exit, and maintains per-CPU histograms to avoid lock contention. A Python test harness automated the full workflow: loading the module, spawning custom generator programs to drive syscall traffic, running the kernel stress tool at three load levels, collecting stats through /proc/syscall-trace, and producing comparison graphs. The whole thing ran on a real Linux system with kernel headers, not in a VM.

What I Worked On

Wrote a kernel module in C using kretprobes to hook read, mmap, futex, and openat
Recorded nanosecond entry/exit timestamps and computed per-call latency
Maintained per-CPU statistics (count, total, max, 16-bucket histogram) to avoid spinlock pressure
Exposed aggregated results through a /proc/syscall-trace interface with custom read/write handlers
Built custom C generator programs to produce predictable syscall workloads for each target
Wrote a Python test harness to automate module loading, test runs, stat collection, and graph generation
Tested across three stress levels (baseline, medium, high) using the stress tool and psutil monitoring
Analyzed latency distributions and produced avg/max comparison and histogram plots with matplotlib

Results

Average and maximum latency for mmap across baseline, medium, and high load. Average latency nearly doubles from baseline to high stress, and the maximum spikes sharply under load.

Latency histograms for mmap at each stress level. Under high load the distribution spreads into the microsecond range, showing how contention pushes more calls into longer tail latencies.

Key Takeaways

Linux kernel module lifecycle (init, cleanup, insmod, rmmod)
kprobes and kretprobes for non-invasive kernel instrumentation
Per-CPU data structures and why they matter for performance
procfs interface design (seq_file, write handlers)
How syscall latency shifts under CPU, memory, and I/O pressure
Debugging kernel code with dmesg and careful logging

View repository →

Full write-up and notes available here.