System call instrumentation on Linux/x86‑64 using memory‑indirect calls, part I
This post delves into the esoteric art of Linux/x86-64 system call instrumentation, dissecting the challenges of patching two-byte syscall instructions with longer jumps. The author explores "instruction punning" and "zpoline" techniques, then embarks on a fascinating, albeit ultimately inconclusive, journey into memory-indirect lcall forms and x86 segmentation. It's a classic HN read for anyone who appreciates a meticulous, low-level systems programming deep dive, even when the path is fraught with "negative results" and unexpected architectural quirks.
The Lowdown
This article, part one of a series, explores advanced techniques for system call instrumentation on Linux/x86-64, focusing on the inherent difficulties of replacing compact system call instructions with longer jump sequences. The author's journey, spurred by the double-trap overhead of their libsystrap library's ud2-based approach, delves into existing research like instruction punning and zpolines before embarking on an original investigation into memory-indirect far calls (lcall) as a potential, albeit ultimately challenging, alternative.
- The Core Problem: System calls are typically two-byte instructions (e.g.,
0f 05), while direct jump instructions require at least five bytes, making direct patching difficult without overwriting subsequent code. - Existing Solutions Reviewed:
- Instruction Punning (Liteinst, E9Patch): Uses the bytes of subsequent instructions as part of a relative jump offset to a trampoline. This statistical approach relies on having enough