HN
Today

The NX bit is not just about security

Sonya recounts a months-long debugging odyssey on an ARM64 bare-metal hypervisor for postmarketOS, where enabling a CTR_EL0 intercept caused random phone lock-ups. The complex saga involved systematically eliminating hypotheses like MRS emulation errors, out-of-spec hardware, and kernel panics, ultimately narrowing the issue to a subtle difference between static and dynamic branch calls. The breakthrough revealed that the Non-Executable (NX) bit, typically a security feature, unexpectedly prevented speculative instruction fetches from MMIO regions, solving a baffling problem and showcasing a deep architectural quirk.

10
Score
0
Comments
#2
Highest Rank
8h
on Front Page
First Seen
Sep 7, 2:00 AM
Last Seen
Sep 7, 9:00 AM
Rank Over Time
523511242120

The Lowdown

This story details Sonya's arduous, months-long debugging journey while developing a bare-metal hypervisor on ARM64 for postmarketOS. The core problem manifested as random phone lock-ups immediately after enabling a CTR_EL0 intercept, a feature crucial for the hypervisor's function. The experience illustrates the intricate challenges of low-level ARM system development and the unexpected behaviors one can encounter with hardware and memory management.

  • Initial investigations focused on verifying the correctness of MRS/MSR instruction emulation and stack handling within the exception handler, both of which appeared sound.
  • A key early hurdle was ARM's lack of Icache/Dcache coherence, which meant dynamically sorting an array of executable instructions at runtime led to issues; moving this to a build-time step resolved it.
  • Hypotheses about out-of-spec hardware or kernel panics were explored through extensive logging and analyzing crash logs, but provided no definitive answers.
  • Manual patching of ctr_el0 instructions in the kernel and rewriting specific handlers in assembly helped isolate the problem to the C handler's execution.
  • Binary bisecting of the C code revealed the subtle but critical difference between a static bl get_ctr_el0 call and a dynamic blr x0 call; the dynamic call, relying on branch prediction, was the culprit.
  • The root cause was uncovered: speculative instruction fetches from the 0x0 address, which in the 1:1 memory mapping corresponded to an unmapped bootrom region.
  • The ultimate solution, confirmed by ARM documentation, was to mark the relevant MMIO pages as non-executable, preventing these speculative instruction fetches.

The debugging saga concludes with the surprising revelation that the Non-Executable (NX) bit, often seen purely as a security mechanism to prevent code execution from data segments, plays an equally vital role on ARM in preventing microarchitectural speculative instruction fetches from device memory. This highlights a nuanced aspect of ARM architecture where memory attributes serve functions beyond their typical security implications.