The Scourge of x86 Emulation

The article from FEX-Emu, an x86 emulator for ARM, meticulously details the inherent difficulties and performance bottlenecks encountered when attempting to perfectly emulate the x86 Total Store Ordering (TSO) memory model on ARM-based systems. This challenge stems from the fundamental differences in how the two architectures handle memory consistency and ordering, with x86 being highly strict and ARM offering a more relaxed, performance-optimized model.

• x86-TSO vs. ARM Weak Ordering: x86's TSO model guarantees strong memory coherency, where memory stores are immediately visible to all processors and loads see all prior stores. ARM, conversely, uses a weaker model that allows for significant hardware optimization by not strictly guaranteeing immediate visibility, relying instead on explicit load-acquire and store-release instructions for ordering.
• Early Emulation Challenges (ARMv8.0-a): Initially, FEX had to map all x86 loads and stores to ARM's load-acquire and store-release, which proved exceedingly costly as these instructions were not designed for such frequent use, severely impacting performance.
• Architectural Improvements (FEAT_LRCPC): Newer ARM extensions like FEAT_LRCPC introduced a RCpc (Release Consistency processor consistent) memory model, specifically designed to aid x86 emulation. These extensions significantly improve aligned memory access performance, with some platforms (like Oryon-3) matching native performance.
• Apple's Hardware Solution: Apple's M1 processors take a unique approach by implementing a hardware-level x86-TSO mode. When enabled, native ARM load/store instructions behave with x86-TSO semantics, offering dramatically superior performance for emulation by eliminating many software-based workarounds.
• Unaligned Accesses and Split-Locks: A major hurdle is x86's tolerance for unaligned memory accesses and "split-locks" – atomic operations that cross cacheline boundaries. ARM architectures typically fault on unaligned accesses or incur massive performance penalties (kernel/userspace context switches) due to lack of direct hardware support.
• Atomic Operations (RMW): While ARMv8.1-a introduced instructions mirroring x86's Read-Modify-Write (RMW) atomics, unaligned versions still suffer. Qualcomm's Oryon-3 makes strides with "coherent cachelines," matching x86 performance for atomics within a cacheline, but not across.
• Kernel Patch Intervention: The Valve Steam Frame (Cortex-X4) employs a Linux kernel patch that allows the kernel to handle unaligned atomics more efficiently, bypassing FEX's slow userspace dance and significantly improving performance.
• The Split-Lock Correctness Dilemma: Emulating x86 split-locks correctly and performantly on ARM without data tearing is currently unachievable without further hardware support. The article proposes a specialized 128-bit CASP instruction as a potential future solution.
• Uncached (Write-Combine) Memory: Another significant bottleneck is the performance of "uncached" (write-combine) memory, particularly for stores, which can be hundreds of times slower than x86. This impacts games, especially those relying on PCIe GPUs, and necessitates workarounds like forcing cached buffers on UMA systems.

In conclusion, the journey to perfectly emulate x86 on ARM is a continuous uphill battle against deeply ingrained architectural differences. While significant progress has been made with dedicated ARM extensions and innovative hardware solutions like Apple's TSO mode, critical edge cases like unaligned atomics and write-combine memory performance remain challenging. The article emphasizes the ongoing commitment from vendors and the hope that future architectural revisions will further close the compatibility and performance gap, ensuring the longevity of x86 software on ARM platforms.