Gvisor on Raspbian
Running gVisor on a Raspberry Pi 5 hits a wall due to an obscure kernel configuration: the virtual address space size. This article meticulously explains why 39-bit VA in Raspbian breaks gVisor's userspace kernel emulation, contrasting it with Ubuntu's 48-bit support. It's a deep technical dive into a specific problem that illuminates broader concepts of kernel memory management and sandboxed container isolation on edge devices.
The Lowdown
The article delves into a peculiar issue encountered when attempting to run gVisor on a Raspberry Pi 5 using the default Raspbian operating system. It reveals that a single, often overlooked, kernel configuration option related to virtual address space size is the root cause of gVisor's failure. This technical deep dive explains the intricacies of virtual memory management on ARM64 architectures and why gVisor, a userspace kernel designed for sandboxed containers, demands a larger address space than Raspbian's default provides.
- gVisor acts as a userspace kernel, intercepting container syscalls to provide VM-like isolation with container-like efficiency, ideal for untrusted workloads.
- On 64-bit ARM, Linux uses either 39-bit (512 GB) or 48-bit (256 TB) virtual address space (VA). More VA bits imply more page table levels and increased overhead, but significantly more addressable space.
- gVisor requires 48-bit VA because, as a userspace process emulating a kernel, it must fit its own code, guest memory mappings, shadow page tables, and goroutine stacks within its single virtual address space.
- Unlike hardware-privileged hypervisors like KVM or Xen, gVisor operates entirely within userspace, making it highly susceptible to these VA constraints.
- Raspbian's kernel defaults to 39-bit VA for compatibility reasons, while Ubuntu's ARM64 kernel defaults to 48-bit VA, enabling gVisor to run without issues.
- The solution involves recompiling the Raspberry Pi kernel to enable
CONFIG_ARM64_VA_BITS_48, with detailed instructions for both on-device and cross-compilation. Understanding such low-level kernel configurations is crucial for deploying advanced cloud-native technologies like gVisor on edge devices such as the Raspberry Pi 5. This fix not only enables gVisor but also highlights the unique challenges and requirements of running userspace kernels compared to traditional virtualisation solutions, particularly on resource-constrained hardware.