Show HN: Threadprocs – executables sharing one address space (0-copy pointers)

This experimental project, 'Threadprocs', blurs the lines between processes and threads by allowing multiple executables to run in a single shared virtual address space. It enables zero-copy pointer-based data sharing, offering a theoretical performance boost by circumventing traditional IPC overhead. Hacker News found it intriguing as a deep dive into operating system boundaries, sparking debate on the trade-offs between performance, safety, and system design philosophy.

Score

Comments

Highest Rank

on Front Page

First Seen

Mar 23, 4:00 PM

Last Seen

Mar 23, 7:00 PM

Rank Over Time

The Lowdown

Threadprocs is an experimental project by jer-irl that proposes a novel approach to inter-process communication by running multiple independent executables, termed "threadprocs," within a single shared virtual address space. This unique setup allows for the direct sharing of pointer-based data structures, eliminating the need for serialization or copying often associated with traditional inter-process communication (IPC) or shared memory systems.

Core Concept: Threadprocs combine the benefits of process-like independence (separate binaries, globals, and lifetimes) with the performance of thread-like shared memory, where pointers remain valid across these pseudo-processes.
Mechanism: The implementation relies on careful manipulation of virtual address space layout, including directing ASLR (Address Space Layout Randomization) at load time, and providing a user-space equivalent of exec().
Key Distinction: Unlike traditional processes, pointers are directly valid across threadprocs. Unlike threads, each threadproc maintains a degree of isolation, behaving like a standalone process with its own main() function.
Technical Challenges: The project highlights significant limitations. Each threadproc has its own C standard library (libc) instance, meaning memory allocated by one cannot be reliably freed by another. It requires position-independent code, has restrictions with brk() and mmap(MAP_FIXED), and lacks support for debugging tools like ptrace().
Author's Perspective: The creator acknowledges that while an interesting exploration of OS boundaries, it's "not clear this is a practical model" for serious software, emphasizing the safety and complexity trade-offs involved.

Ultimately, Threadprocs serves as a proof-of-concept exploring how to relax traditional process memory boundaries, offering potential performance gains for highly trusted, cooperating components, but at the cost of the robust isolation provided by modern operating systems.

The Gossip

Prudent Process Protection vs. Performance Payoffs

A central discussion revolved around the fundamental trade-off between the isolation and security of traditional process boundaries and the performance gains offered by Threadprocs' shared address space. Many commenters voiced concerns about memory corruption and security vulnerabilities, drawing parallels to less protected, older operating systems. The author acknowledged these significant trade-offs, positioning the project as an experiment rather than a general-purpose solution, noting that "nothing" prevents one threadproc from corrupting another's memory.

Echoes of Architectures Past and Present

Users connected Threadprocs to various historical and contemporary systems that explored similar concepts of shared address spaces or process-like isolation within a single process. Examples included the Opal OS from the 1990s, AmigaOS, early Windows, and even Python's subinterpreters. One commenter detailed their own research into "Inter-Process Remote Execution" (IPRE), which also aims for low-latency IPC through merged address spaces, highlighting ongoing interest in these advanced architectural patterns.

The Python GIL and Pointer Predicaments

The potential for Threadprocs to circumvent Python's Global Interpreter Lock (GIL) was quickly raised as a possible performance application. However, the discussion swiftly moved to the complexities of Python's object model, questioning whether shared memory alone could truly "defeat the GIL" without addressing how Python objects handle parallel access. The author engaged, suggesting potential handoff mechanisms to C/C++ libraries as a possible path.