The Fil-C Optimized Calling Convention

Fil-C presents a sophisticated optimized calling convention aiming to enforce memory safety within C programs, even under deliberately "adversarial" conditions like misusing function pointers or mismatched signatures. The core challenge addressed is how to implement these robust safety checks without introducing significant performance overhead for the vast majority of legitimate function calls. The solution involves a multi-layered approach that prioritizes efficiency in the common case while ensuring safety when violations occur.

• Adversarial C Safety: Fil-C's calling convention guarantees memory safety against common C misbehaviors, such as calling function pointers with incorrect signatures, passing wrong argument types or counts, or importing/exporting symbols with type mismatches. It either panics or defines safe behavior for these scenarios.
• Optimized Common Paths: For typical, well-behaved function calls, Fil-C achieves near-native performance by passing arguments and return values in registers, minimizing overhead.
• Generic Calling Convention: A robust, yet less efficient, generic calling convention serves as a universal fallback. It involves rigorous capability checks, temporary buffer allocation for arguments and return values, and "getter" calls for symbol resolution.
• Register Calling Convention via Arithmetic Signatures: This optimization uses a 64-bit arithmetic encoding to represent function signatures. If the caller's expected signature matches the callee's encoded signature, a highly efficient fast_entrypoint is used. Otherwise, specialized thunks translate between calling conventions, leveraging the generic path. This yielded a >1% speed-up.
• Direct Call Optimization with ELF Tricks: For direct function calls, Fil-C aims to completely bypass capability checks and symbol resolution by calling a signature-mangled implementation directly. This requires intricate ELF symbol manipulation, including weak/strong symbol rules and COMDAT handling, to ensure the correct implementation is called without infinite loops or NULL dereferences. This provided another >1% speed-up.
• Performance and Safety Synergy: The project demonstrates that comprehensive memory safety for C can be achieved without major performance compromises, by intelligently deferring or eliminating checks in provably safe scenarios through deep compiler and linker integration. Ultimately, Fil-C's optimized calling convention transforms inefficient, check-heavy function calls into highly efficient, direct register-based transfers for the common case. This is accomplished through innovative arithmetic signature encoding and a sophisticated dance with ELF's symbol resolution mechanisms, proving that memory safety in C can coexist with performance.