Launch HN: Respan (YC W24) – Open-source Rust AI gateway built around plugins
Respan launches an open-source AI gateway built in Rust, emphasizing a plugin-centric architecture to address the complexity of existing LLM gateway solutions. Its core idea is a minimal kernel with every feature, from providers to security, implemented as a compile-time Rust plugin, offering unparalleled extensibility. This approach sparks discussion on the trade-offs of compile-time plugins, supply chain security, and developer experience for such a foundational AI infrastructure component.
The Lowdown
Respan, a YC W24 startup, introduces an open-source AI gateway written in Rust, aiming to solve the increasing complexity and inflexibility of existing LLM API gateways. The team previously managed a substantial wrapper around LiteLLM, encountering significant maintenance overhead and breakage during upgrades, which spurred the creation of Respan with a fundamentally different architectural philosophy.
- Problem Statement: Current LLM gateways often become monolithic as users add custom features, leading to complex, hard-to-maintain codebases and the need to build entirely new gateways for specific problems.
- Core Solution: Respan proposes a "small core" gateway, inspired by Nginx, where "everything is a plugin." This includes AI providers (OpenAI, Anthropic), communication protocols (chat, voice), data modalities (images, videos), and critical gateway features like smart routing, fallbacks, and PII redaction.
- Plugin Mechanism: Unlike many plugin systems, Respan's plugins are not dynamically loaded or interpreted; they are natively compiled Rust crates. This decision was made to allow plugins direct access to resources like databases and queues without the overhead or limitations of sandboxing (e.g., WASM).
- Functionality: The gateway parses requests into a shared internal structure, allowing for flexible translation between different client and provider APIs while preserving provider-specific fields when formats align. It supports tools, images, and reasoning protocols automatically.
- Open Source & Business Model: Respan is open-source under MIT/Apache-2.0, allowing users to run it locally. The company monetizes through a hosted platform that offers native observability, evaluations, and unified billing.
- Roadmap: The primary roadmap item is to foster an ecosystem of community-contributed plugins.
In essence, Respan aims to provide a highly customizable and maintainable foundation for AI infrastructure, allowing developers to precisely tailor their gateway needs through a modular, compiled plugin approach, all while offering a managed service for those who prefer it.
The Gossip
Plugin Paradigms: Compile-Time vs. Convenience
Users questioned Respan's choice of compile-time Rust crates for plugins, noting that "plugins" typically imply dynamic loading without recompilation. This sparked debate over the developer experience: while some appreciated the performance and deeper integration capabilities (like database access) that compiled plugins offer, others highlighted the friction of needing a full CI/build process for customization, contrasting it with the "just pull and run" expectation for gateways. The Respan team clarified their approach is more akin to GStreamer's statically compiled plugins and that official builds with default plugins are available.
Code & Credibility: AI-Assisted and Supply Chain Security
A significant point of discussion revolved around the use of AI for code generation. The founder confirmed Claude was used but emphasized that every line was reviewed. This led to broader concerns about code quality, especially for critical infrastructure components like auth systems, and the inherent supply chain risks associated with relying on third-party Rust crates, which can come from "random developers." Some argued that manual review isn't sufficient given human fallibility and the obscurity of upstream dependencies for many organizations.
Deployment Hurdles: Docker & Rust Adoption
The discussion touched upon practical deployment challenges. The absence of a readily available Docker image at launch, requiring users to build it themselves, was seen as a barrier to adoption. Furthermore, the requirement to compile custom plugins means teams need Rust expertise and a build pipeline, which many organizations might lack or be unwilling to invest in for a gateway. Commenters suggested a "batteries included" approach with a pre-built image, even if APIs are still evolving, to lower the entry barrier.