Show HN: We built open OpenRouter that turns usage into a better model
Experiential presents an open-source, Rust-native gateway designed to unify management of diverse LLMs with an OpenAI-compatible API, boasting sub-millisecond latency. Its unique selling proposition is the ability to automatically optimize model selection based on real-world production traffic, leading to better cost-quality trade-offs. This project appeals to Hacker News for its technical sophistication, open-source model, and tackling a critical infrastructure challenge in the rapidly evolving AI landscape.
The Lowdown
Experiential is an open-source, Rust-native model gateway that provides a unified control plane for managing a variety of large language models, including self-hosted, frontier, and custom models. It offers an OpenAI-compatible API, streamlining integration and providing a single point of access for developers.
- Unified Management: Consolidates access to diverse LLMs (OpenAI, Anthropic, Gemini, Azure, Bedrock, etc.) and local models through one API.
- High Performance: Engineered with Rust for concurrency, it adds negligible latency (under 1ms for BYOK requests, under 2ms with Experiential-supplied keys).
- Intelligent Routing: Utilizes OpenTelemetry traces from production traffic to identify representative tasks. It then uses text world models, LLM judges, and nearest neighbor classifiers to simulate and determine the optimal model for each request, aiming for an improved cost-quality Pareto curve.
- Optimization Features: The system can suggest cache hit optimizations, new model recommendations, and facilitate model training based on usage data.
- Open Source & Fair Pricing: The project is open source, takes no markup on usage, and allows users to mix local models with a hosted marketplace.
- Flexible Deployment: Users can deploy it on their own infrastructure or use a hosted version with zero markup.
By leveraging real-world usage data to intelligently route and optimize model calls, Experiential aims to provide a performant, cost-effective, and flexible solution for managing complex LLM workflows.
The Gossip
Gateway Gratification & Key Contrasts
Commenters expressed appreciation for the project's open-source nature and the absence of markup. Several users inquired about its similarities and differentiators compared to existing solutions like LiteLLM. The creators highlighted that the main distinction lies in its unique router and model optimization capabilities, which derive insights from production traffic, along with offering a hosted marketplace.
Caching Quandaries & Switching Strategies
A significant point of discussion revolved around the cost implications of model switching, particularly concerning the caching of input tokens. Users worried that frequent model changes could significantly increase costs. The developers clarified that the system is designed to 'rarely switch' models, or to do so primarily at task boundaries, often concluding that a single model is optimal for a given task, thus mitigating potential cost spikes.
Rust's Robustness & Performance Prowess
The impressive claim of sub-millisecond latency for the gateway garnered positive attention. There was also curiosity regarding the technical foundation, specifically whether the project was initially developed in Python before being rewritten in Rust. The creator confirmed this, underscoring a deliberate choice to prioritize performance and concurrency through Rust.