Launch HN: Laminar (YC S24) – We trained a model to catch silent agent failures
Laminar, a YC S24 startup, has launched flow-1, an AI model designed to detect subtle, silent failures in AI agent workflows. This model, trained with reinforcement learning, boasts detection quality comparable to high-end models like GPT-6-sol but at a significantly lower cost. It addresses a critical pain point in AI development by making comprehensive agent observability affordable and actionable, promising to streamline the debugging and improvement of complex AI systems.
The Lowdown
Laminar, a new startup from Y Combinator's S24 batch, introduces flow-1, a novel AI model engineered to pinpoint hard-to-detect 'silent failures' within AI agent operations. These are errors where an agent might complete its actions successfully, but fails to achieve its intended goal, like clicking the wrong button or citing an incorrect document. Laminar claims flow-1 offers a cost-effective solution for comprehensive agent observability, a field often hampered by expensive analysis tools or insufficient data collection.
- The Problem: Traditional agent observability often falls short by merely collecting traces. Silent failures, where an agent's individual actions succeed but the overall task fails implicitly, are difficult to catch without understanding the agent's intent and inspecting deep into its LLM calls and tool results. Expensive, high-quality LLMs can do this, but their cost makes them impractical for analyzing every production trace.
- Laminar's Solution (flow-1): Laminar developed flow-1 by fine-tuning a smaller open-source model and then post-training it with reinforcement learning on thousands of synthetic, realistic agent workflows. The result is a model that matches the detection quality of advanced models like GPT-6-sol while being 23 times cheaper to run.
- How it Works (Signals): flow-1 operates within a harness called Signals. Signals treats an agent trace like a code repository, with each span (LLM call, tool call) as a file. It can process a compressed trace and user prompt, then intelligently grep and read relevant spans to identify issues. This system is flexible, allowing users to ask general questions like identifying user frustration or verifying checkout completion, and returns findings as customizable JSON 'signal events.'
- Beyond Detection: Recognizing that analyzing every trace generates too many individual events, Laminar automatically groups similar signal events into clusters. This allows developers to quickly identify recurring problems, understand their frequency, and drill down into specific instances. These clustered traces can then form evaluation datasets to test fixes and ensure they don't introduce new issues.
- Underlying Innovations: Laminar has also developed other core technologies, including 20x trace compression, a custom SQL query engine, and automatic task extraction, all aimed at enhancing the efficiency and utility of their observability platform.
- Availability: Laminar is open-source (Apache 2.0) and self-hostable, with a managed cloud service available.
By providing an affordable yet powerful way to analyze every agent trace, Laminar aims to close the loop between detecting, diagnosing, and fixing agent failures, thereby accelerating the development and reliability of AI agent systems.