Show HN: Product analytics (and evals) for agent sessions on your MCP
Armature introduces a specialized product analytics platform for AI agent sessions, addressing the critical blind spot where user-agent interactions occur within AI clients rather than traditional UIs. This tool captures user intent, agent thinking, and tool calls, enabling product teams to understand actual usage, identify issues, and refine their AI-powered services. It's a vital solution for any developer building on LLMs, providing the crucial visibility needed to improve agent-based product experiences and prevent costly failures.
The Lowdown
Armature, a YC-backed startup, introduces a novel product analytics solution specifically designed for AI agent sessions on Managed Calling Platforms (MCPs), effectively solving the challenge of understanding how users interact with AI agents when those interactions occur outside traditional UI environments. This "Show HN" highlights a tool crucial for developers and product teams building with LLMs, offering deep insights into otherwise opaque agent behaviors.
- Problem Solved: Traditional analytics tools are blind to interactions happening within AI clients (like Claude or ChatGPT), leaving product teams unaware of how users' agents utilize their services, their intent, or encountered frustrations. This contrasts with tools like LangSmith or Langfuse, which focus on agent builders rather than product teams understanding user-agent experiences.
- Core Functionality: Armature provides an SDK that wraps an MCP, reconstructing entire user-agent sessions, including user prompts, agent thought processes, and tool calls.
- Key Features: It identifies and ranks popular user use cases (even unsupported ones) by clustering sessions, pinpoints frequent issues and their root causes (even when APIs return 200 OK), and allows for detailed session replay to debug problems.
- Automated Evals: The platform is evolving to integrate automated evaluations, allowing for the identification of top workflows, recommendations for fixes, large-scale testing, and even the generation of pull requests to ship improvements.
- Real-World Impact: An example illustrates how Armature helped a marketing platform avoid a catastrophic bug where small LLMs hallucinated audience IDs, which could have led to sending campaigns to all contacts, preventing a disaster.
- Privacy & Performance: The solution prioritizes data safety with client-side PII redaction and ensures minimal performance degradation on MCPs (achieving success rates comparable to uninstrumented systems).
- Accessibility: It offers a self-serve setup taking minutes, a generous free tier (up to 1,000 sessions/month), and SDKs for Typescript, Python, and Go.
By providing unprecedented visibility into the agent experience, Armature empowers product teams to move beyond guesswork, ensuring their AI-powered products are robust, user-friendly, and truly effective.