HN
Today

Not Everything Needs an LLM

This article argues that not every AI task requires a full-blown Large Language Model, advocating for 'decision models' as a cheaper, faster, and more explicit alternative for simpler classifications. Hacker News finds this practical approach refreshing, particularly as it addresses the growing costs and computational overhead of over-relying on LLMs for every problem. It sparks discussion on proper tool selection, the true nature of these 'decision models,' and the ever-present concerns around data processing.

66
Score
23
Comments
#21
Highest Rank
2h
on Front Page
First Seen
Nov 17, 10:00 PM
Last Seen
Nov 17, 11:00 PM
Rank Over Time
2123

The Lowdown

In a world increasingly dominated by Large Language Models (LLMs) that love to wax poetic and charge premium rates, PostHog makes the case for "decision models" – a more focused AI tool for straightforward tasks. These models offer structured, typed answers with explicit probability scores, making their uncertainty transparent, unlike the often overconfident prose of LLMs.

Here's what PostHog has learned about using decision models:

  • What They Are: Decision models provide structured answers (Yes/No, Choice, Score) with a confidence probability (e.g., 0.99 for 99% certainty).
  • Key Advantages: They are significantly cheaper (up to 444x) and faster (up to 193x) than frontier LLMs.
  • Practical Applications: PostHog uses them to:
    • Gate LLM Calls: Filter out 98.7% of summaries that don't need PII scrubbing, saving expensive LLM processing.
    • Route Models: Direct user queries to the cheapest capable LLM based on task complexity.
    • Rerank Search Results: Improve search relevance by evaluating how well initial embedding-based results match a query.
    • Suggest Charts: Recommend optimal visualizations for SQL query results based on data patterns.
  • Threshold Management: Set confidence thresholds based on the cost of false positives vs. false negatives.
  • Limitations: Decision models excel at single-pass judgments; multi-step reasoning is still best left to LLMs.
  • Future Trends: Hybrid models like PostHog's 'Jeeves' are emerging, which can reason before deciding when unsure.

Ultimately, decision models serve as a crucial middle ground: they handle tasks too fuzzy for simple regex but too simple or cost-sensitive for an LLM, ensuring the right tool is used for the right job in the evolving AI landscape.

The Gossip

Decision Model Definitions & Delineations

Many commenters questioned the novelty and definition of 'decision models,' asking if they are merely specialized LLMs, BERT-like models, or just a new marketing term for simpler classifiers. Some argued that the distinction is pedantic, while others felt it represented a return to basic machine learning principles that have been overshadowed by LLM hype. The consensus leans towards decision models being a more constrained, discriminative form of AI, potentially even a fine-tuned LLM, rather than a fundamentally new paradigm.

LLM Leverage & Limits

The broader discussion revolved around the appropriate use cases for LLMs. Many agreed that LLMs are often over-engineered solutions for simpler problems, leading to unnecessary cost and complexity. However, a contrarian viewpoint emerged, suggesting that for teams lacking the expertise to build robust evaluation systems (evals), using a more expensive LLM might ironically be the 'safer' choice to avoid poorly implemented custom logic. This highlighted the trade-off between cost, complexity, and the difficulty of accurate model evaluation.

Data Dilemmas & Privacy Practices

A significant concern was raised regarding PostHog's use of these models on customer data, particularly for PII scrubbing. Commenters expressed unease about the collection and processing of user session data, even with anonymization efforts, labeling it as 'surveillanceware.' The discussion touched upon the inherent risks and discomfort associated with companies handling sensitive customer information, especially when involving external AI services like Anthropic and OpenAI for processing.

Eval-uation Exasperation

A tangent discussion explored the significant challenges of creating effective 'evals' (evaluation metrics or systems) for AI models, particularly LLMs. Commenters noted the difficulty in ensuring consistency, inter-model agreement, and accuracy against human judgment when the evaluation tool itself might be another probabilistic LLM. This complexity fuels the argument that many teams struggle to build reliable evals, making simpler, more expensive LLM solutions sometimes preferable as a default.