Kimi K3, and what we can still learn from the pelican benchmark
Moonshot AI just launched Kimi K3, a 2.8-trillion-parameter model, impressing with its benchmarks and a notable price tag. Simon Willison's signature 'pelican riding a bicycle' test, while no longer a primary benchmark, still offers practical insights into the model's cost and capabilities. Hacker News dissects K3's competitive standing, the swift progress of Chinese AI, and the ongoing debate about the pelican's place in model training data.
The Lowdown
Moonshot AI has unveiled Kimi K3, their latest and largest language model, boasting 2.8 trillion parameters and promising an open-weight release soon. This model positions itself as a strong contender against leading Western models, outperforming some while trailing others in official benchmarks. Simon Willison's analysis delves into K3's performance and cost using his traditional 'pelican riding a bicycle' SVG generation test.
The Gossip
Pelican's Enduring Eccentricity
The discussion debates the ongoing relevance and validity of the 'pelican riding a bicycle' benchmark. While some commenters question its utility, suggesting that models might be 'training for the benchmark' due to its widespread use or that it's overrepresented in training data, others defend its continued value. They argue it serves as a practical 'hello world' test for initial model interaction, assessing SVG generation, and providing quick insights into cost, speed, and basic reasoning, despite its limitations for frontier models.
K3's Cost and Caliber
Commenters rigorously scrutinize Kimi K3's pricing and overall value proposition. Some critics express concern over its relatively high cost, especially for tasks like the pelican test, and its competitiveness against established Western models, citing potential IP issues and challenges in running open-source models at scale. Conversely, proponents highlight K3's strong coding benchmarks at a competitive price point compared to some top-tier models, the promise of an open-weight release, and the broader benefit of increasing competition in the AI market. Many also point out that engineering focus on marginal cost differences can be 'silly' when compared to human labor.
Asian Ascent in AI
A prominent theme in the comments is the rapid advancement of Chinese AI labs and their increasing ability to rival or even surpass Western counterparts. Commenters express both surprise and curiosity regarding how Chinese labs are managing to train such massive models with potentially fewer compute resources. The discussion notes the shrinking gap in model capabilities and release timelines, with Kimi K3 being only months behind leading US models. Some also delve into the technical debate about the importance of sheer parameter count versus the efficiency of attention mechanisms in defining model 'intelligence.'
Token Tally & System Secrets
The peculiar observation of an 85-token overhead for a simple 'hi' prompt in Kimi K3 sparks a technical discussion around tokenization and hidden system prompts. Commenters speculate that this might be an injected 'reasoning-effort prompt' within the model's internal structure. This highlights the opaque nature of some LLM implementations and the challenges developers face in accurately calculating input costs, especially when internal mechanics are not transparent.