Ternlight – 7 MB embedding model that runs in browser (WASM)
Ternlight introduces a minuscule 7MB embedding model that runs entirely in the browser using WASM, offering fast, CPU-only semantic search without server calls. This groundbreaking client-side AI capability excites developers for its potential in on-device applications, while also sparking heated debates about browser resource consumption and security implications. The community grapples with the promise of powerful local inference versus the privacy and performance concerns of offloading complex AI directly to user machines.
The Lowdown
Ternlight unveils a remarkably compact embedding model, clocking in at just 7MB (or 5MB for its 'mini' variant), designed to run directly within web browsers using WebAssembly (WASM). This innovative approach allows for rapid, CPU-only semantic search and text embedding without requiring any API calls to a backend server. The project positions itself as a way to enable performant, on-device AI functionalities, with a demo showcasing real-time semantic search of React documentation.
- Miniature Model Size: The model is highly optimized, weighing only 7MB, with a 'mini' version at 5MB, making it feasible for browser-based deployment.
- Client-Side Execution: It runs entirely on the user's CPU within the browser, leveraging Rust compiled to WASM SIMD, eliminating the need for server-side processing.
- Fast Embeddings: Capable of generating text embeddings in approximately 5 milliseconds, enabling responsive, search-as-you-type experiences.
- Semantic Search Focus: While not an LLM, it produces 384-dimensional vectors from text, allowing for semantic similarity comparisons crucial for applications like semantic search, FAQ matching, and text clustering.
- Developer-Friendly: Packaged as a single npm package,
@ternlight/base, simplifying integration without model download steps. - Origins: Distilled from a small sentence encoder (MiniLM) using ternary quantization-aware training.
This technology paves the way for sophisticated, privacy-preserving AI features directly in web applications, shifting computational burdens and data processing away from centralized servers.
The Gossip
Browser-Based Brains: Potentials & Pitfalls
The discussion immediately zeroes in on the dual nature of running ML models directly in the browser. While excited by the prospect of powerful on-device features like performant semantic search without API dependencies, fostering innovation in client-side AI, commenters also raise significant concerns. These include potential resource consumption (CPU, memory), privacy implications of local inference, and the risk of malicious use (e.g., website hogging resources or running unwanted inferences), leading to provocative comparisons with 'disabling JavaScript' for security.
Performance & Practicalities
Commenters delve into the practicalities of implementing and optimizing such a browser-based model. There's curiosity about initial embedding times and the feasibility of pre-calculating embeddings server-side before shipping them to the client. The technical comparison to existing models like `gte-small` is also a key point, with the author providing insights into the distillation process. Furthermore, the discussion explores real-world applications, such as integrating the model into offline search engines or allowing users to index their own custom corpuses.
The Future of On-Device AI
The conversation naturally expands to the broader implications of this technology for the future of on-device machine learning. Some envision a paradigm shift, advocating for standardized browser APIs for AI tasks—similar to a 'W3C LLM API'—to enable rich, complex client-side functionalities. The reduced dependency on backend services and enhanced privacy through local data processing are highlighted as major benefits, drawing parallels to efforts like Apple's focus on on-device AI for core features.