I Built a 500k-Domain Search Engine for Makers in a Weekend for $10
A developer, fed up with corporate search results, spent a weekend building a personal search engine for "makers" that indexed over half a million domains for about $10. Leveraging a small local LLM on a rented GPU, they overcame numerous challenges like SEO spam and quality control. This impressive feat showcases the power of focused hacking and strategic use of AI for personal projects.
The Lowdown
In a testament to weekend hacking, a developer created a highly personalized search engine called 'Marlin' that indexes over 560,000 domains for niche content like portfolios and zines, all for around $10. Frustrated by the corporate and SEO-dominated landscape of mainstream search engines, the author set out to build a cleaner, more relevant alternative. The project, completed over a single weekend, highlights efficient resource management and clever problem-solving using modern AI tools.
- The core idea was to build a personal search index using a crawler that only focuses on homepages, and a small local language model (Gemma 4B) to summarize each site with a name, summary, category, and tags.
- The architecture involved four processes: a fetcher, a worker (for LLM summarization), a steward (for quality control/blocking), and a simple API/UI. Data was stored in Postgres.
- Key challenges included filtering out corporate spam, handling model errors, preventing domain-specific platforms (like Tumblr) from dominating the index, and managing various "link farms" and low-quality sites.
- Solutions involved a weighted crawl queue to prioritize desired content, strict rules for model processing based on page content, subdomain caps for large platforms, and an AI-powered "steward" process to automatically block problematic domains.
- The author rented a cloud GPU for efficient summarization, optimizing the setup to achieve around 600 summaries per minute at a cost of $0.35 per hour, contributing significantly to the low overall expense.
- By the end of the weekend, the index contained over half a million sites, evolving from an initial bias towards blogs to a diverse collection including non-profits and community sites.
- Future builders are advised to manage categories and tags carefully (the model's freestyle approach created 671 categories and 121,000 tags) and to embrace an iterative, data-driven approach to crawl steering.
- The project's code will be open-sourced, enabling others to build their own personalized search indexes.
This weekend project demonstrates that powerful, bespoke tools can be created quickly and cheaply by combining focused intent with strategic application of modern technologies like local LLMs and cloud GPUs. It serves as an inspiring blueprint for anyone looking to reclaim agency over their digital information consumption and build tailored solutions to common frustrations.