How We Made IPFS Content Publishing 10x Faster
IPFS has made a significant leap, announcing 'Optimistic Provide' which slashes content publishing times by over 90%, from ~15 seconds to under a second. This deep technical dive details how clever statistical heuristics, network estimation, and early returns overcome traditional DHT bottlenecks. The upgrade, now default in Kubo 0.39.0, addresses a longstanding performance issue, making IPFS more viable for real-time, delay-sensitive applications and exciting decentralized tech enthusiasts on HN.
The Lowdown
Publishing content on Distributed Hash Tables (DHTs), like IPFS's Amino DHT, has historically been a slow process, often taking dozens of seconds due to the network's large size and frequent node churn. The ProbeLab team identified this bottleneck and developed 'Optimistic Provide,' an optimization now integrated into IPFS's Kubo 0.39.0, which drastically accelerates content publication.
- The Problem: Traditional IPFS 'provide' operations involved a rigid two-phase process: a 'DHT Walk' to find the 20 closest peers to a content identifier (CID) and a 'Follow-Up' phase to push the record to them. The DHT Walk was particularly slow due to strict termination conditions, where the system would wait for unreachable peers and backtrack, leading to median latencies of around 20 seconds.
- The Optimistic Provide Solution: This optimization consists of three core ideas:
- Immediate Storage: Records are stored immediately with peers during the DHT walk if they are statistically likely to be among the 20 closest network-wide peers.
- Predictive Termination: The DHT walk terminates early, once there is a 90% confidence that the set of 20 closest peers has been discovered, instead of waiting for specific confirmations.
- Early Return: Control is returned to the user as soon as a sufficient subset (e.g., 15 out of 20) of peers confirm storage, with the remaining requests completing asynchronously in the background.
- Key Enabling Technologies:
- Network Size Estimation: A lightweight, bias-corrected algorithm estimates the global network size by piggybacking on routing table refresh mechanisms, using statistical insights from Beta distributions and order statistics, incurring no additional networking overhead.
- Probabilistic Decisions: The network size estimate allows the system to make probabilistic decisions about peer closeness and termination conditions, avoiding delays caused by unresponsive nodes.
- Impact and Results: Optimistic Provide has reduced IPFS content upload latency from an average of approximately 15 seconds to about 0.7 seconds, a more than 10x speedup. It also maintains record availability and reduces networking overhead by over 40%. This dramatically improves the user experience, enabling near real-time content publishing.
- Limitations and Future Work: The optimization's effectiveness relies on accurate network size estimation, which can be affected by undialable peers. Future improvements include filtering these peers, deeper integration with the reprovide sweep, and persisting network size estimates to disk for faster cold starts.
Optimistic Provide, coupled with the Reprovide Sweep, fundamentally redefines IPFS content publishing. It resolves a significant performance bottleneck in Kademlia-based DHTs, making IPFS a more practical solution for delay-sensitive applications and marking a substantial advancement for the decentralized web. Users are encouraged to update their Kubo deployments to v0.39.0 or later to benefit from these improvements.