HN
Today

Highest Random Weight in Elixir

This post dives deep into Rendezvous Hashing (HRW) as a stateless alternative to Discord's ExHashRing for consistent hashing in Elixir. It showcases HRW's simplicity and comparable performance for smaller node counts, then introduces an optimized O(log n) 'skeleton' approach. The article culminates with an analysis of distribution efficiency and the announcement of a new hrw Elixir library, providing practical insights for distributed system architects.

17
Score
0
Comments
#8
Highest Rank
6h
on Front Page
First Seen
May 23, 3:00 PM
Last Seen
May 23, 8:00 PM
Rank Over Time
899132222

The Lowdown

The article explores Rendezvous Hashing (HRW) as a simpler, stateless alternative to the widely used ExHashRing for consistent hashing in Elixir. While ExHashRing is robust, it requires managing stateful processes, prompting the author to investigate HRW's pure functional approach.

  • HRW Basics: Rendezvous Hashing (also known as Highest Random Weight or HRW) is introduced as a more general and simpler alternative to consistent hashing. It allows nodes to determine ownership of a key without stateful processes by calculating a score for each node and selecting the highest.
  • Performance Trade-offs: The basic HRW algorithm has O(n) complexity, making it less performant than ExHashRing for very large numbers of nodes. However, for typical use cases with fewer than ~14 nodes, its performance is nearly identical.
  • HRW Skeleton Optimization: To address scalability, the article describes an O(log n) hierarchical 'skeleton' approach for HRW. This method significantly improves performance for larger node counts, making it comparable to ExHashRing, though it reintroduces a form of state (the skeleton struct).
  • Distribution Efficiency: Benchmarks demonstrate that HRW, even with simple :erlang.phash2, achieves excellent key distribution across nodes. Notably, it often outperforms ExHashRing's default settings in distribution consistency, especially as node counts increase.
  • New Library: The author announces the release of the hrw Elixir library, which includes the discussed HRW implementations, as well as advanced strategies like HRW.Weighted for heterogeneous clusters and HRW.Bounded for specific key distributions.

In conclusion, HRW offers a compelling, often simpler, and performant option for consistent hashing in Elixir, particularly when considering its stateless nature for smaller clusters or the skeleton optimization for larger ones, alongside its strong distribution characteristics.