Show HN: misa77 - a codec that decodes 2x faster than LZ4 (at better ratios)
A new codec, misa77, claims to decompress data 2x faster than LZ4, often with better ratios, by leveraging microarchitectural optimizations for modern CPUs. This 'write-once, read-many' codec deliberately trades slow compression for blazing-fast decompression, positioning itself on the Pareto frontier for speedy codecs. Hacker News users are dissecting its technical design, questioning its robustness, and eagerly testing its real-world performance across diverse data types and architectures.
The Lowdown
misa77 is a newly developed LZ-based compression codec designed for "write-once, read-many" scenarios, prioritizing extremely high decompression throughput over compression speed. Its creator claims it can decompress data 1.5 to 3 times faster than the popular LZ4 codec, often achieving better compression ratios. The core innovation lies in reducing branches and creating a decompression format highly optimized for out-of-order CPU cores.
- Decompression Speed: Achieves state-of-the-art single-threaded decompression throughput, significantly outperforming LZ4 on x86-64 and ARM.
- Compression Ratios: Offers decent ratios comparable to LZ4 at high effort levels, particularly excelling with highly compressible files.
- Trade-offs: Compression is notably slow, a deliberate design choice to maximize decompression performance.
- Memory Footprint: Features constant and low memory usage (<= 5 MB for compression, 0 MB for decompression).
- Experimental Nature: Currently in v0.x.y, meaning the format may change, and the decoder assumes valid input, making it experimental and not yet hardened against malicious data.
- Configurability: Includes two compression effort levels (0 and 1) and experimental modes for adaptive tuning.
In essence, misa77 targets a specific niche where data is compressed once and read many times, offering a compelling performance advantage for decompression-heavy workloads, though its early-stage development means users should proceed with caution regarding its stability and security.
The Gossip
Decompressor's Design Demystified
Commenters were keenly interested in the technical insights behind misa77's superior decompression speed. The author explained the gains stem from a simple decompressor and optimizing for out-of-order cores by reducing branches. The discussion also explored the common expectation that highly compressible files decompress faster and drew parallels to other highly optimized LZ4 implementations, such as the one found in ClickHouse.
Robustness and Reliability Ruminations
A significant point of discussion revolved around misa77's robustness and security, particularly its handling of corrupted or malicious input given its "experimental" status. Commenters highlighted that the current decoder assumes valid input, unlike battle-tested codecs like LZ4 which offer defined behavior. The author acknowledged this limitation, confirming plans to integrate a safe, input-hardened version in a future release (v0.3.0) with only a minimal performance impact.
Benchmarking Brouhaha & Real-World Revelations
Users eagerly requested and shared diverse benchmarks, probing misa77's performance against competitors like Oodle Selkie and across different architectures. A debate emerged over its AArch64 performance, with conflicting claims from a user testing on Graviton 4 and the author providing M3 Mac benchmarks. The author also provided impromptu benchmarks on game assets (maps, textures), demonstrating good, albeit more modest, gains compared to its stellar performance on textual data, suggesting data type sensitivity.
Integration Insights & Weissman Wisecracks
Some commenters lauded misa77's promising performance and requested clearer code examples in the README to streamline developer integration. Amidst the serious technical discussion, a commenter lightheartedly invoked the 'Weissman score' from *Silicon Valley*, reflecting the community's playful engagement with advanced compression metrics and the quest for ultimate efficiency.