HN
Today

The Hardest Tarball?

This post meticulously crafts a 'hardest tarball' – a single archive designed to extract in eight distinct ways depending on the utility used. It delves deep into the ambiguous POSIX specifications around hard links and pax headers, showcasing the surprising and often inconsistent behaviors of common archiving tools. This kind of low-level exploration of system utilities and their edge cases is a perennial favorite for the technically curious HN audience.

20
Score
1
Comments
#4
Highest Rank
1h
on Front Page
First Seen
Nov 10, 3:00 PM
Last Seen
Nov 10, 3:00 PM

The Lowdown

The author presents a fascinating curiosity: a single tar archive, dubbed the 'hardest tarball', that extracts with wildly varying results across different tools. This highly engineered tarball exploits ambiguities in the POSIX standard regarding hard links and the PAX format, particularly concerning how size attributes are handled for hard-linked files. The article methodically demonstrates how these subtle differences lead to a diverse array of outcomes, highlighting the practical challenges of interoperability in fundamental system utilities.

  • Tarball Construction: The hardest.tar is created with a single GNU tar command, archiving file, a hard link to file (renamed to café), and empty. A PAX header is specifically crafted to include size=512 for the café entry, which is exactly the size of a tar header.
  • POSIX Ambiguity: The core issue stems from POSIX's vague wording that data blocks for a hard link 'may be included' in PAX archives, leading to different interpretations of whether a hard link can possess its own data payload separate from its source.
  • Extraction Divergence: Various popular tar implementations interpret and extract this tarball in unique ways:
    • GNU tar, BusyBox, Python, Go, Perl: Generally ignore the hard link's size and extract file, café, and empty as expected.
    • bsdtar (macOS): Interprets the 512 bytes as data for the hard link, resulting in café containing the header of empty.
    • 7-Zip, node-tar, Rust's tar, etc.: Recognize the 512 bytes as data but skip them, leading to empty being omitted from extraction.
    • Java's Commons Compress: Creates a separate file for the empty header as café now has data.
    • PHP's PharData: Makes café a separate, empty file and extracts empty.
    • RubyGems: Does not extract café at all, but creates an empty PaxHeaders directory.
    • R: Extracts empty with 512 null bytes, misapplying the size to empty and using end-of-archive markers.
    • Haskell's tar: Similar to GNU tar but suffers from string decoding issues, rendering café as café.
    • .NET, Julia, uv's tar-codec: These implementations simply reject the malformed tarball, which the author posits might be the most sensible, if not strictly POSIX-compliant, approach.
  • Related Project: This 'hardest tarball' is one of 134 tricky examples collected in the author's haampie/tar-minefield test suite, which aims to systematically document and compare tar implementation behaviors.

This demonstration serves as a stark reminder that even seemingly fundamental data formats can harbor deep ambiguities, leading to significant interoperability challenges across different software implementations. The author's tar-minefield project is an invaluable resource for understanding these edge cases.