HN
Today

Tracking down a Zsh history data loss bug

Michael Stapelberg embarks on a meticulous technical journey, leveraging advanced Linux tracing and custom Zsh patching, to finally uncover the elusive Zsh history data loss bug that plagued him for years. This deep dive into low-level debugging and operating system mechanics is a classic Hacker News favorite, showcasing perseverance and technical prowess. Adding a contemporary twist, the author evaluates if modern AI models could replicate his sophisticated bug-finding process, sparking further interest in AI's diagnostic capabilities.

18
Score
2
Comments
#8
Highest Rank
10h
on Front Page
First Seen
Aug 15, 11:00 PM
Last Seen
Aug 16, 8:00 AM
Rank Over Time
8111113142019201718

The Lowdown

For years, the author, Michael Stapelberg, encountered an irritating and mysterious problem: his Zsh command history would occasionally truncate, losing valuable recent entries. Despite backups, the persistence of this data loss prompted a deep, systematic investigation into the Zsh source code and Linux's tracing mechanisms, ultimately leading to the discovery of a long-standing bug now fixed in Zsh 5.9.2.

  • The Symptom: The primary issue was the .zsh_history file intermittently losing newer commands, appearing to revert to a much older state without visible corruption, just fewer entries.
  • Initial Tracing with inotify: Observing file system events revealed Zsh's history saving method: it reads the existing history, writes it to a new temporary file (.zsh_history.new), and then renames the temporary file over the original, effectively deleting the old one.
  • Advanced Tracing with fatrace and bpftrace: To gain deeper insight into Zsh's interactions with the history file, fatrace was used to identify process IDs, and bpftrace was employed to monitor system calls, including open, read, write, and close, providing detailed stack traces and byte counts for file operations.
  • Reproducing the Truncation: bpftrace logs showed that during truncation events, Zsh was reading significantly fewer bytes from the history file than in normal operations, yet still writing that incomplete data to the new file. To force a reproducible crash, the author patched Zsh to intentionally crash if it wrote a .zsh_history.new file below a certain size.
  • Uncovering the Root Cause: Analysis of the resulting core dump and Zsh's hist.c source code pinpointed the readhistfile function. A signal (errflag & ERRFLAG_INT) could interrupt its read loop, causing it to prematurely stop reading the history. The savehistfile function, which is called upon shell exit, would then write this incompletely read history back to the file, leading to truncation.
  • User Behavior as Trigger: The author identified his own habit of repeatedly using Ctrl+C and Ctrl+D during terminal session teardown as the likely trigger, as these actions could send signals that interrupt Zsh's history processing.
  • Resolution and Release: A bug report was filed, and a fix was provided by Bart Schaefer. Due to an oversight, it missed the Zsh 5.9.1 release but was eventually included in Zsh 5.9.2.
  • Related Issues: The post also detailed a "bonus footgun" where Emacs TRAMP mode or other shell configurations exporting HISTFILE could inadvertently lead to history truncation.
  • AI Evaluation: As a fascinating aside, the author conducted an evaluation to see if modern large language models (LLMs) could identify this complex bug based on the symptoms and bpftrace output. Frontier models like Claude Opus 5 and GPT 5.6 Sol were able to find the bug, especially with hints about user habits, demonstrating AI's growing diagnostic capabilities for intricate code issues.

This detailed exposition highlights the persistence required for deep technical debugging and the importance of seemingly minor implementation details in critical system tools. The resolution of this bug, which caused silent data loss for potentially a decade, underscores the impact of thorough software maintenance and community contributions.