FreeBSD ate my RAM

Ever wondered why your FreeBSD server's RAM usage reports are all over the map? This deep dive explains how virtual memory, ZFS ARC, and various monitoring tools' differing heuristics create a confusing mess, culminating in the author's successful patches to btop, htop, and fastfetch. It's popular on HN because it demystifies a common sysadmin head-scratcher with expert technical analysis and a satisfying resolution.

Score

Comments

Highest Rank

21h

on Front Page

First Seen

Jul 3, 8:00 PM

Last Seen

Jul 4, 5:00 PM

Rank Over Time

The Lowdown

The author, Bruno Croci, embarked on a journey to understand why memory usage reporting on FreeBSD is so inconsistent, particularly after migrating his server and noticing discrepancies between fastfetch and btop. Drawing parallels to the 'Linux ate my RAM' phenomenon, he explains that modern operating systems cache disk data in RAM for performance, making 'unused' RAM a misnomer, as it's readily reclaimable.

Virtual Memory Fundamentals: The article introduces the concept of Virtual Memory (VM), where physical RAM is divided into 4KiB pages managed by the kernel. These pages are assigned to different queues, such as PQ_ACTIVE (actively used), PQ_INACTIVE (recently unused but reclaimable), PQ_LAUNDRY (destined for swap), PQ_UNSWAPPABLE (kernel/wired memory), and PQ_FREE (truly unused).
Disk Caching Explained: FreeBSD, especially with ZFS, heavily utilizes disk caching. ZFS's Adaptive Replacement Cache (ARC) dynamically caches frequently accessed data within 'wired' memory, shrinking or expanding based on system memory demands. This cache, along with vfs.bufspace, constitutes a significant portion of what appears to be used memory but is actually reclaimable.
Heuristic Headaches: Different memory monitoring tools (fastfetch, btop, htop) employ distinct heuristics to define 'used' memory, leading to wildly varying reports. For example, fastfetch includes inactive and cache in 'free', while btop historically counted only active and wired as 'used'.
btop's Blunders: The author meticulously debugged btop's FreeBSD memory reporting. He uncovered two major flaws: 1) The use of 32-bit unsigned integers (u_int) for memory counts, causing integer wrapping and incorrect values for systems with more than 4GiB of RAM. 2) btop was querying vm.stats.vm.v_cache_count, which has been a 'Dummy for compatibility' sysctl returning zero since FreeBSD 12.0, meaning btop never correctly reported cache.
Crafting the Fix: After identifying the issues, the author developed patches. For htop, he submitted a pull request (merged) to unify cache and buffers and correctly account for ARC within 'wired' memory. For btop, he created a pull request (pending review) to use 64-bit integers and accurately calculate reclaimable cache from vfs.bufspace and ZFS ARC, adjusting 'wired' memory accordingly.
fastfetch's Evolution: The author also contributed a PR to fastfetch, which, while not directly merged, influenced the tool's eventual implementation of proper ZFS ARC detection across various Unix-like systems.

This detailed investigation not only clarified the intricacies of FreeBSD's memory management but also led to tangible improvements in popular system monitoring tools. The author concludes by reflecting on his deep dive into OS internals, rekindling a childhood fascination with operating systems and highlighting the value of open-source contribution.

The Gossip

Memory Reporting Riddle

Commenters grappled with the inherent complexity of memory reporting, debating whether memory tools should rely on 'heuristics' or provide an 'exact' number. Many appreciated the article's clarity on why 'used' memory isn't simply 'total minus free,' emphasizing that cached memory is effectively available. The discussion highlighted the distinction between physical memory usage and virtual memory allocation.

BSD vs. the World

A common Hacker News theme emerged, comparing FreeBSD's memory management to Linux's. Some commenters expressed a preference for Linux, citing perceived accessibility or historical 'momentum beats academic perfection' arguments. Others recalled historical aspects of FreeBSD's swap logic and broader OS development philosophies, like NetBSD's past multi-platform reach, and generally appreciated that FreeBSD's memory issues were being debugged, leading to better tools for everyone.

Educational Echoes

The author's nostalgic mention of the MINIX book sparked a discussion about textbooks, then and now. Commenters pondered whether modern students retain physical books (especially with online or rented options), the value of reference books versus learning materials, and the convenience of e-textbooks. The conversation touched upon the evolving landscape of educational resources.

Community Commendation

The community overwhelmingly praised the author for the high quality of the post and his dedication to debugging and fixing real-world problems in widely used tools. Many expressed gratitude for the detailed explanation of a topic that often causes confusion, especially the successful merging of his fixes into `htop` and the positive impact on `fastfetch` and `btop`.