The death and rebirth of my home server

The author chronicles the demise of their Raspberry Pi home server's SD card due to wear and a power outage, prompting a detailed technical rebuild using NixOS. This comprehensive overhaul focuses on write minimization, Btrfs data redundancy, and robust backups to ensure future stability. Hacker News found this narrative highly relatable, sparking extensive discussion on Raspberry Pi storage solutions and the merits of durable home server configurations.

Score

Comments

Highest Rank

14h

on Front Page

First Seen

Jul 19, 12:00 PM

Last Seen

Jul 20, 1:00 AM

Rank Over Time

The Lowdown

This story details the author's journey in rebuilding a Raspberry Pi 4B home server after its microSD card failed, likely due to prolonged 24/7 operation and write-intensive usage. The failure, hastened by a power outage, served as a catalyst to implement more resilient and durable configurations. The author leverages NixOS to declaratively define the new server architecture, emphasizing longevity and data integrity.

The initial issue was a non-booting Raspberry Pi, traced to a corrupted microSD card that had been serving as the root filesystem without write-minimization precautions.
The rebuild focused on three key principles: reducing writes to the new microSD card, implementing data redundancy for external storage, and ensuring robust backups.
To minimize SD card wear, the author configured zram for swap, used a RAM disk for /tmp, set journald to volatile storage, and disabled atime on the root filesystem.
External data was moved to a Btrfs RAID1 pool created from two old, mismatched HDDs, named "ponkotsu" (piece of junk). This setup utilizes Btrfs subvolumes, managed declaratively with a custom autosubvol NixOS module, allowing per-service mount points and dependencies.
Backup strategy involves restic integrated with sops-nix for encrypted, versioned backups to an S3 bucket, with local btrfs snapshots as a future consideration.
Shared data access across services (e.g., Transmission, Jellyfin) is managed using a media group with the setgid bit, ensuring new files inherit appropriate group ownership.
Future plans include implementing smartd for drive health monitoring and potentially Prometheus/Grafana for broader system insights.
A "high endurance" 64GB microSD card was selected for the new setup, acknowledging the reduced write pressure due to the other optimizations.
NixOS simplifies deployment through the creation of a bootable SD image directly from the configuration, streamlining the setup process.

The successful reconstruction allowed the author to quickly restore essential services like Navidrome, demonstrating the effectiveness of the new, more robust home server design.

The Gossip

SD Card Scrutiny

Discussion revolves around the notorious unreliability of SD cards in Raspberry Pis, with many users sharing similar experiences of corruption and wear. Solutions proposed include booting from USB SSDs, using NVMe hats, or configuring a read-only root filesystem to mitigate write cycles. There's a consensus that SD cards are generally unsuitable for persistent, write-heavy operations.

Hardware Hardiness & Homelab Longevity

Commenters delve into the expected lifespan of various storage devices (HDDs vs. SSDs), the usefulness (or lack thereof for consumers) of SMART data, and the importance of a solid backup strategy over preventative hardware replacement. Many share anecdotes of incredibly long-lived hardware, contrasting with the SD card issues. The general sentiment is that while enterprise hardware might have predictive failures, for home users, backups are paramount.

NixOS Navigation & Mini-PC Alternatives

Some comments express interest in learning NixOS for similar setups or inquire about running specific services like Immich on a Raspberry Pi. There's also a strong undercurrent suggesting that dedicated mini-PCs (like NUCs, Beelinks, or used enterprise SFF desktops) might be a more robust and energy-efficient alternative to Raspberry Pis for 24/7 home server operations, especially given current Pi pricing.

AI's Assisting Admin

A couple of comments highlight an emerging trend of using Large Language Models (LLMs) to help with debugging or modernizing complex, legacy server configurations, treating them as "clankers" that can automate problem-solving.