HN
Today

PostgreSQL and the OOM Killer: Why You Must Use Strict Memory Overcommit

This post dives deep into how Linux's OOM Killer can catastrophically impact PostgreSQL and why strict memory overcommit is a critical safeguard. It explains a subtle kernel bug that broke this protection and how the authors tracked it down to a one-character error. The article concludes with a practical heuristic for configuring the commit limit effectively.

8
Score
0
Comments
#2
Highest Rank
5h
on Front Page
First Seen
Jul 3, 1:00 PM
Last Seen
Jul 3, 5:00 PM
Rank Over Time
722510

The Lowdown

The article from Ubicloud details the critical importance of using strict memory overcommit with PostgreSQL to prevent system-wide outages caused by the Linux Out-Of-Memory (OOM) Killer. It explains how PostgreSQL's architecture, with shared memory segments, makes it uniquely vulnerable to OOM kills, potentially leading to data corruption or prolonged downtime if its child processes are terminated.

  • Linux's default memory overcommit can lead to the OOM killer terminating processes when physical memory runs out, which for PostgreSQL, means the postmaster will assume data corruption and shut down the entire database.
  • Strict memory overcommit (vm.overcommit_memory=2) forces the kernel to track total virtual memory and refuse allocations beyond a CommitLimit, causing an ENOMEM error that PostgreSQL can handle gracefully for individual connections.
  • The authors encountered a significant issue where Committed_AS values were hugely inflated on their PostgreSQL fleet, even with ample physical memory, forcing them to temporarily disable strict overcommit.
  • Through detailed investigation and statistical analysis correlating with kernel versions, they identified a kernel bug introduced in Linux 6.5.
  • The bug was a one-character error in mm/mremap.c (an inverted conditional check !do_vmi_munmap instead of do_vmi_munmap < 0), which incorrectly caused memory accounting to increase on successful remapping operations.
  • This subtle bug went unnoticed for many because Committed_AS is largely informational in default overcommit modes, only causing issues when strict overcommit is enabled and the CommitLimit is enforced.
  • Linus Torvalds himself fixed the bug, restoring correct memory accounting.
  • The article provides a recommended heuristic for setting the CommitLimit: 80% of total physical memory + 2 GB, explaining that the 20% holdback covers kernel memory and the 2GB buffer accounts for sidecar processes (like Go programs) that reserve large virtual memory upfront.

In conclusion, strict memory overcommit is a vital configuration for PostgreSQL deployments, converting potentially disastrous OOM events into manageable, localized errors. However, its effectiveness relies on accurate kernel accounting and careful configuration of the CommitLimit to balance protection with application needs.