A Theory of Deep Learning

Deep learning has achieved remarkable success, yet its theoretical underpinnings remain a significant mystery. Traditional statistical learning theory often fails to explain phenomena such as benign overfitting, where highly overparameterized models interpolate noise but still generalize well, or the 'double descent' curve, where test error paradoxically decreases after the interpolation threshold. This article presents a novel theory that attempts to resolve these long-standing puzzles by re-framing deep learning's generalization capabilities.

The core of the theory proposes abandoning the analysis of neural networks in parameter space for a dynamical systems approach in output space. Key ideas include:

• The Problem: Classical theory predicts overfitting for highly expressive networks, yet deep learning exhibits 'benign overfitting,' 'double descent,' 'implicit bias,' and 'grokking,' all defying conventional explanations.
• Output Space Analysis: The theory analyzes networks by tracking how predictions evolve, focusing on the flow of error rather than individual parameters.
• Neural Tangent Kernel (eNTK): It leverages the empirical Neural Tangent Kernel (eNTK) to understand how gradient steps on one training point affect predictions on others.
• Signal Channel and Reservoir: By integrating the eNTK over time, the theory identifies a 'signal channel' where loss is dissipated (learned signal) and a 'reservoir' where training dissipates nothing (test-invisible noise). Overparameterization creates a large reservoir to sequester noise.
• Unifying Puzzles: Benign overfitting occurs when noise resides in the test-invisible reservoir. Double descent is explained by noise moving between the signal channel and reservoir. Implicit bias is the spectral filling of the signal channel, learning parsimonious modes first. Grokking is signal migrating from the reservoir to the signal channel late in training.
• Practical Implications: The theory suggests a new optimization algorithm that updates parameters only if their 'batch signal' exceeds their 'leave-one-out noise,' promising faster grokking and improved fine-tuning without validation sets.
• Future Directions: This framework could lead to more efficient training (analytically 'jumping' to the final state), direct population risk minimization, and new model architectures designed to optimally sequester label noise in the reservoir.

Ultimately, this theoretical framework offers a unified, mechanistic explanation for several perplexing aspects of deep learning generalization, moving beyond descriptive observations to a more structured understanding.