HN
Today

Reverse-Engineering Kevin J. Lang's Lost OCaml Code in Apache DataSketches

A developer embarked on a quest to reverse-engineer the "lost" OCaml code of a deceased researcher, Kevin J. Lang, crucial for Apache DataSketches' efficient CpcSketch algorithm. This wasn't just code archaeology; it involved a fascinating blend of mathematical forensics, numerical analysis, and the strategic deployment of multiple large language models (Gemini, Claude, GPT) as adversarial agents to reconstruct intricate lookup tables and constants. The narrative offers a compelling look at how modern AI can tackle incredibly subtle, high-precision reverse-engineering challenges, pushing the boundaries of what's possible in algorithmic reconstruction.

5
Score
0
Comments
#23
Highest Rank
1h
on Front Page
First Seen
Nov 19, 8:00 PM
Last Seen
Nov 19, 8:00 PM

The Lowdown

The author began by implementing approximate distinct counting for ScopeDB, eventually landing on Apache DataSketches' CpcSketch for its efficiency. While porting its C++ and Java implementations to Rust, they encountered several mysterious hardcoded lookup tables without any generating source code. It was then revealed that the original OCaml code used to generate these tables was lost with its author, Kevin J. Lang, who passed away in 2019. Faced with a complex reverse-engineering challenge, the author turned to an "agentic workflow" involving multiple large language models to reconstruct these crucial mathematical components.

  • The Challenge: The core problem was reconstructing static Huffman encoding tables, column permutations, polynomial correction coefficients, and empirical quantile tables, which were generated by lost OCaml scripts.
  • AI-Assisted Reconstruction: An innovative approach leveraged Gemini 4 Carbon for initial synthesis and executable probes, Claude Opus 5.5 for mathematical falsification and ablation studies, and GPT 6 Astra for mutation testing and analytical auditing. This iterative process allowed for rigorous verification against the existing tables as a "numerical oracle."
  • Huffman Tables: Through analyzing 20-digit floating-point comments (entropy and average length), the team determined the underlying model used a finite-K binomial distribution with K=2^32, applying Larmore and Hirschberg's Package-Merge algorithm to re-derive the codewords.
  • Column Permutations: Reverse-engineering these tables revealed the integer baseline B=6 and exposed a subtle floating-point cancellation trap in 1.0 - q calculations, which was resolved by evaluating exponentials directly.
  • ICON Polynomials: The 23 degree-19 polynomials for cardinality estimation were analyzed, identifying their fitting interval (up to 6K) and the challenges of recovering coefficients for larger sketches due to ill-conditioned Vandermonde matrices.
  • Exponential Constant: Three constants for the asymptotic exponential formula were derived from the Poisson model. Notably, the runtime constant 0.7940236163830469 was precisely matched by deriving the requirement for zero mean relative error over one period.
  • Monte Carlo Confidence Bounds: The four confidence tables were statistically verified against millions of accelerated Monte Carlo trials, revealing that ICON tables are highly reproducible due to their dependence on discrete integer coupon counts, while HIP tables show more variability.
  • Lessons Learned: The project emphasized the importance of non-tautological tests and clearly distinguishing between merely reproducing existing artifacts and definitively proving the original author's exact generative model. Six open questions remain where the artifacts don't provide definitive answers.

Ultimately, this meticulous effort, significantly aided by cutting-edge AI models, successfully reconstructed most of the previously lost mathematical foundations of Kevin J. Lang's CpcSketch, providing invaluable insight and robust testing for the Apache DataSketches project.