HN
Today

A Better Vocabulary for Testing

This post meticulously dissects the muddled language of software testing, proposing a precise vocabulary spanning scopes, purposes, oracles, and methods. It's a deep dive into bringing clarity and actionable strategies to a fundamental yet often fuzzy engineering practice. HN users appreciate the rigorous thinking applied to an everyday developer problem.

22
Score
5
Comments
#5
Highest Rank
5h
on Front Page
First Seen
Nov 22, 4:00 PM
Last Seen
Nov 22, 8:00 PM
Rank Over Time
56788

The Lowdown

The author argues that contemporary software testing vocabulary is woefully inadequate, relying on vague terms like "unit" or "integration" that lack actionable definitions. This piece proposes a structured framework to replace these "vibes-based" classifications with a more precise lexicon for discussing, designing, and implementing tests.

  • Scopes of Testing: The article redefines test units along two axes: programming language constructs (function, class, module, system) and statefulness (stateful or stateless). It highlights how input spaces differ across these scopes and introduces Design by Contract (DbC) as a means to constrain function inputs effectively.
  • Purposes and Hypotheses: Testing is reframed as falsifying hypotheses across various domains, including functional correctness, performance, security, and resilience (chaos testing). Each hypothesis necessitates a specific "oracle" to judge its outcome.
  • Oracles of Testing: Four primary types of oracles are identified:
    • Human Judgment ("Tinker Testing"): Manual exploration and intuition-based assessment.
    • Examples: Concrete input/output pairs, encompassing snapshot and expect testing.
    • Inline Assertions: Programmatic checks like preconditions, postconditions, and invariants.
    • Properties: Universal, abstract statements about code behavior, including global invariants, metamorphic properties, roundtrip properties, compatibility (differential testing), and idempotency, extending to temporal logic for liveness.
  • Methods of Testing: This section details how inputs are provided to tests:
    • Static Analysis: Tools like type systems, refinement types (e.g., Liquid Haskell), and model checkers that verify properties without execution.
    • Enumeration: Theoretically exhaustive testing of all inputs, though often impractical.
    • Randomness: Generating inputs randomly, which includes mutation-based and coverage-guided fuzzing. This category also covers Property-Based Testing (PBT) for functions, and simulation environments for systems, emphasizing techniques like deterministic simulation testing (DST) for injecting physical faults.

In conclusion, the author posits that software testing is still in its infancy and that random testing, in its various forms, will be pivotal in expanding our ability to rigorously test software behavior.

The Gossip

Functional Follies: The Non-Functional Conundrum

Commenters vigorously debated the common distinction between "functional" and "non-functional" testing and requirements. Many found this dichotomy problematic and argued it often causes more confusion than clarity, suggesting it's an artificial or even philosophical barrier. Some proposed alternative terminology, like "cross-functional testing," to better encompass aspects like performance, security, and usability. Others defended the distinction, noting its practical implications in verification, where functional properties might be boolean while non-functional ones involve continuous qualitative assessments.