Surprise! Measuring Novelty as Expectation Violation

Novelty assessment is central to the study and management of innovation. Here we argue that new technologies, discoveries, and cultural products are deemed novel insofar as they seem unlikely or improbable, conditional on perceptions of prior knowledge and estimations of the inventive search process. This implies that novelty has different manifestations in fields with distinct prior knowledge and processes of invention; that measuring novelty is sensitive to context and therefore “objectively subjective.” Consequently, different novelty measures will be appropriate for different fields. We then survey and systematize existing ex ante novelty measures. We array them according to the speed of simulated search and the complexity of the space over which search is simulated. Using data from 157,595 U.S. patents granted in 2000 and 90,421 patents granted in 1990, we demonstrate that inventive fields vary in their distribution of novelty measures. We also find that familiar impact-based correlates of novelty are predicted by distinct characterizations of novelty in different fields. Consistent with our hypothesis that different inventive processes imply different novelty measures, we find that nearby fields, which share similar inventive processes, also manifest similar profiles in the relationship between novelty and impact. We conclude with principles of measure selection that could lead to more credible analyses of innovation.