Evaluating multilateral price indices in a dynamic item universe

Statistics Norway has a long history of using scanner data in the Consumer Price Index (CPI). The early research – in Norway as well as internationally – was focused on supermarket data which consists largely of stable items. The attention has since gradually shifted towards the parts of consumption market that are characterized by high item churn, where the methodology initially introduced for supermarket data is no longer adequate. Several National Statistical Institutes (NSIs) including Statistics Norway have been researching on a more generic scanner data methodology. The overall goal is to implement an approach that incorporates expenditure shares at the most detailed level without suffering from chain drift, and that works well across different commodity groups including those with high item churn. A variety of methods and index formulas are currently being tested and implemented for CPIs in different parts of the world. We propose a systematic approach to the investigation process, which has recently been developed at Statistics Norway. This consists mainly of two parts: a Total Effect Framework (TEF), and a set of generic diagnostics. The TEF is defined by the necessary choices required and the elements that affect these choices and we review, synthesize and develop a set of generic diagnostics. Most indices employed in such diagnostics are not genuine candidates for real production, but they are designed and introduced to generate useful empirical evidences, on which a plausible final choice of index method can be based. We shall illustrate the generic diagnostics using scanner datasets mainly from the markets of sport equipment which have high item churn. To this end we summarize our own experiences and put forward some preliminary conclusions of a generic scanner data price index methodology.