Rowena Jacobs () (Centre for Health Economics, The University of York) Maria Goddard () (Centre for Health Economics, The University of York) Peter C Smith () (Centre for Health Economics, The University of York)
Abstract
A composite indicator is an aggregated index comprising individual performance indicators. Composite indicators integrate a large amount of information in a format that is easily understood and are therefore a valuable tool for conveying a summary assessment of performance in priority areas. This research investigates the degree to which composite measures are an appropriate metric for evaluating performance in the public sector. Do they reflect accurately the performance of organisations? To what degree are they influenced by the uncertainty surrounding underlying indicators on which they are based? Are they robust and stable over time? The construction of composite measures creates specific methodological challenges that make such questions especially pertinent. We address these through a series of quantitative analyses of panel data relating to healthcare (Star ratings of NHS acute Trusts) and local government (Comprehensive Performance Assessment (CPA) ratings of authorities) in England where composites have been widely used. The creation of a composite comprises a number of important steps, each of which requires careful judgement. These include the specification of the choice of indicators, the transformation of measured performance on individual indicators, the specification of a set of weights on individual indicators, and combining the indicators using aggregation methods or decision rules. We use Monte Carlo simulations to examine the robustness of performance judgements to these different technical choices. We show the extent to which composites provide stable performance rankings of organisations over time and assess whether variations are due to genuine performance improvement or merely the result of random statistical variation. The analysis suggests that the judgements that have to be made in the construction of the composite can have a significant impact on the resulting score. Technical and analytical issues in the design of composite indicators have important policy implications. We highlight the issues which need to be considered in the construction of robust composite indicators so that they can be designed in ways which will minimise the potential for producing misleading performance information which may fail to deliver the expected improvements or even induce unwanted side-effects.
Download Info
To download:
If you experience problems downloading a file, check if you have the
proper application to
view it first. Information about this may be contained
in the File-Format links below. In case of further problems read
the IDEAS help
page. Note that these files are not on the IDEAS
site. Please be patient as the files may be large.
Publisher Info
Paper provided by Centre for Health Economics, University of York in its series Working Papers with number
016cherp.
Cited by: (explanations, Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.)