FoolWatch: A Case study of econometric analysis and evidenced-based-policy making in the Australian Government

The decision to introduce a national FuelWatch scheme provides a timely case study of `evidence-based- policy' making in Australia. The government based its decision on econometric work by the ACCC who refuse to release the data. They claim that their analysis is robust because it has been subject to scrutiny within the ACCC and by Treasury. Experience with `evidence based policy' making in the United Kingdom (UK) raises doubts about such claims. The UK experience led to the term `policy-based-evidence' to describe the end result where government agencies filtered out information that was inconsistent with government policy. The data used by the ACCC can be digitized from a graph in their report. I then use the data so obtained to assess the robustness of the ACCC analysis and econometrics. Since the government and their advisors did not realize that the data could be obtained in this way it provides a `natural experiment' in which their claims can be evaluated and tested. I find that the ACCC apply the wrong tests to the wrong variable. Specifically, they study the nominal retail margin when economic theory suggests that analysis of anything but the real retail margin to producers creates a misspecified model inconsistent with the econometric assumptions used. The econometrics in Appendix S of the ACCC report on petrol is substandard in application of techniques and in reporting of what was done. This should not be seen as an adverse reflection on the econometricians employed by the ACCC. Rather, it is a reflection on the process through which their work is incorporated into reports and used by government. The ACCC findings are not robust. When I apply the correct version of the procedures used by the ACCC to the correct variable I find that the data does not support the original ACCC finding. Specifically, it is not possible to conclude as the ACCC did that FuelWatch did not raise petrol prices in Western Australia. FuelWatch of itself is unimportant. The important issues here relate to the integrity of the government's `evidence-based' approach to policy. The UK experience clearly shows that relying on the untested opinion of `experts' leads to fudging' of the evidence or overstatement of the conclusions that can be supported by the data. This ultimately corrupts the evidence-based-policy approach. The FuelWatch experience shows that these dangers are present for Australian policy makers. Greater transparency by government, publication of data and analysis underpinning government decisions and independent review of econometric work are the only protections against this danger. Failure to implement these protections will invalidate claims about the evidence base of future policy decisions.