Using Performance Indicators With Child Welfare Policy Makers and Managers

"If you cannot measure it you cannot manage it" has become a mantra for public social service managers. At the same time outcome based management has had increasing popularity and attention because it apparently provides accountability under more flexible funding schemes. Further, it is more in keeping with an entrepreneurial management style. For these reasons the public sector concern about measuring service performance may well be at an all time high. Yet, public service managers and policy makers are often much more certain about what they can no longer use to manage programs and justify budgetary goals--that is, simple descriptions of the numbers of clients served per FTE--than they are about what they do need to measure. Administrators have relatively little understanding of the complexities and challenges of measuring outcomes over time. Nor do they routinely call for assessment of case characteristics so differential outcomes can be measured. They fail to fully understand that case characteristics may be more important determinants of outcomes than the contributions of the service program themselves. Some have learned to use as many outcomes as possible whereas others are still searching for a single outcome that will indicate optimal program performance. In the last eight years of work with the federal government, the California Department of Social Services, and local county governments in California, we have learned a variety of lessons about developing and communicating indicators of child welfare services performance. We began much of our work simply repackaging publicly available data and have moved on to analyzing massive amounts of highly confidential data. We have been fortunate during these years to have worked with many policy makers and managers who have shaped our strategies for presenting and discussing our data with candor, probing questions, expressions of confusion, and expectations for ever improving work. The ideas contained within are largely drawn from that experiential base. In this work we have drawn on a wide variety of data sets that are most clearly described to be part of the emerging California Children's Services Archive. This Archive includes both basic demographics and service dynamics for all children who were born in California since 1967; children from 10 counties who have been abused and neglected; wards of the California Youth Authority; children in special education; and children in conventional foster care, kinship foster care, treatment foster care and group home care. The challenges of merging data from different sources with different identifiers in order to create policy and management relevant analyses are invigorating and consequential. There are also many stimulating challenges about what to do with those data once merged. Choices must be made about the proper analyses, about the level of certainty or robustness of findings required before they are presented to public social service agencies, and about ways to present those data so that they can be optimally understood and minimally misunderstood. These choices have too rarely been discussed in the scholarly literature. Much of our academic discussion has to do with ways to generate statistics with the assumption that those statistics will ultimately be read, understood, and used by public policy makers and managers. This paper is not based on scientific study of the process of research utilization. It is, instead, based on a set of impressions and field notes from countless interactions with public policy makers and managers.