Who Makes It Into PISA?: Understanding the Impact of PISA Sample Eligibility Using Turkey as a Case Study (PISA 2003 - PISA 2012)

Of the OECD countries that participate in the Programme for International Student Assessment (PISA), Turkey has one of the lowest levels of performance and the highest rates of improvement in PISA scores between 2003 and 2012. New evidence presented in this paper suggests that existing accounts have underestimated both progress and inequity in Turkey because they did not take into account the large proportion of 15-16 year-olds that are ineligible for the PISA sampling frame, either because they are no longer in school or because they are severely delayed. Using Turkey’s Demographic and Health Survey (DHS) data for 2003, 2008 and 2013 we show that the proportion of 15-16 year-olds that were eligible for the PISA sample in Turkey has nearly doubled from 45% in PISA 2003, to 80% in PISA 2012. By combining DHS data on access/attainment and PISA survey data on learning outcomes we show that: (1) the improvement in the percentage of 15-16 year-olds reaching Level 2 in PISA (functional literacy and functional numeracy) is up to twice as large as that reflected in official PISA reports, (2) the gap in functional literacy rates between rich and poor youth in 2012 is 2.3 times as large as was previously thought, and (3) contrary to earlier research the gap between rich and poor has not declined between 2003 and 2012. The paper emphasises the importance of accounting for sample eligibility and representivity when making inter-country and inter-temporal comparisons using international assessment data, particularly for developing countries with expanding education systems.