IDEAS home Printed from https://ideas.repec.org/a/kap/jproda/v47y2017i1d10.1007_s11123-017-0491-9.html
   My bibliography  Save this article

An international comparison of educational systems: a temporal analysis in presence of bad outputs

Author

Listed:
  • Víctor Giménez

    (Universitat Autònoma de Barcelona)

  • Claudio Thieme

    (Universidad Diego Portales)

  • Diego Prior

    (Universitat Autònoma de Barcelona)

  • Emili Tortosa-Ausina

    (Universitat Jaume I)

Abstract

This study uses the global non-radial Malmquist index to measure performance change in the educational systems of 29 countries/economies participating in PISA 2003 and 2012 for students at age 15 in the disciplines of mathematics and reading. This methodology is particularly appropriate both for its desirable properties as well as its suitability for the educational context. Results indicate a positive evolution in educational systems’ performance during this period. This improvement is mainly due a positive efficiency change, which offsets the negative technological change observed. Nevertheless, a deeper scrutiny at the country level shows that results varied remarkably among them.

Suggested Citation

  • Víctor Giménez & Claudio Thieme & Diego Prior & Emili Tortosa-Ausina, 2017. "An international comparison of educational systems: a temporal analysis in presence of bad outputs," Journal of Productivity Analysis, Springer, vol. 47(1), pages 83-101, February.
    • Handle: RePEc:kap:jproda:v:47:y:2017:i:1:d:10.1007_s11123-017-0491-9
    DOI: 10.1007/s11123-017-0491-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11123-017-0491-9
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s11123-017-0491-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Johnes, Jill, 2015. "Operational Research in education," European Journal of Operational Research, Elsevier, vol. 243(3), pages 683-696.
    2. Yoruk, BarIs K. & Zaim, Osman, 2005. "Productivity growth in OECD countries: A comparison with Malmquist indices," Journal of Comparative Economics, Elsevier, vol. 33(2), pages 401-420, June.
    3. Tone, Kaoru & Sahoo, Biresh K., 2003. "Scale, indivisibilities and production function in data envelopment analysis," International Journal of Production Economics, Elsevier, vol. 84(2), pages 165-192, May.
    4. Afonso, Antonio & St. Aubyn, Miguel, 2005. "Non-Parametric Approaches to Education and Health Efficiency in OECD Countries," Journal of Applied Economics, Universidad del CEMA, vol. 8(2), pages 1-20, November.
    5. Kristof De Witte & Laura López-Torres, 2017. "Efficiency in education: a review of literature and a way forward," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 339-363, April.
    6. V Giménez & D Prior & C Thieme, 2007. "Technical efficiency, managerial efficiency and objective-setting in the educational system: an international comparison," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 58(8), pages 996-1007, August.
    7. Stijn Reinhard & C. A. Knox Lovell & Geert Thijssen, 2002. "Analysis of Environmental Efficiency Variation," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 84(4), pages 1054-1065.
    8. Andrew Worthington, 2001. "An Empirical Survey of Frontier Efficiency Measurement Techniques in Education," Education Economics, Taylor & Francis Journals, vol. 9(3), pages 245-268.
    9. Kumar, Surender, 2006. "Environmentally sensitive productivity growth: A global analysis using Malmquist-Luenberger index," Ecological Economics, Elsevier, vol. 56(2), pages 280-293, February.
    10. repec:dau:papers:123456789/7958 is not listed on IDEAS
    11. Bert Balk, 2001. "Scale Efficiency and Productivity Change," Journal of Productivity Analysis, Springer, vol. 15(3), pages 159-183, May.
    12. Caves, Douglas W & Christensen, Laurits R & Diewert, W Erwin, 1982. "The Economic Theory of Index Numbers and the Measurement of Input, Output, and Productivity," Econometrica, Econometric Society, vol. 50(6), pages 1393-1414, November.
    13. Fare, Rolf, et al, 1989. "Multilateral Productivity Comparisons When Some Outputs Are Undesirable: A Nonparametric Approach," The Review of Economics and Statistics, MIT Press, vol. 71(1), pages 90-98, February.
    14. Dyson, R. G. & Allen, R. & Camanho, A. S. & Podinovski, V. V. & Sarrico, C. S. & Shale, E. A., 2001. "Pitfalls and protocols in DEA," European Journal of Operational Research, Elsevier, vol. 132(2), pages 245-259, July.
    15. Dominique Deprins & Léopold Simar & Henry Tulkens, 2006. "Measuring Labor-Efficiency in Post Offices," Springer Books, in: Parkash Chander & Jacques Drèze & C. Knox Lovell & Jack Mintz (ed.), Public goods, environmental externalities and fiscal competition, chapter 0, pages 285-309, Springer.
    16. Herbert Scarf, 1994. "The Allocation of Resources in the Presence of Indivisibilities," Journal of Economic Perspectives, American Economic Association, vol. 8(4), pages 111-128, Fall.
    17. Golany, B & Roll, Y, 1989. "An application procedure for DEA," Omega, Elsevier, vol. 17(3), pages 237-250.
    18. Pastor, Jesus T. & Lovell, C.A. Knox, 2005. "A global Malmquist productivity index," Economics Letters, Elsevier, vol. 88(2), pages 266-271, August.
    19. Afonso, Antonio & St. Aubyn, Miguel, 2006. "Cross-country efficiency of secondary education provision: A semi-parametric analysis with non-discretionary inputs," Economic Modelling, Elsevier, vol. 23(3), pages 476-491, May.
    20. Douglas Sutherland & Robert Price & Eric Gonand, 2009. "Improving public spending efficiency in primary and secondary education," OECD Journal: Economic Studies, OECD Publishing, vol. 2009(1), pages 1-30.
    21. Giorgina Brown & John Micklewright & Sylke V. Schnepf & Robert Waldmann, 2007. "International surveys of educational achievement: how robust are the findings?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 170(3), pages 623-646, July.
    22. Henderson,Daniel J. & Parmeter,Christopher F., 2015. "Applied Nonparametric Econometrics," Cambridge Books, Cambridge University Press, number 9780521279680.
    23. Oleg Badunenko & Diego Romero‐Ávila, 2013. "Financial Development And The Sources Of Growth And Convergence," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 54(2), pages 629-663, May.
    24. Quah, D., 1990. "Galton'S Fallacy And The Tests Of The Convergence Hypothesis," Working papers 552, Massachusetts Institute of Technology (MIT), Department of Economics.
    25. Aristovnik, Aleksander & Obadić, Alka, 2014. "Measuring relative efficiency of secondary education in selected EU and OECD countries: the case of Slovenia and Croatia," MPRA Paper 63936, University Library of Munich, Germany.
    26. Gupta, Sanjeev & Verhoeven, Marijn, 2001. "The efficiency of government expenditure: experiences from Africa," Journal of Policy Modeling, Elsevier, vol. 23(4), pages 433-467, May.
    27. Quah, Danny, 1993. "Empirical cross-section dynamics in economic growth," European Economic Review, Elsevier, vol. 37(2-3), pages 426-434, April.
    28. Deutsch, Joseph & Dumas, Audrey & Silber, Jacques, 2013. "Estimating an educational production function for five countries of Latin America on the basis of the PISA data," Economics of Education Review, Elsevier, vol. 36(C), pages 245-262.
    29. William L. Weber & Bruce Domazlicky, 2001. "Productivity Growth and Pollution in State Manufacturing," The Review of Economics and Statistics, MIT Press, vol. 83(1), pages 195-199, February.
    30. B. Hollingsworth & P. Smith, 2003. "Use of ratios in data envelopment analysis," Applied Economics Letters, Taylor & Francis Journals, vol. 10(11), pages 733-735.
    31. Dong-hyun Oh, 2010. "A global Malmquist-Luenberger productivity index," Journal of Productivity Analysis, Springer, vol. 34(3), pages 183-197, December.
    32. Golany, Boaz & Thore, Sten, 1997. "The Economic and Social Performance of Nations: Efficiency and Returns to Scale," Socio-Economic Planning Sciences, Elsevier, vol. 31(3), pages 191-204, September.
    33. Timo Kuosmanen, 2005. "Weak Disposability in Nonparametric Production Analysis with Undesirable Outputs," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 87(4), pages 1077-1082.
    34. José Manuel Cordero & Daniel Santín & Rosa Simancas, 2017. "Assessing European primary school performance through a conditional nonparametric model," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 364-376, April.
    35. Nakano, Makiko & Managi, Shunsuke, 2008. "Regulatory reforms and productivity: An empirical analysis of the Japanese electricity industry," Energy Policy, Elsevier, vol. 36(1), pages 201-209, January.
    36. Daniel J. Henderson & R. Robert Russell, 2005. "Human Capital And Convergence: A Production-Frontier Approach ," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 46(4), pages 1167-1205, November.
    37. Olesen, Ole Bent & Petersen, Niels Christian & Podinovski, Victor V., 2015. "Efficiency analysis with ratio measures," European Journal of Operational Research, Elsevier, vol. 245(2), pages 446-462.
    38. Geraint Johnes & Jill Johnes (ed.), 2004. "International Handbook on the Economics of Education," Books, Edward Elgar Publishing, number 2847.
    39. Shawna Grosskopf & Kathy J. Hayes & Lori L. Taylor, 2014. "Efficiency in Education: Research and Implications," Applied Economic Perspectives and Policy, Agricultural and Applied Economics Association, vol. 36(2), pages 175-210.
    40. Emili GRIFELL‐TATJÉ & Kristiaan KERSTENS, 2008. "Incentive Regulation And The Role Of Convexity In Benchmarking Electricity Distribution: Economists Versus Engineers," Annals of Public and Cooperative Economics, Wiley Blackwell, vol. 79(2), pages 227-248, June.
    41. Zhang, Ning & Zhou, P. & Choi, Yongrok, 2013. "Energy efficiency, CO2 emission performance and technology gaps in fossil fuel electricity generation in Korea: A meta-frontier non-radial directional distance functionanalysis," Energy Policy, Elsevier, vol. 56(C), pages 653-662.
    42. Fare, Rolf & Grosskopf, Shawna, 2004. "Modeling undesirable factors in efficiency evaluation: Comment," European Journal of Operational Research, Elsevier, vol. 157(1), pages 242-245, August.
    43. Watanabe, Michio & Tanaka, Katsuya, 2007. "Efficiency analysis of Chinese industry: A directional distance function approach," Energy Policy, Elsevier, vol. 35(12), pages 6323-6331, December.
    44. Atakelty Hailu & Terrence S. Veeman, 2001. "Non-parametric Productivity Analysis with Undesirable Outputs: An Application to the Canadian Pulp and Paper Industry," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 83(3), pages 605-616.
    45. Aparicio, Juan & Crespo-Cebada, Eva & Pedraja-Chaparro, Francisco & Santín, Daniel, 2017. "Comparing school ownership performance using a pseudo-panel database: A Malmquist-type index approach," European Journal of Operational Research, Elsevier, vol. 256(2), pages 533-542.
    46. Zhang, Ning & Choi, Yongrok, 2013. "Total-factor carbon emission performance of fossil fuel power plants in China: A metafrontier non-radial Malmquist index analysis," Energy Economics, Elsevier, vol. 40(C), pages 549-559.
    47. Tommaso Agasisti & Pablo Zoido, 2015. "The Efficiency of Secondary Schools in an International Perspective: Preliminary Results from PISA 2012," OECD Education Working Papers 117, OECD Publishing.
    48. Fare, Rolf & Grosskopf, Shawna & Noh, Dong-Woon & Weber, William, 2005. "Characteristics of a polluting technology: theory and practice," Journal of Econometrics, Elsevier, vol. 126(2), pages 469-492, June.
    49. Subodh Kumar & R. Robert Russell, 2002. "Technological Change, Technological Catch-up, and Capital Deepening: Relative Contributions to Growth and Convergence," American Economic Review, American Economic Association, vol. 92(3), pages 527-548, June.
    50. Zhou, P. & Ang, B.W. & Wang, H., 2012. "Energy and CO2 emission performance in electricity generation: A non-radial directional distance function approach," European Journal of Operational Research, Elsevier, vol. 221(3), pages 625-635.
    51. Seiford, Lawrence M. & Zhu, Joe, 2002. "Modeling undesirable factors in efficiency evaluation," European Journal of Operational Research, Elsevier, vol. 142(1), pages 16-20, October.
    52. Sueyoshi, Toshiyuki & Goto, Mika, 2011. "Measurement of Returns to Scale and Damages to Scale for DEA-based operational and environmental assessment: How to manage desirable (good) and undesirable (bad) outputs?," European Journal of Operational Research, Elsevier, vol. 211(1), pages 76-89, May.
    53. William W. Cooper & Lawrence M. Seiford & Kaoru Tone, 2007. "Data Envelopment Analysis," Springer Books, Springer, edition 0, number 978-0-387-45283-8, December.
    54. Mei Xue & Patrick T. Harker, 2002. "Note: Ranking DMUs with Infeasible Super-Efficiency DEA Models," Management Science, INFORMS, vol. 48(5), pages 705-710, May.
    55. Quah, Danny, 1993. " Galton's Fallacy and Tests of the Convergence Hypothesis," Scandinavian Journal of Economics, Wiley Blackwell, vol. 95(4), pages 427-443, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cordero, Jose M. & Polo, Cristina & Santín, Daniel & Simancas, Rosa, 2018. "Efficiency measurement and cross-country differences among schools: A robust conditional nonparametric analysis," Economic Modelling, Elsevier, vol. 74(C), pages 45-60.
    2. Arbona, Alexei & Giménez, Víctor & López-Estrada, Sebastián & Prior, Diego, 2022. "Efficiency and quality in Colombian education: An application of the metafrontier Malmquist-Luenberger productivity index," Socio-Economic Planning Sciences, Elsevier, vol. 79(C).
    3. Aparicio, Juan & Cordero, Jose M. & Ortiz, Lidia, 2019. "Measuring efficiency in education: The influence of imprecision and variability in data on DEA estimates," Socio-Economic Planning Sciences, Elsevier, vol. 68(C).
    4. Cinzia Alcidi & Francesco Corti & Daniel Gros, 2022. "A Golden Rule for Social Investments: How to Do It," Intereconomics: Review of European Economic Policy, Springer;ZBW - Leibniz Information Centre for Economics;Centre for European Policy Studies (CEPS), vol. 57(1), pages 26-32, January.
    5. Stumbrienė, Dovilė & Želvys, Rimantas & Žilinskas, Julius & Dukynaitė, Rita & Jakaitienė, Audronė, 2022. "Efficiency and effectiveness analysis based on educational inclusion and fairness of European countries," Socio-Economic Planning Sciences, Elsevier, vol. 82(PB).
    6. Inés P. Murillo & José L. Raymond & Jorge Calero, 2017. "Efficiency in the transformation of schooling into competences: A cross-country analysis using PIAAC data," Working Papers 2017/12, Institut d'Economia de Barcelona (IEB).
    7. Corti, Francesco & Alcidi, Cinzia & Gros, Daniel & Liscai, Alessandro & Shamsfakhr, Farzaneh, 2022. "A qualified treatment for green and social investments within a revised EU fiscal framework," CEPS Papers 36574, Centre for European Policy Studies.
    8. Camanho, Ana S. & Stumbriene, Dovile & Barbosa, Flávia & Jakaitiene, Audrone, 2023. "The assessment of performance trends and convergence in education and training systems of European countries," European Journal of Operational Research, Elsevier, vol. 305(1), pages 356-372.
    9. Cordero, Jose Manuel & Polo, Cristina & Simancas, Rosa, 2022. "Assessing the efficiency of secondary schools: Evidence from OECD countries participating in PISA 2015," Socio-Economic Planning Sciences, Elsevier, vol. 81(C).
    10. K. Kounetas & G. Androulakis & M. Kaisari & G. Manousakis, 2023. "Educational reforms and secondary school's efficiency performance in Greece: a bootstrap DEA and multilevel approach," Operational Research, Springer, vol. 23(1), pages 1-29, March.
    11. Jorge Calero & Inés P. Murillo Huertas & José L. Raymond, 2021. "Efficiency in the transformation of schooling into competences: A cross‐country analysis using PIAAC data," Bulletin of Economic Research, Wiley Blackwell, vol. 73(2), pages 252-275, April.
    12. Aparicio, Juan & Cordero, Jose M. & Gonzalez, Martin & Lopez-Espin, Jose J., 2018. "Using non-radial DEA to assess school efficiency in a cross-country perspective: An empirical analysis of OECD countries," Omega, Elsevier, vol. 79(C), pages 9-20.
    13. Alexander Arévalo S & Víctor Giménez G & Diego Prior J, 2022. "Análisis de eficiencia en educación: una aplicación del método StoNED," Revista Desarrollo y Sociedad, Universidad de los Andes,Facultad de Economía, CEDE, vol. 92(2), pages 45-91, October.
    14. Víctor Giménez & Claudio Thieme & Diego Prior & Emili Tortosa-Ausina, 2019. "Comparing the Performance of National Educational Systems: Inequality Versus Achievement?," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(2), pages 581-609, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Víctor Giménez & Claudio Thieme & Diego Prior & Emili Tortosa-Ausina, 2019. "Comparing the Performance of National Educational Systems: Inequality Versus Achievement?," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(2), pages 581-609, January.
    2. Víctor Giménez & Claudio Thieme & Diego Prior & Emili Tortosa-Ausina, 2014. "An international comparison of educational systems: an application of the global Malmquist-Luenberger index," Working Papers 2014/18, Economics Department, Universitat Jaume I, Castellón (Spain).
    3. Víctor Giménez García & Diego Prior Jiménez & Emili Tortosa-Ausina & Claudio Thieme, 2016. "Una comparación internacional de la eficiencia de sistemas educativos. Un enfoque dinámico para el periodo 2003-2012," Investigaciones de Economía de la Educación volume 11, in: José Manuel Cordero Ferrera & Rosa Simancas Rodríguez (ed.), Investigaciones de Economía de la Educación 11, edition 1, volume 11, chapter 36, pages 653-670, Asociación de Economía de la Educación.
    4. Giménez, Víctor & Prieto, William & Prior, Diego & Tortosa-Ausina, Emili, 2019. "Evaluation of efficiency in Colombian hospitals: An analysis for the post-reform period," Socio-Economic Planning Sciences, Elsevier, vol. 65(C), pages 20-35.
    5. Aparicio, Juan & Cordero, Jose M. & Gonzalez, Martin & Lopez-Espin, Jose J., 2018. "Using non-radial DEA to assess school efficiency in a cross-country perspective: An empirical analysis of OECD countries," Omega, Elsevier, vol. 79(C), pages 9-20.
    6. Aparicio, Juan & Ortiz, Lidia & Santín, Daniel, 2021. "Comparing group performance over time through the Luenberger productivity indicator: An application to school ownership in European countries," European Journal of Operational Research, Elsevier, vol. 294(2), pages 651-672.
    7. Cordero, Jose M. & Polo, Cristina & Santín, Daniel & Simancas, Rosa, 2018. "Efficiency measurement and cross-country differences among schools: A robust conditional nonparametric analysis," Economic Modelling, Elsevier, vol. 74(C), pages 45-60.
    8. Kristof De Witte & Laura López-Torres, 2017. "Efficiency in education: a review of literature and a way forward," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 339-363, April.
    9. Cherchye, Laurens & Rock, Bram De & Walheer, Barnabé, 2015. "Multi-output efficiency with good and bad outputs," European Journal of Operational Research, Elsevier, vol. 240(3), pages 872-881.
    10. Valentin Zelenyuk, 2023. "Productivity analysis: roots, foundations, trends and perspectives," Journal of Productivity Analysis, Springer, vol. 60(3), pages 229-247, December.
    11. Jorge Calero & Inés P. Murillo Huertas & José L. Raymond, 2021. "Efficiency in the transformation of schooling into competences: A cross‐country analysis using PIAAC data," Bulletin of Economic Research, Wiley Blackwell, vol. 73(2), pages 252-275, April.
    12. Harald Dyckhoff & Rainer Souren, 2023. "Are important phenomena of joint production still being neglected by economic theory? A review of recent literature," Journal of Business Economics, Springer, vol. 93(6), pages 1015-1053, August.
    13. Cordero, Jose Manuel & Polo, Cristina & Simancas, Rosa, 2022. "Assessing the efficiency of secondary schools: Evidence from OECD countries participating in PISA 2015," Socio-Economic Planning Sciences, Elsevier, vol. 81(C).
    14. Emrouznejad, Ali & Yang, Guo-liang, 2016. "CO2 emissions reduction of Chinese light manufacturing industries: A novel RAM-based global Malmquist–Luenberger productivity index," Energy Policy, Elsevier, vol. 96(C), pages 397-410.
    15. Aparicio, Juan & López-Torres, Laura & Santín, Daniel, 2018. "Economic crisis and public education. A productivity analysis using a Hicks-Moorsteen index," Economic Modelling, Elsevier, vol. 71(C), pages 34-44.
    16. Dubrocard, Anne & Prombo, Michel, 2012. "International comparison of Environmental performance," MPRA Paper 48072, University Library of Munich, Germany, revised 05 Jul 2013.
    17. Arbona, Alexei & Giménez, Víctor & López-Estrada, Sebastián & Prior, Diego, 2022. "Efficiency and quality in Colombian education: An application of the metafrontier Malmquist-Luenberger productivity index," Socio-Economic Planning Sciences, Elsevier, vol. 79(C).
    18. Sueyoshi, Toshiyuki & Yuan, Yan & Goto, Mika, 2017. "A literature study for DEA applied to energy and environment," Energy Economics, Elsevier, vol. 62(C), pages 104-124.
    19. Barnabé Walheer, 2022. "Global Malmquist and cost Malmquist indexes for group comparison," Journal of Productivity Analysis, Springer, vol. 58(1), pages 75-93, August.
    20. Yu, Yanni & Qian, Tao & Du, Limin, 2017. "Carbon productivity growth, technological innovation, and technology gap change of coal-fired power plants in China," Energy Policy, Elsevier, vol. 109(C), pages 479-487.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kap:jproda:v:47:y:2017:i:1:d:10.1007_s11123-017-0491-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.