IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v308y2023i1p353-368.html
   My bibliography  Save this article

Data-driven support for policy and decision-making in university research management: A case study from Germany

Author

Listed:
  • Zharova, Alona
  • Härdle, Wolfgang Karl
  • Lessmann, Stefan

Abstract

The management of universities requires data on teaching and research performance. While teaching quality can be measured via student performance and teacher evaluation programs, the connection between research output and its antecedents is much harder to check, test, and understand. To inform research governance and policy-making at universities, we clarify the relationship between grant money and research performance. We examine the interdependence structure between third-party expenses, publications, citations, and academic age. To describe the relationship between these factors, we analyze individual-level data from a sample of professorships and a Scopus database from 2001 to 2015. Using estimates from a panel vector autoregressive model with exogenous variables, impulse response functions, and forecast error variance decomposition, we show that decision-making based on university-level data is inappropriate and does not reflect the behavior of individual faculties. Our results quantify the difference between the quality and quantity of research output, a better understanding of which is important to design incentive schemes and promotion programs. The paper also proposes a visualization of the cooperation between faculties and research interdisciplinarity via the co-authorship structure among publications. We discuss the implications for policy and decision-making and make recommendations for university research management.

Suggested Citation

  • Zharova, Alona & Härdle, Wolfgang Karl & Lessmann, Stefan, 2023. "Data-driven support for policy and decision-making in university research management: A case study from Germany," European Journal of Operational Research, Elsevier, vol. 308(1), pages 353-368.
  • Handle: RePEc:eee:ejores:v:308:y:2023:i:1:p:353-368
    DOI: 10.1016/j.ejor.2022.10.016
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0377221722008013
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ejor.2022.10.016?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Johnes, Jill, 2015. "Operational Research in education," European Journal of Operational Research, Elsevier, vol. 243(3), pages 683-696.
    2. Rodrigo Costas & Thed N. van Leeuwen & María Bordons, 2010. "A bibliometric classificatory approach for the study and assessment of research performance at the individual level: The effects of age on productivity and impact," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(8), pages 1564-1581, August.
    3. Fedderke, J.W. & Goldschmidt, M., 2015. "Does massive funding support of researchers work?: Evaluating the impact of the South African research chair funding initiative," Research Policy, Elsevier, vol. 44(2), pages 467-482.
    4. Joshua L Rosenbloom & Donna K Ginther & Ted Juhl & Joseph A Heppert, 2015. "The Effects of Research & Development Funding on Scientific Productivity: Academic Chemistry, 1990-2009," PLOS ONE, Public Library of Science, vol. 10(9), pages 1-23, September.
    5. Payne A. Abigail & Siow Aloysius, 2003. "Does Federal Research Funding Increase University Research Output?," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 3(1), pages 1-24, May.
    6. Boczek, Michał & Hovana, Anton & Hutník, Ondrej & Kaluszka, Marek, 2021. "New monotone measure-based integrals inspired by scientific impact problem," European Journal of Operational Research, Elsevier, vol. 290(1), pages 346-357.
    7. Beaudry, Catherine & Allaoui, Sedki, 2012. "Impact of public and private research funding on scientific production: The case of nanotechnology," Research Policy, Elsevier, vol. 41(9), pages 1589-1606.
    8. Thomas Fomby & Yuki Ikeda & Norman V. Loayza, 2013. "The Growth Aftermath Of Natural Disasters," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(3), pages 412-434, April.
    9. Levin, Sharon G & Stephan, Paula E, 1991. "Research Productivity over the Life Cycle: Evidence for Academic Scientists," American Economic Review, American Economic Association, vol. 81(1), pages 114-132, March.
    10. Mingers, John & Xu, Fang, 2010. "The drivers of citations in management science journals," European Journal of Operational Research, Elsevier, vol. 205(2), pages 422-430, September.
    11. Sabatier, Mareva & Chollet, Barthélemy, 2017. "Is there a first mover advantage in science? Pioneering behavior and scientific production in nanotechnology," Research Policy, Elsevier, vol. 46(2), pages 522-533.
    12. Stefan Hornbostel, 2001. "Third party funding of German universities. An indicator of research activity?," Scientometrics, Springer;Akadémiai Kiadó, vol. 50(3), pages 523-537, March.
    13. Canova, Fabio & Ciccarelli, Matteo, 2013. "Panel Vector Autoregressive Models: A Survey," CEPR Discussion Papers 9380, C.E.P.R. Discussion Papers.
    14. Ashkan Ebadi & Andrea Schiffauerova, 2016. "How to boost scientific production? A statistical analysis of research funding and other influencing factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(3), pages 1093-1116, March.
    15. Daraio, Cinzia & Bonaccorsi, Andrea & Simar, Léopold, 2015. "Rankings and university performance: A conditional multidimensional approach," European Journal of Operational Research, Elsevier, vol. 244(3), pages 918-930.
    16. Norris, Michael & Oppenheim, Charles, 2007. "Comparing alternatives to the Web of Science for coverage of the social sciences’ literature," Journal of Informetrics, Elsevier, vol. 1(2), pages 161-169.
    17. Holtz-Eakin, Douglas & Newey, Whitney & Rosen, Harvey S, 1988. "Estimating Vector Autoregressions with Panel Data," Econometrica, Econometric Society, vol. 56(6), pages 1371-1395, November.
    18. Richard K. Green & Stephen Malpezzi & Stephen K. Mayo, 2005. "Metropolitan-Specific Estimates of the Price Elasticity of Supply of Housing, and Their Sources," American Economic Review, American Economic Association, vol. 95(2), pages 334-339, May.
    19. Koop, Gary & Korobilis, Dimitris, 2010. "Bayesian Multivariate Time Series Methods for Empirical Macroeconomics," Foundations and Trends(R) in Econometrics, now publishers, vol. 3(4), pages 267-358, July.
    20. Dées, Stéphane & Güntner, Jochen, 2014. "Analysing and forecasting price dynamics across euro area countries and sectors: a panel VAR approach," Working Paper Series 1724, European Central Bank.
    21. DJIGBENOU-KRE, Marie-Louise & Park, Hail, 2016. "The effects of global liquidity on global imbalances," International Review of Economics & Finance, Elsevier, vol. 42(C), pages 1-12.
    22. Kevin W. Boyack & Katy Börner, 2003. "Indicator‐assisted evaluation and funding of research: Visualizing the influence of grants on the number and citation counts of research papers," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 54(5), pages 447-461, March.
    23. Lütkepohl, Helmut, 1999. "Vector autoregressive analysis," SFB 373 Discussion Papers 1999,31, Humboldt University of Berlin, Interdisciplinary Research Project 373: Quantification and Simulation of Economic Processes.
    24. Michael R. M. Abrigo & Inessa Love, 2016. "Estimation of panel vector autoregression in Stata," Stata Journal, StataCorp LP, vol. 16(3), pages 778-804, September.
    25. Carayol, Nicolas & Matt, Mireille, 2004. "Does research organization influence academic production?: Laboratory level evidence from a large European university," Research Policy, Elsevier, vol. 33(8), pages 1081-1102, October.
    26. Anthony F. J. Raan & Thed N. Leeuwen & Martijn S. Visser, 2011. "Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(2), pages 495-498, August.
    27. Jacob, Brian A. & Lefgren, Lars, 2011. "The impact of research grant funding on scientific productivity," Journal of Public Economics, Elsevier, vol. 95(9), pages 1168-1177.
    28. Harald Dyckhoff & Sylvia Rassenhövel & Kirsten Sandfort, 2009. "Empirische Produktionsfunktion betriebswirtschaftlicher Forschung: Eine Analyse der Daten des Centrums für Hochschulentwicklung," Schmalenbach Journal of Business Research, Springer, vol. 61(1), pages 22-56, February.
    29. Michael Roach & Wesley M. Cohen, 2013. "Lens or Prism? Patent Citations as a Measure of Knowledge Flows from Public Research," Management Science, INFORMS, vol. 59(2), pages 504-525, October.
    30. Bolli, Thomas & Somogyi, Frank, 2011. "Do competitively acquired funds induce universities to increase productivity?," Research Policy, Elsevier, vol. 40(1), pages 136-147, February.
    31. Bonaccorsi, Andrea & Belingheri, Paola & Secondi, Luca, 2021. "The research productivity of universities. A multilevel and multidisciplinary analysis on European institutions," Journal of Informetrics, Elsevier, vol. 15(2).
    32. Wolfgang Glänzel & András Schubert, 2003. "A new classification scheme of science fields and subfields designed for scientometric evaluation purposes," Scientometrics, Springer;Akadémiai Kiadó, vol. 56(3), pages 357-367, March.
    33. Sarrico, Cláudia S. & Teixeira, Pedro N. & Rosa, Maria J. & Cardoso, Margarida F., 2009. "Subject mix and productivity in Portuguese universities," European Journal of Operational Research, Elsevier, vol. 197(1), pages 287-295, August.
    34. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2011. "National-scale research performance assessment at the individual level," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(2), pages 347-364, February.
    35. Wang, Jue & Xu, Wei & Ma, Jian & Wang, Shouyang, 2013. "A vague set based decision support approach for evaluating research funding programs," European Journal of Operational Research, Elsevier, vol. 230(3), pages 656-665.
    36. Marie-Louise Djigbenou & Hail Park, 2016. "The Effects of Global Liquidity on Global Imbalances," Post-Print hal-03897233, HAL.
    37. Fandel, Gunter & Gal, Tomas, 2001. "Redistribution of funds for teaching and research among universities: The case of North Rhine-Westphalia," European Journal of Operational Research, Elsevier, vol. 130(1), pages 111-120, April.
    38. Rodrigo Costas & Thed N. van Leeuwen & María Bordons, 2010. "A bibliometric classificatory approach for the study and assessment of research performance at the individual level: The effects of age on productivity and impact," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(8), pages 1564-1581, August.
    39. Fredrik Niclas Piro & Dag W. Aksnes & Kristoffer Rørstad, 2013. "A macro analysis of productivity differences across fields: Challenges in the measurement of scientific publishing," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 307-320, February.
    40. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    41. Philippe Mongeon & Adèle Paul-Hus, 2016. "The journal coverage of Web of Science and Scopus: a comparative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 213-228, January.
    42. Lawrence R. Jauch & William F. Glueck, 1975. "Evaluation of University Professors' Research Performance," Management Science, INFORMS, vol. 22(1), pages 66-75, September.
    43. Thed N. Van Leeuwen & Henk F. Moed & Robert J. W. Tijssen & Martijn S. Visser & Anthony F. J. Van Raan, 2001. "Language biases in the coverage of the Science Citation Index and its consequencesfor international comparisons of national research performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 51(1), pages 335-346, April.
    44. Giovanni Abramo & Ciriaco Andrea D’Angelo & Gianluca Murgia, 2016. "The combined effects of age and seniority on research performance of full professors," Science and Public Policy, Oxford University Press, vol. 43(3), pages 301-319.
    45. Michael R.M. Abrigo & Inessa Love, 2016. "Estimation of Panel Vector Autoregression in Stata: a Package of Programs," Working Papers 201602, University of Hawaii at Manoa, Department of Economics.
    46. Sharmistha Nag & Hui Yang & Steven Buccola & David Ervin, 2013. "Productivity and financial support in academic bioscience," Applied Economics, Taylor & Francis Journals, vol. 45(19), pages 2817-2826, July.
    47. Pfaff, Bernhard, 2008. "VAR, SVAR and SVEC Models: Implementation Within R Package vars," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 27(i04).
    48. Andrea Bonaccorsi & Cinzia Daraio, 2003. "Age effects in scientific productivity," Scientometrics, Springer;Akadémiai Kiadó, vol. 58(1), pages 49-90, September.
    49. Barletta, Florencia & Yoguel, Gabriel & Pereira, Mariano & Rodríguez, Sergio, 2017. "Exploring scientific productivity and transfer activities: Evidence from Argentinean ICT research groups," Research Policy, Elsevier, vol. 46(8), pages 1361-1369.
    50. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    51. Waltman, Ludo & van Eck, Nees Jan, 2015. "Field-normalized citation impact indicators and the choice of an appropriate counting method," Journal of Informetrics, Elsevier, vol. 9(4), pages 872-894.
    52. Lilia Cavallari & Stefano D’Addona, 2014. "Trade Margins And Exchange Rate Regimes: New Evidence From A Panel Varx Model," Working Papers 0514, CREI Università degli Studi Roma Tre, revised 2014.
    53. Tomaz Bartol & Gordana Budimir & Doris Dekleva-Smrekar & Miro Pusnik & Primoz Juznic, 2014. "Assessment of research fields in Scopus and Web of Science in the view of national research evaluation in Slovenia," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1491-1504, February.
    54. Vincent Larivière & Benoit Macaluso & Éric Archambault & Yves Gingras, 2010. "Which scientific elites? On the concentration of research funds, publications and citations," Research Evaluation, Oxford University Press, vol. 19(1), pages 45-53, March.
    55. James H. Stock & Mark W. Watson, 2001. "Vector Autoregressions," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 101-115, Fall.
    56. Werner Marx & Lutz Bornmann, 2015. "On the causes of subject-specific citation rates in Web of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1823-1827, February.
    57. Wolfgang Karl Härdle & Lukas Borke, 2017. "GitHub API based QuantNet Mining infrastructure in R," SFB 649 Discussion Papers SFB649DP2017-008, Sonderforschungsbereich 649, Humboldt University, Berlin, Germany.
    58. Jung Cheol Shin & William K. Cummings, 2010. "Multilevel analysis of academic publishing across disciplines: research preference, collaboration, and time on research," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 581-594, November.
    59. Grit Laudel, 2005. "Is external research funding a valid indicator for research performance?," Research Evaluation, Oxford University Press, vol. 14(1), pages 27-34, April.
    60. Jian Wang, 2013. "Citation time window choice for research impact evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 851-872, March.
    61. Wildgaard, Lorna, 2016. "A critical cluster analysis of 44 indicators of author-level performance," Journal of Informetrics, Elsevier, vol. 10(4), pages 1055-1078.
    62. Fredrik Niclas Piro & Dag W. Aksnes & Kristoffer Rørstad, 2013. "A macro analysis of productivity differences across fields: Challenges in the measurement of scientific publishing," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 307-320, February.
    63. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2011. "Assessing the varying level of impact measurement accuracy as a function of the citation window length," Journal of Informetrics, Elsevier, vol. 5(4), pages 659-667.
    64. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2013. "Individual research performance: A proposal for comparing apples to oranges," Journal of Informetrics, Elsevier, vol. 7(2), pages 528-539.
    65. Fandel, Gunter, 2007. "On the performance of universities in North Rhine-Westphalia, Germany: Government's redistribution of funds judged using DEA efficiency measures," European Journal of Operational Research, Elsevier, vol. 176(1), pages 521-533, January.
    66. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alona Zharova & Wolfgang K. Härdle & Stefan Lessmann, 2017. "Is Scientific Performance a Function of Funds?," SFB 649 Discussion Papers SFB649DP2017-028, Sonderforschungsbereich 649, Humboldt University, Berlin, Germany.
    2. Andrea Bonaccorsi & Brigida Blasi & Carmela Anna Nappi & Sandra Romagnosi, 2022. "Quality of research as source and signal: revisiting the valorization process beyond substitution vs complementarity," The Journal of Technology Transfer, Springer, vol. 47(2), pages 407-434, April.
    3. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    4. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    5. Lars Herberholz & Berthold U. Wigger, 2020. "Efficiency of European Universities: A Comparison of Peers," CESifo Working Paper Series 8044, CESifo.
    6. Kazuki Nakajima & Kazuyuki Shudo & Naoki Masuda, 2023. "Higher-order rich-club phenomenon in collaborative research grant networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(4), pages 2429-2446, April.
    7. Lutz Bornmann & Klaus Wohlrabe, 2019. "Normalisation of citation impact in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 841-884, August.
    8. Herberholz, Lars & Wigger, Berthold U., 2021. "Efficiency of European universities: A comparison of peers," Research Policy, Elsevier, vol. 50(9).
    9. Marek Kwiek, 2020. "Internationalists and locals: international research collaboration in a resource-poor system," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 57-105, July.
    10. Leila Tahmooresnejad & Catherine Beaudry & Andrea Schiffauerova, 2015. "The role of public funding in nanotechnology scientific production: Where Canada stands in comparison to the United States," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 753-787, January.
    11. Marek Kwiek & Wojciech Roszka, 2022. "Academic vs. biological age in research on academic careers: a large-scale study with implications for scientifically developing systems," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3543-3575, June.
    12. Yin, Zhifeng & Liang, Zheng & Zhi, Qiang, 2018. "Does the concentration of scientific research funding in institutions promote knowledge output?," Journal of Informetrics, Elsevier, vol. 12(4), pages 1146-1159.
    13. Mike Thelwall & Nabeil Maflahi, 2020. "Academic collaboration rates and citation associations vary substantially between countries and fields," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(8), pages 968-978, August.
    14. Massimiliano Serati & Fausto Pacicco, 2018. "A proposal for a micro-territorial well-being index: the WIT," RIEDS - Rivista Italiana di Economia, Demografia e Statistica - The Italian Journal of Economic, Demographic and Statistical Studies, SIEDS Societa' Italiana di Economia Demografia e Statistica, vol. 72(3), pages 77-99, July-Sept.
    15. Nabil Amara & Réjean Landry & Norrin Halilem, 2015. "What can university administrators do to increase the publication and citation scores of their faculty members?," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(2), pages 489-530, May.
    16. Andrea Bonaccorsi & Luca Secondi, 2017. "The determinants of research performance in European universities: a large scale multilevel analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1147-1178, September.
    17. Rørstad, Kristoffer & Aksnes, Dag W., 2015. "Publication rate expressed by age, gender and academic position – A large-scale analysis of Norwegian academic staff," Journal of Informetrics, Elsevier, vol. 9(2), pages 317-333.
    18. Margaret E. Blume-Kohout, 2023. "The case of the interrupting funder: dynamic effects of R&D funding and patenting in U.S. universities," The Journal of Technology Transfer, Springer, vol. 48(4), pages 1221-1242, August.
    19. Bornmann, Lutz & Haunschild, Robin, 2016. "Citation score normalized by cited references (CSNCR): The introduction of a new citation impact indicator," Journal of Informetrics, Elsevier, vol. 10(3), pages 875-887.
    20. Marszk, Adam & Lechman, Ewa, 2019. "New technologies and diffusion of innovative financial products: Evidence on exchange-traded funds in selected emerging and developed economies," Journal of Macroeconomics, Elsevier, vol. 62(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:308:y:2023:i:1:p:353-368. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/eor .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.