IDEAS home Printed from https://ideas.repec.org/a/vrs/offsta/v38y2022i4p1235-1251n1.html
   My bibliography  Save this article

A User-Driven Method for Using Research Products to Empirically Assess Item Importance in National Surveys

Author

Listed:
  • Ong Ai Rene
  • Schultz Robert
  • West Brady T
  • Wagner James

    (Institute for Social Research, University of Michigan, 426 Thompson St, Ann Arbor, MI 48104, U.S.A.)

  • Sinozich Sofi

    (University of Maryland, College Park, MD 20742, U.S.A.)

  • Sinibaldi Jennifer
  • Finamore John

    (The National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation, 2415 Eisenhower Avenue, Alexandria, Virginia 22314, U.S.A.)

Abstract

Large-scale, nationally representative surveys serve many vital functions, but these surveys can be long and burdensome for respondents. Cutting survey length can help to reduce respondent burden and may improve data quality but removing items from these surveys is not a trivial matter. We propose a method to empirically assess item importance and associated burden in national surveys and guide this decision-making process using different research products produced from such surveys. This method is demonstrated using the Survey of Doctorate Recipients (SDR), a biennial survey administered to individuals with a science, engineering, and health doctorate. We used three main sources of information on the SDR variables: a bibliography of documents using the SDR data as a measure of item use and importance, SDR data table download statistics from the Scientists and Engineers Statistical Data System as an additional measure of item use, and web timing paradata and break-off rates as a measure of burden. Putting this information together, we identified 35 unused items (17% of the survey) and found that the most burdensome items are highly important. We conclude with general recommendations for those hoping to employ similar methodologies in the future.

Suggested Citation

  • Ong Ai Rene & Schultz Robert & West Brady T & Wagner James & Sinozich Sofi & Sinibaldi Jennifer & Finamore John, 2022. "A User-Driven Method for Using Research Products to Empirically Assess Item Importance in National Surveys," Journal of Official Statistics, Sciendo, vol. 38(4), pages 1235-1251, December.
    • Handle: RePEc:vrs:offsta:v:38:y:2022:i:4:p:1235-1251:n:1
    DOI: 10.2478/jos-2022-0052
    as

    Download full text from publisher

    File URL: https://doi.org/10.2478/jos-2022-0052
    Download Restriction: no
    File URL: https://libkey.io/10.2478/jos-2022-0052?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Elisabeth Deutskens & Ko de Ruyter & Martin Wetzels & Paul Oosterveld, 2004. "Response Rate and Response Quality of Internet-Based Surveys: An Experimental Study," Marketing Letters, Springer, vol. 15(1), pages 21-36, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Srinivasan, V. Seenu & Netzer, Oded, 2007. "Adaptive Self-Explication of Multi-attribute Preferences," Research Papers 1979, Stanford University, Graduate School of Business.
    2. Tatsushi Fukaya & Masayuki Suzuki & Ikumi Ozawa & Takumi Nakagoshi, 2022. "An Examination of Related Factors of Mathematical Pedagogical Content Knowledge in Elementary School Teachers: Focusing on Conceptions of Teaching and Learning and Test Utilization Strategy," SAGE Open, , vol. 12(4), pages 21582440221, October.
    3. Lusk, Jayson L., 2012. "The political ideology of food," Food Policy, Elsevier, vol. 37(5), pages 530-542.
    4. Rolf Becker, 2023. "Short- and long-term effects of reminders on panellists’ survey participation in a probability-based panel study with a sequential mixed-mode design," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(5), pages 4095-4119, October.
    5. Vahid Sobhani & Mohammadjavad Rostamizadeh & Seyed Morteza Hosseini & Seyed Ebrahim Hashemi & Ignacio Refoyo Román & Daniel Mon-López, 2022. "Anthropometric, Physiological, and Psychological Variables That Determine the Elite Pistol Performance of Women," IJERPH, MDPI, vol. 19(3), pages 1-10, January.
    6. Barbara Ryan & Rachel King, 2020. "How ready is ready? Measuring physical preparedness for severe storms," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 104(1), pages 171-199, October.
    7. Yang Liu & Jan Hannig & Abhishek Pal Majumder, 2019. "Second-Order Probability Matching Priors for the Person Parameter in Unidimensional IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 84(3), pages 701-718, September.
    8. Elisabeth Deutskens & Ad Jong & Ko Ruyter & Martin Wetzels, 2006. "Comparing the generalizability of online and mail surveys in cross-national service quality research," Marketing Letters, Springer, vol. 17(2), pages 119-136, April.
    9. Giancarlo Condello & Laura Capranica & Mojca Doupona & Kinga Varga & Verena Burk, 2019. "Dual-career through the elite university student-athletes’ lenses: The international FISU-EAS survey," PLOS ONE, Public Library of Science, vol. 14(10), pages 1-18, October.
    10. María del Mar Serrano-Arcos & Raquel Sánchez-Fernández & Juan Carlos Pérez-Mesa, 2021. "Analysis of Product-Country Image from Consumer’s Perspective: The Impact of Subjective Knowledge, Perceived Risk and Media Influence," Sustainability, MDPI, vol. 13(4), pages 1-25, February.
    11. Björn Hofman & Gerdien de Vries & Geerten van de Kaa, 2022. "Keeping Things as They Are: How Status Quo Biases and Traditions along with a Lack of Information Transparency in the Building Industry Slow Down the Adoption of Innovative Sustainable Technologies," Sustainability, MDPI, vol. 14(13), pages 1-20, July.
    12. Bader, Benjamin & Berg, Nicola & Holtbrügge, Dirk, 2015. "Expatriate performance in terrorism-endangered countries: The role of family and organizational support," International Business Review, Elsevier, vol. 24(5), pages 849-860.
    13. Wells, Victoria K. & Manika, Danae & Gregory-Smith, Diana & Taheri, Babak & McCowlen, Clair, 2015. "Heritage tourism, CSR and the role of employee environmental behaviour," Tourism Management, Elsevier, vol. 48(C), pages 399-413.
    14. Selmi, Noureddine & Chaney, Damien, 2018. "A measure of revenue management orientation and its mediating role in the relationship between market orientation and performance," Journal of Business Research, Elsevier, vol. 89(C), pages 99-109.
    15. Toepoel, V. & Das, J.W.M. & van Soest, A.H.O., 2005. "Design of Web Questionnaires : A Test for Number of Items per Screen," Discussion Paper 2005-114, Tilburg University, Center for Economic Research.
    16. Olivier Toubia & Eric Johnson & Theodoros Evgeniou & Philippe Delquié, 2013. "Dynamic Experiments for Estimating Preferences: An Adaptive Method of Eliciting Time and Risk Parameters," Management Science, INFORMS, vol. 59(3), pages 613-640, June.
    17. Philip X Fuchs & Mojca Doupona & Kinga Varga & Marta Bon & Cristina Cortis & Andrea Fusco & Loriana Castellani & Niko Niemisalo & Heikki Hannola & Patrice Giron & Jörg Förster & Laura Capranica & Herb, 2021. "Multi-national perceptions on challenges, opportunities, and support structures for Dual Career migrations in European student-athletes," PLOS ONE, Public Library of Science, vol. 16(6), pages 1-18, June.
    18. Zsombor Szádoczki & Sándor Bozóki & Patrik Juhász & Sergii V. Kadenko & Vitaliy Tsyganok, 2023. "Incomplete pairwise comparison matrices based on graphs with average degree approximately 3," Annals of Operations Research, Springer, vol. 326(2), pages 783-807, July.
    19. Elena-Loreni Baciu & Delia Vîrgă & Theofild-Andrei Lazăr & Delia Gligor & Cecilia-Nicoleta Jurcuț, 2020. "The Association between Entrepreneurial Perceived Behavioral Control, Personality, Empathy, and Assertiveness in a Romanian Sample of Nascent Entrepreneurs," Sustainability, MDPI, vol. 12(24), pages 1-16, December.
    20. Heider, Raphael & Moeller, Sabine, 2012. "Outlet patronage in on-the-go consumption: An analysis of patronage preference drivers for convenience outlets versus traditional retail outlets," Journal of Retailing and Consumer Services, Elsevier, vol. 19(3), pages 313-324.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:vrs:offsta:v:38:y:2022:i:4:p:1235-1251:n:1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Golla (email available below). General contact details of provider: https://www.sciendo.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.