IDEAS home Printed from https://ideas.repec.org/p/wbk/wbrwps/7612.html

Is random forest a superior methodology for predicting poverty ? an empirical assessment

Author

Listed:
  • Pave Sohnesen,Thomas
  • Stender,Niels

Abstract

Random forest is in many fields of research a common method for data driven predictions. Within economics and prediction of poverty, random forest is rarely used. Comparing out-of-sample predictions in surveys for same year in six countries shows that random forest is often more accurate than current common practice (multiple imputations with variables selected by stepwise and Lasso), suggesting that this method could contribute to better poverty predictions. However, none of the methods consistently provides accurate predictions of poverty over time, highlighting that technical model fitting by any method within a single year is not always, by itself, sufficient for accurate predictions of poverty over time.

Suggested Citation

  • Pave Sohnesen,Thomas & Stender,Niels, 2016. "Is random forest a superior methodology for predicting poverty ? an empirical assessment," Policy Research Working Paper Series 7612, The World Bank.
  • Handle: RePEc:wbk:wbrwps:7612
    as

    Download full text from publisher

    File URL: http://documents.worldbank.org/curated/en/777401467987858907/pdf/WPS7612.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Talip Kilic & Thomas Pave Sohnesen, 2019. "Same Question But Different Answer: Experimental Evidence on Questionnaire Design's Impact on Poverty Measured by Proxies," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 65(1), pages 144-165, March.
    2. Chris Elbers & Jean O. Lanjouw & Peter Lanjouw, 2003. "Micro--Level Estimation of Poverty and Inequality," Econometrica, Econometric Society, vol. 71(1), pages 355-364, January.
    3. Hai-Anh H. Dang & Peter F. Lanjouw & Umar Serajuddin, 2017. "Updating poverty estimates in the absence of regular and comparable consumption data: methods and illustration with reference to a middle-income country," Oxford Economic Papers, Oxford University Press, vol. 69(4), pages 939-962.
    4. Astrid Mathiassen, 2013. "Testing Prediction Performance of Poverty Models: Empirical Evidence from U ganda," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 59(1), pages 91-112, March.
    5. Luc Christiaensen & Peter Lanjouw & Jill Luoto & David Stifel, 2012. "Small area estimation-based prediction methods to track poverty: validation and applications," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 10(2), pages 267-297, June.
    6. Jennifer L. Castle & Xiaochuan Qin & W. Robert Reed, 2009. "How To Pick The Best Regression Equation: A Review And Comparison Of Model Selection Algorithms," Working Papers in Economics 09/13, University of Canterbury, Department of Economics and Finance.
    7. Calogero Carletto & Talip Kilic, 2011. "Moving Up the Ladder? The Impact of Migration Experience on Occupational Mobility in Albania," Journal of Development Studies, Taylor & Francis Journals, vol. 47(6), pages 846-869.
    8. Kasirye, Ibrahim & Ssewanyana, Sarah N., "undated". "Impacts and determinants of panel survey attrition: The case of Northern Uganda survey 2004-2008," Research Series 127536, Economic Policy Research Centre (EPRC).
    9. Hal R. Varian, 2014. "Big Data: New Tricks for Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 28(2), pages 3-28, Spring.
    10. Sudarno Sumarto & Daniel Suryadarma & Asep Suryahadi, 2007. "Predicting Consumption Poverty using Non-Consumption Indicators: Experiments using Indonesian Data," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 81(3), pages 543-578, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Al Kez, Dlzar & Foley, Aoife & Abdul, Zrar Khald & Del Rio, Dylan Furszyfer, 2024. "Energy poverty prediction in the United Kingdom: A machine learning approach," Energy Policy, Elsevier, vol. 184(C).
    2. Gianni Betti & Ruzhdie Bici & Laura Neri & Thomas Pave Sohnesen & Ledia Thomo, 2018. "Local Poverty and Inequality in Albania," Eastern European Economics, Taylor & Francis Journals, vol. 56(3), pages 223-245, May.
    3. Baez, Javier E. & Kshirsagar, Varun & Skoufias, Emmanuel, 2024. "Drought-sensitive targeting and child growth faltering in Southern Africa," World Development, Elsevier, vol. 182(C).
    4. Walter Sosa-Escudero & Maria Victoria Anauati & Wendy Brau, 2022. "Poverty, Inequality and Development Studies with Machine Learning," Advanced Studies in Theoretical and Applied Econometrics, in: Felix Chan & László Mátyás (ed.), Econometrics with Machine Learning, chapter 0, pages 291-335, Springer.
    5. Arranhado, Esmeralda & Barbosa, Lágida & Bastos, João A., 2025. "Multidimensional poverty in Benin," Socio-Economic Planning Sciences, Elsevier, vol. 99(C).
    6. Ratzanyel Rincón, 2023. "Quarterly multidimensional poverty estimates in Mexico using machine learning algorithms/Estimaciones trimestrales de pobreza multidimensional en México mediante algoritmos de aprendizaje de máquina," Estudios Económicos, El Colegio de México, Centro de Estudios Económicos, vol. 38(1), pages 3-68.
    7. Echevin, Damien & Fotso, Guy & Bouroubi, Yacine & Coulombe, Harold & Li, Qing, 2025. "Combining survey and census data for improved poverty prediction using semi-supervised deep learning," Journal of Development Economics, Elsevier, vol. 172(C).
    8. Thomas Pave Sohnesen & Peter Fisker & David Malmgren‐Hansen, 2022. "Using Satellite Data to Guide Urban Poverty Reduction," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 68(S2), pages 282-294, December.
    9. Peter Fisker & Jordi Gallego-Ayala & David Malmgren Hansen & Thomas Pave Sohnesen & Edmundo Murrugarra, 2022. "Guiding Social Protection Targeting Through Satellite Data in São Tomé and Príncipe," World Bank Publications - Reports 38222, The World Bank Group.
    10. Emmanuel A. Onsay & Jason Alinsunurin & Jomar F. Rabajante, 2025. "Optimizing machine learning algorithms for multidimensional poverty prediction in the Philippines," SN Business & Economics, Springer, vol. 5(10), pages 1-40, October.
    11. Beltramo, Theresa P. & Calvi, Rossella & De Giorgi, Giacomo & Sarr, Ibrahima, 2023. "Child poverty among refugees," World Development, Elsevier, vol. 171(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Astrid Mathiassen & Bjørn K. Getz Wold, 2021. "Predicting poverty trends by survey-to-survey imputation: the challenge of comparability," Oxford Economic Papers, Oxford University Press, vol. 73(3), pages 1153-1174.
    2. Hai‐Anh H. Dang & Talip Kilic & Kseniya Abanokova & Calogero Carletto, 2025. "Poverty Imputation in Contexts Without Consumption Data: A Revisit With Further Refinements," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 71(1), February.
    3. Luc Christiaensen & Ethan Ligon & Thomas Pave Sohnesen, 2022. "Consumption Subaggregates Should Not Be Used to Measure Poverty," The World Bank Economic Review, World Bank, vol. 36(2), pages 413-432.
    4. Dang, Hai-Anh H & Lanjouw, Peter F., 2021. "Data Scarcity and Poverty Measurement," IZA Discussion Papers 14631, IZA Network @ LISER.
    5. Hai-Anh H. Dang & Peter F. Lanjouw & Umar Serajuddin, 2017. "Updating poverty estimates in the absence of regular and comparable consumption data: methods and illustration with reference to a middle-income country," Oxford Economic Papers, Oxford University Press, vol. 69(4), pages 939-962.
    6. Talip Kilic & Thomas Pave Sohnesen, 2019. "Same Question But Different Answer: Experimental Evidence on Questionnaire Design's Impact on Poverty Measured by Proxies," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 65(1), pages 144-165, March.
    7. Ligon, Ethan & Christiaensen, Luc & Sohnesen, Thomas P, 2020. "Should Consumption Sub-Aggregates be Used to Measure Poverty?," Department of Agricultural & Resource Economics, UC Berkeley, Working Paper Series qt9b9929jh, Department of Agricultural & Resource Economics, UC Berkeley.
    8. Cuesta, Jose & Chagalj, Cristian, 2019. "Measuring poverty with administrative data in data deprived contexts: The case of Nicaragua," Economics Letters, Elsevier, vol. 183(C), pages 1-1.
    9. Dang, Hai-Anh H & Kilic, Talip & Hlasny, Vladimir & Abanokova, Kseniya & Carletto, Calogero, 2024. "Using Survey-to-Survey Imputation to Fill Poverty Data Gaps at a Low Cost: Evidence from a Randomized Survey Experiment," IZA Discussion Papers 16792, IZA Network @ LISER.
    10. Hai-Anh H. Dang & Peter F. Lanjouw, 2023. "Regression-based imputation for poverty measurement in data-scarce settings," Chapters, in: Jacques Silber (ed.), Research Handbook on Measuring Poverty and Deprivation, chapter 13, pages 141-150, Edward Elgar Publishing.
    11. Lain,Jonathan William & Schoch,Marta & Vishwanath,Tara, 2022. "Estimating a Poverty Trend for Nigeria between 2009 and 2019," Policy Research Working Paper Series 9974, The World Bank.
    12. Hai‐Anh Dang & Dean Jolliffe & Calogero Carletto, 2019. "Data Gaps, Data Incomparability, And Data Imputation: A Review Of Poverty Measurement Methods For Data‐Scarce Environments," Journal of Economic Surveys, Wiley Blackwell, vol. 33(3), pages 757-797, July.
    13. Hai‐Anh H. Dang, 2021. "To impute or not to impute, and how? A review of poverty‐estimation methods in the absence of consumption data," Development Policy Review, Overseas Development Institute, vol. 39(6), pages 1008-1030, November.
    14. Hai-Anh H. Dang & Paolo Verme, 2023. "Estimating poverty for refugees in data-scarce contexts: an application of cross-survey imputation," Journal of Population Economics, Springer;European Society for Population Economics, vol. 36(2), pages 653-679, April.
    15. Betti,Gianni & Molini,Vasco & Mori,Lorenzo, 2022. "New Algorithm to Estimate Inequality Measures in Cross-Survey Imputation : An Attemptto Correct the Underestimation of Extreme Values," Policy Research Working Paper Series 10013, The World Bank.
    16. Tanida Arayavechkit & Aziz Atamanov & Karen Y. Barreto Herrera & Nadia Belhaj Hassine Belghith & R. Andres Castaneda Aguilar & Tony H. M. J. Fujs & Reno Dewina & Carolina Diaz-Bonilla & Ifeanyi N. Edo, 2021. "March 2021 PovcalNet Update: What's New," Global Poverty Monitoring Technical Note Series 15, The World Bank.
    17. Brown, Caitlin & Ravallion, Martin & van de Walle, Dominique, 2018. "A poor means test? Econometric targeting in Africa," Journal of Development Economics, Elsevier, vol. 134(C), pages 109-124.
    18. Dang, Hai-Anh & Kilic, Talip & Abanokova, Kseniya, 2026. "Disaggregating Imputed Poverty Estimates by Population Groups: New Evidence from a Multi-Country Analysis," IZA Discussion Papers 18743, IZA Network @ LISER.
    19. Paolo Brunori & Pedro Salas-Rojo & Paolo Verme, 2022. "Estimating Inequality with Missing Incomes," Working Papers 616, ECINEQ, Society for the Study of Economic Inequality.
    20. Hai-Anh H. Dang & Peter F. Lanjouw, 2018. "Poverty Dynamics in India between 2004 and 2012: Insights from Longitudinal Analysis Using Synthetic Panel Data," Economic Development and Cultural Change, University of Chicago Press, vol. 67(1), pages 131-170.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wbk:wbrwps:7612. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Roula I. Yazigi (email available below). General contact details of provider: https://edirc.repec.org/data/dvewbus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.