IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2512.19843.html

Numerical Analysis of Test Optimality

Author

Listed:
  • Philipp Ketz
  • Adam McCloskey
  • Jan Scherer

Abstract

In nonstandard testing environments, researchers often derive ad hoc tests with correct (asymptotic) size, but their optimality properties are typically unknown a priori and difficult to assess. This paper develops a numerical framework for determining whether an ad hoc test is effectively optimal - approximately maximizing a weighted average power criterion for some weights over the alternative and attaining a power envelope generated by a single weighted average power-maximizing test. Our approach uses nested optimization algorithms to approximate the weight function that makes an ad hoc test's weighted average power as close as possible to that of a true weighted average power-maximizing test, and we show the surprising result that the rejection probabilities corresponding to the latter form an approximate power envelope for the former. We provide convergence guarantees, discuss practical implementation and apply the method to the weak instrument-robust conditional likelihood ratio test and a recently-proposed test for when a nuisance parameter may be on or near its boundary.

Suggested Citation

  • Philipp Ketz & Adam McCloskey & Jan Scherer, 2025. "Numerical Analysis of Test Optimality," Papers 2512.19843, arXiv.org.
    • Handle: RePEc:arx:papers:2512.19843
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2512.19843
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ketz, Philipp, 2018. "Subvector inference when the true parameter vector may be near or at the boundary," Journal of Econometrics, Elsevier, vol. 207(2), pages 285-306.
    2. Marcelo J. Moreira & Mahrad Sharifvaghefi & Geert Ridder, 2017. "Optimal Invariant Tests in an Instrumental Variables Regression With Heteroskedastic and Autocorrelated Errors," Papers 1705.00231, arXiv.org, revised Aug 2021.
    3. Graham Elliott & Ulrich K. Müller & Mark W. Watson, 2015. "Nearly Optimal Tests When a Nuisance Parameter Is Present Under the Null Hypothesis," Econometrica, Econometric Society, vol. 83, pages 771-811, March.
    4. Philipp Ketz & Adam McCloskey, 2025. "Short and Simple Confidence Intervals When the Directions of Some Effects are Known," The Review of Economics and Statistics, MIT Press, vol. 107(3), pages 820-834, May.
    5. Campbell, John Y. & Yogo, Motohiro, 2006. "Efficient tests of stock return predictability," Journal of Financial Economics, Elsevier, vol. 81(1), pages 27-60, July.
    6. Giuseppe Cavaliere & Adam McCloskey & Rasmus S. Pedersen & Anders Rahbek, 2025. "Uniform Critical Values for Likelihood Ratio Tests in Boundary Problems," Papers 2507.19603, arXiv.org.
    7. Andrews, Donald W.K. & Moreira, Marcelo J. & Stock, James H., 2008. "Efficient two-sided nonsimilar invariant tests in IV regression with weak instruments," Journal of Econometrics, Elsevier, vol. 146(2), pages 241-254, October.
    8. Donald W. K. Andrews & Patrik Guggenberger, 2015. "Identification- and Singularity-Robust Inference for Moment Condition," Cowles Foundation Discussion Papers 1978, Cowles Foundation for Research in Economics, Yale University.
    9. Frank Kleibergen, 2002. "Pivotal Statistics for Testing Structural Parameters in Instrumental Variables Regression," Econometrica, Econometric Society, vol. 70(5), pages 1781-1803, September.
    10. Van de Sijpe, Nicolas & Windmeijer, Frank, 2023. "On the power of the conditional likelihood ratio and related tests for weak-instrument robust inference," Journal of Econometrics, Elsevier, vol. 235(1), pages 82-104.
    11. Donald W. K. Andrews & Marcelo J. Moreira & James H. Stock, 2006. "Optimal Two-Sided Invariant Similar Tests for Instrumental Variables Regression," Econometrica, Econometric Society, vol. 74(3), pages 715-752, May.
    12. Isaiah Andrews & Anna Mikusheva, 2016. "Conditional Inference With a Functional Nuisance Parameter," Econometrica, Econometric Society, vol. 84(4), pages 1571-1612, July.
    13. Gregory Fletcher Cox, 2024. "A Simple and Adaptive Confidence Interval when Nuisance Parameters Satisfy an Inequality," Papers 2409.09962, arXiv.org.
    14. Yurii Nesterov, 2018. "Lectures on Convex Optimization," Springer Optimization and Its Applications, Springer, edition 2, number 978-3-319-91578-4, March.
    15. Moreira, Humberto & Moreira, Marcelo J., 2019. "Optimal two-sided tests for instrumental variables regression with heteroskedastic and autocorrelated errors," Journal of Econometrics, Elsevier, vol. 213(2), pages 398-433.
    16. Andrews, Donald W K & Ploberger, Werner, 1994. "Optimal Tests When a Nuisance Parameter Is Present Only under the Alternative," Econometrica, Econometric Society, vol. 62(6), pages 1383-1414, November.
    17. Montiel Olea, José Luis, 2020. "Admissible, Similar Tests: A Characterization," Econometric Theory, Cambridge University Press, vol. 36(2), pages 347-366, April.
    18. Isaiah Andrews & Anna Mikusheva, 2016. "Conditional Inference With a Functional Nuisance Parameter," Econometrica, Econometric Society, vol. 84, pages 1571-1612, July.
    19. Donald W. K. Andrews & Patrik Guggenberger, 2019. "Identification‐ and singularity‐robust inference for moment condition models," Quantitative Economics, Econometric Society, vol. 10(4), pages 1703-1746, November.
    20. Moreira, Humberto Ataíde & Moreira, Marcelo J., 2013. "Contributions to the Theory of Optimal Tests," FGV EPGE Economics Working Papers (Ensaios Economicos da EPGE) 747, EPGE Brazilian School of Economics and Finance - FGV EPGE (Brazil).
    21. Elliott, Graham & Müller, Ulrich K., 2014. "Pre and post break parameter inference," Journal of Econometrics, Elsevier, vol. 180(2), pages 141-157.
    22. Andrews, Donald W.K. & Guggenberger, Patrik, 2010. "Applications of subsampling, hybrid, and size-correction methods," Journal of Econometrics, Elsevier, vol. 158(2), pages 285-305, October.
    23. Patrik Guggenberger & Frank Kleibergen & Sophocles Mavroeidis, 2019. "A more powerful subvector Anderson Rubin test in linear instrumental variables regression," Quantitative Economics, Econometric Society, vol. 10(2), pages 487-526, May.
    24. Marcelo J. Moreira, 2003. "A Conditional Likelihood Ratio Test for Structural Models," Econometrica, Econometric Society, vol. 71(4), pages 1027-1048, July.
    25. Donald W. K. Andrews & Vadim Marmer & Zhengfei Yu, 2019. "On optimal inference in the linear IV model," Quantitative Economics, Econometric Society, vol. 10(2), pages 457-485, May.
    26. repec:hal:journl:halshs-05162918 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Antoine, Bertille & Lavergne, Pascal, 2023. "Identification-robust nonparametric inference in a linear IV model," Journal of Econometrics, Elsevier, vol. 235(1), pages 1-24.
    2. Andrews, Donald W.K. & Cheng, Xu & Guggenberger, Patrik, 2020. "Generic results for establishing the asymptotic size of confidence sets and tests," Journal of Econometrics, Elsevier, vol. 218(2), pages 496-531.
    3. Kleibergen, Frank, 2021. "Efficient size correct subset inference in homoskedastic linear instrumental variables regression," Journal of Econometrics, Elsevier, vol. 221(1), pages 78-96.
    4. Wang, Wenjie, 2021. "Wild Bootstrap for Instrumental Variables Regression with Weak Instruments and Few Clusters," MPRA Paper 106227, University Library of Munich, Germany.
    5. Horowitz, Joel L., 2021. "Bounding the difference between true and nominal rejection probabilities in tests of hypotheses about instrumental variables models," Journal of Econometrics, Elsevier, vol. 222(2), pages 1057-1082.
    6. Dennis Lim & Wenjie Wang & Yichong Zhang, 2022. "A Conditional Linear Combination Test with Many Weak Instruments," Papers 2207.11137, arXiv.org, revised Apr 2023.
    7. Lim, Dennis & Wang, Wenjie & Zhang, Yichong, 2024. "A conditional linear combination test with many weak instruments," Journal of Econometrics, Elsevier, vol. 238(2).
    8. Dennis Lim & Wenjie Wang & Yichong Zhang, 2024. "A Dimension-Agnostic Bootstrap Anderson-Rubin Test For Instrumental Variable Regressions," Papers 2412.01603, arXiv.org, revised Sep 2025.
    9. Wenjie Wang & Yichong Zhang, 2021. "Wild Bootstrap for Instrumental Variables Regressions with Weak and Few Clusters," Papers 2108.13707, arXiv.org, revised Jan 2024.
    10. Marcelo Moreira & Geert Ridder, 2019. "Efficiency loss of asymptotically efficient tests in an instrumental variables regression," CeMMAP working papers CWP03/19, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    11. Johannes W. Ligtenberg, 2023. "Inference in clustered IV models with many and weak instruments," Papers 2306.08559, arXiv.org, revised Oct 2025.
    12. Moreira, Humberto & Moreira, Marcelo J., 2019. "Optimal two-sided tests for instrumental variables regression with heteroskedastic and autocorrelated errors," Journal of Econometrics, Elsevier, vol. 213(2), pages 398-433.
    13. Cox, Gregory Fletcher, 2025. "Weak identification with bounds in a class of minimum distance models," Journal of Econometrics, Elsevier, vol. 252(PA).
    14. Xu Cheng & Winston Wei Dou & Zhipeng Liao, 2022. "Macro‐Finance Decoupling: Robust Evaluations of Macro Asset Pricing Models," Econometrica, Econometric Society, vol. 90(2), pages 685-713, March.
    15. Bertille Antoine & Pascal Lavergne, 2020. "Identification-Robust Nonparametric Interference in a Linear IV Model," Discussion Papers dp20-03, Department of Economics, Simon Fraser University.
    16. Purevdorj Tuvaandorj, 2021. "Robust Permutation Tests in Linear Instrumental Variables Regression," Papers 2111.13774, arXiv.org, revised Jul 2024.
    17. Wang, Wenjie & Zhang, Yichong, 2024. "Wild bootstrap inference for instrumental variables regressions with weak and few clusters," Journal of Econometrics, Elsevier, vol. 241(1).
    18. Patrik Guggenberger & Frank Kleibergen & Sophocles Mavroeidis, 2021. "A Powerful Subvector Anderson Rubin Test in Linear Instrumental Variables Regression with Conditional Heteroskedasticity," Papers 2103.11371, arXiv.org, revised Oct 2022.
    19. Van de Sijpe, Nicolas & Windmeijer, Frank, 2023. "On the power of the conditional likelihood ratio and related tests for weak-instrument robust inference," Journal of Econometrics, Elsevier, vol. 235(1), pages 82-104.
    20. McCloskey, Adam, 2017. "Bonferroni-based size-correction for nonstandard testing problems," Journal of Econometrics, Elsevier, vol. 200(1), pages 17-35.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2512.19843. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.