IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2606.14386.html

Discovery under Hypothesis Redundancy: A Geometric Theory of Discovery Bottlenecks

Author

Listed:
  • Li Xia
  • Baoxun Wang

Abstract

Scientific discovery saturates when new hypotheses cease to provide independent information, even if the nominal hypothesis space remains large. We study hybrid discovery systems that combine structured local search with LLM-generated non-local proposals and pose the Search Compression Hypothesis: non-local exploration helps only when three geometric conditions co-occur: spectral compression, orthogonal escape from the explored span, and residual signal alignment with the target. We formalize these conditions, derive necessary conditions for hybrid advantage, and test the mechanism in controlled synthetic environments, large-scale A-share factor discovery, and symbolic-regression benchmarks; a public tabular operational sanity check tests the associated budget-allocation implication. Signal-planting and directed-versus-random experiments show that novelty alone is insufficient: random orthogonal jumps expand coverage but do not improve yield without predictive alignment. Across compression sweeps, real factor archives, and LLM-SRBench tasks, hybrid gains concentrate in weakly represented but target-bearing directions and vanish as the hypothesis space approaches full rank. The framework turns LLM-guided discovery from generic novelty search into a diagnostic procedure for deciding when directed non-local exploration is warranted.

Suggested Citation

  • Li Xia & Baoxun Wang, 2026. "Discovery under Hypothesis Redundancy: A Geometric Theory of Discovery Bottlenecks," Papers 2606.14386, arXiv.org.
  • Handle: RePEc:arx:papers:2606.14386
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2606.14386
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    2. Guanhao Feng & Stefano Giglio & Dacheng Xiu, 2020. "Taming the Factor Zoo: A Test of New Factors," Journal of Finance, American Finance Association, vol. 75(3), pages 1327-1370, June.
    3. Jianqing Fan & Yingying Fan & Xiao Han & Jinchi Lv, 2022. "Asymptotic Theory of Eigenvectors for Random Matrices With Diverging Spikes," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 117(538), pages 996-1009, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cujean, Julien & Andrei, Daniel & Fournier, Mathieu, 2019. "The Low-Minus-High Portfolio and the Factor Zoo," CEPR Discussion Papers 14153, Centre for Economic Policy Research.
    2. Dichtl, Hubert & Drobetz, Wolfgang & Neuhierl, Andreas & Wendt, Viktoria-Sophie, 2021. "Data snooping in equity premium prediction," International Journal of Forecasting, Elsevier, vol. 37(1), pages 72-94.
    3. Hadi Movaghari & Serafeim Tsoukas & Evangelos Vagenas‐Nanos, 2025. "Corporate cash policy and double machine learning," International Journal of Finance & Economics, John Wiley & Sons, Ltd., vol. 30(3), pages 3261-3279, July.
    4. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    5. Söhnke M. Bartram & Harald Lohre & Peter F. Pope & Ananthalakshmi Ranganathan, 2021. "Navigating the factor zoo around the world: an institutional investor perspective," Journal of Business Economics, Springer, vol. 91(5), pages 655-703, July.
    6. David M. Ritzwoller & Joseph P. Romano, 2019. "Uncertainty in the Hot Hand Fallacy: Detecting Streaky Alternatives to Random Bernoulli Sequences," Papers 1908.01406, arXiv.org, revised Apr 2021.
    7. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    8. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    9. Rute M. Caeiro & Pedro C. Vicente, 2020. "Knowledge of vitamin A deficiency and crop adoption: Evidence from a field experiment in Mozambique," Agricultural Economics, International Association of Agricultural Economists, vol. 51(2), pages 175-190, March.
    10. Brenøe, Anne Ardila & Eyibak, Zeynep & Heursen, Lea & Ranehill, Eva & Weber, Roberto A., 2024. "Gender Identity and Economic Decision Making," IZA Discussion Papers 17564, IZA Network @ LISER.
    11. Orazio Attanasio & Sarah Cattan & Emla Fitzsimons & Costas Meghir & Marta Rubio-Codina, 2020. "Estimating the Production Function for Human Capital: Results from a Randomized Controlled Trial in Colombia," American Economic Review, American Economic Association, vol. 110(1), pages 48-85, January.
    12. Meghir, Costas & Palme, Mårten & Schnabel, Marieke, 2011. "The Effect of Education Policy on Crime: An Intergenerational Perspective," Research Papers in Economics 2011:23, Stockholm University, Department of Economics.
    13. Bhalotra, Sonia & Clarke, Damian & Mühlrad, Hanna & Palme, Mårten, 2021. "Health and Labor Market Impacts of Twin Birth : Evidence from a Swedish IVF Policy Mandate," The Warwick Economics Research Paper Series (TWERPS) 1391, University of Warwick, Department of Economics.
    14. Bajgrowicz, Pierre & Scaillet, Olivier, 2012. "Technical trading revisited: False discoveries, persistence tests, and transaction costs," Journal of Financial Economics, Elsevier, vol. 106(3), pages 473-491.
    15. Jaschke Philipp & Sulin Sardoschau & Marco Tabellini, 2021. "Scared Straight? Threat and Assimilation of Refugees in Germany," RFBerlin Discussion Paper Series 2136, ROCKWOOL Foundation Berlin (RFBerlin).
    16. Bakalli, Gaetan & Guerrier, Stéphane & Scaillet, Olivier, 2023. "A penalized two-pass regression to predict stock returns with time-varying risk premia," Journal of Econometrics, Elsevier, vol. 237(2).
    17. Grácio, Matilde & Vicente, Pedro C., 2021. "Information, get-out-the-vote messages, and peer influence: Causal effects on political behavior in Mozambique," Journal of Development Economics, Elsevier, vol. 151(C).
    18. Pierfrancesco Rolla & Patricia Justino, 2022. "The social consequences of organized crime in Italy," WIDER Working Paper Series wp-2022-106, World Institute for Development Economic Research (UNU-WIDER).
    19. Pablo Muñoz & Mounu Prem, 2024. "Managers' Productivity and Recruitment in the Public Sector," American Economic Journal: Economic Policy, American Economic Association, vol. 16(4), pages 223-253, November.
    20. Chicu, Mark & Masten, Matthew A., 2013. "A specification test for discrete choice models," Economics Letters, Elsevier, vol. 121(2), pages 336-339.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2606.14386. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.