IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v15y2022i16p5776-d883799.html
   My bibliography  Save this article

Well Construction Action Planning and Automation through Finite-Horizon Sequential Decision-Making

Author

Listed:
  • Gurtej Singh Saini

    (Cockrell School of Engineering, The University of Texas at Austin, Austin, TX 78712, USA)

  • Oney Erge

    (Cockrell School of Engineering, The University of Texas at Austin, Austin, TX 78712, USA)

  • Pradeepkumar Ashok

    (Cockrell School of Engineering, The University of Texas at Austin, Austin, TX 78712, USA)

  • Eric van Oort

    (Cockrell School of Engineering, The University of Texas at Austin, Austin, TX 78712, USA)

Abstract

Well construction operations require continuous complex decision-making and multi-step action planning. Action selection at every step demands a careful evaluation of the vast action space, while guided by long-term objectives and desired outcomes. Current human-centric decision-making introduces a degree of bias, which can result in reactive rather than proactive decisions. This can lead from minor operational inefficiencies all the way to catastrophic health and safety issues. This paper details the steps in structuring unbiased purpose-built sequential decision-making systems. Setting up such systems entails representing the operation as a Markov decision process (MDP). This requires explicitly defining states and action values, defining goal states, building a digital twin to model the process, and appropriately shaping reward functions to measure feedback. The digital twin, in conjunction with the reward function, is utilized for simulating and quantifying the different action sequences. A finite-horizon sequential decision-making system, with discrete state and action space, was set up to advise on hole cleaning during well construction. The state was quantified by the cuttings bed height and the equivalent circulation density values, and the action set was defined using a combination of controllable drilling parameters (including mud density and rheology, drillstring rotation speed, etc.). A non-sparse normalized reward structure was formulated as a function of the state and action values. Hydraulics, cuttings transport, and rig state detection models were integrated to build the hole cleaning digital twin. This system was then used for performance tracking and scenario simulations (with each scenario defined as a finite-horizon action sequence) on real-world oil wells. The different scenarios were compared by monitoring state–action transitions and the evolution of the reward with actions. This paper presents a novel method for setting up well construction operations as long-term finite-horizon sequential decision-making systems, and defines a way to quantify and compare different scenarios. The proper construction of such systems is a crucial step towards automating intelligent decision-making.

Suggested Citation

  • Gurtej Singh Saini & Oney Erge & Pradeepkumar Ashok & Eric van Oort, 2022. "Well Construction Action Planning and Automation through Finite-Horizon Sequential Decision-Making," Energies, MDPI, vol. 15(16), pages 1-28, August.
    • Handle: RePEc:gam:jeners:v:15:y:2022:i:16:p:5776-:d:883799
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/15/16/5776/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/15/16/5776/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. David Silver & Julian Schrittwieser & Karen Simonyan & Ioannis Antonoglou & Aja Huang & Arthur Guez & Thomas Hubert & Lucas Baker & Matthew Lai & Adrian Bolton & Yutian Chen & Timothy Lillicrap & Fan , 2017. "Mastering the game of Go without human knowledge," Nature, Nature, vol. 550(7676), pages 354-359, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Daníelsson, Jón & Macrae, Robert & Uthemann, Andreas, 2022. "Artificial intelligence and systemic risk," Journal of Banking & Finance, Elsevier, vol. 140(C).
    2. Yang, Zhengzhi & Zheng, Lei & Perc, Matjaž & Li, Yumeng, 2024. "Interaction state Q-learning promotes cooperation in the spatial prisoner's dilemma game," Applied Mathematics and Computation, Elsevier, vol. 463(C).
    3. Zhang, Yihao & Chai, Zhaojie & Lykotrafitis, George, 2021. "Deep reinforcement learning with a particle dynamics environment applied to emergency evacuation of a room with obstacles," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 571(C).
    4. Keller, Alexander & Dahm, Ken, 2019. "Integral equations and machine learning," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 161(C), pages 2-12.
    5. Canhoto, Ana Isabel & Clear, Fintan, 2020. "Artificial intelligence and machine learning as business tools: A framework for diagnosing value destruction potential," Business Horizons, Elsevier, vol. 63(2), pages 183-193.
    6. Zhaobin Mo & Xuan Di & Rongye Shi, 2023. "Robust Data Sampling in Machine Learning: A Game-Theoretic Framework for Training and Validation Data Selection," Games, MDPI, vol. 14(1), pages 1-13, January.
    7. Yang, Kaiyuan & Huang, Houjing & Vandans, Olafs & Murali, Adithya & Tian, Fujia & Yap, Roland H.C. & Dai, Liang, 2023. "Applying deep reinforcement learning to the HP model for protein structure prediction," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 609(C).
    8. Yifeng Guo & Xingyu Fu & Yuyan Shi & Mingwen Liu, 2018. "Robust Log-Optimal Strategy with Reinforcement Learning," Papers 1805.00205, arXiv.org.
    9. Xueqing Yan & Yongming Li, 2023. "A Novel Discrete Differential Evolution with Varying Variables for the Deficiency Number of Mahjong Hand," Mathematics, MDPI, vol. 11(9), pages 1-21, May.
    10. Jianjun Chen & Weihao Hu & Di Cao & Bin Zhang & Qi Huang & Zhe Chen & Frede Blaabjerg, 2019. "An Imbalance Fault Detection Algorithm for Variable-Speed Wind Turbines: A Deep Learning Approach," Energies, MDPI, vol. 12(14), pages 1-15, July.
    11. Andrew G. Haldane & Arthur E. Turrell, 2019. "Drawing on different disciplines: macroeconomic agent-based models," Journal of Evolutionary Economics, Springer, vol. 29(1), pages 39-66, March.
    12. Antonopoulos, Ioannis & Robu, Valentin & Couraud, Benoit & Kirli, Desen & Norbu, Sonam & Kiprakis, Aristides & Flynn, David & Elizondo-Gonzalez, Sergio & Wattam, Steve, 2020. "Artificial intelligence and machine learning approaches to energy demand-side response: A systematic review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 130(C).
    13. Lu Wang & Wenqing Ai & Tianhu Deng & Zuo‐Jun M. Shen & Changjing Hong, 2020. "Optimal production ramp‐up in the smartphone manufacturing industry," Naval Research Logistics (NRL), John Wiley & Sons, vol. 67(8), pages 685-704, December.
    14. Karwowski, Jan & Mańdziuk, Jacek, 2019. "A Monte Carlo Tree Search approach to finding efficient patrolling schemes on graphs," European Journal of Operational Research, Elsevier, vol. 277(1), pages 255-268.
    15. Young Joon Park & Yoon Sang Cho & Seoung Bum Kim, 2019. "Multi-agent reinforcement learning with approximate model learning for competitive games," PLOS ONE, Public Library of Science, vol. 14(9), pages 1-20, September.
    16. Hai Wang & Shengnan Chen, 2023. "Insights into the Application of Machine Learning in Reservoir Engineering: Current Developments and Future Trends," Energies, MDPI, vol. 16(3), pages 1-11, January.
    17. Morato, P.G. & Andriotis, C.P. & Papakonstantinou, K.G. & Rigo, P., 2023. "Inference and dynamic decision-making for deteriorating systems with probabilistic dependencies through Bayesian networks and deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    18. Yanning Ge & Tao Huang & Xiaoding Wang & Guolong Zheng & Xu Yang, 2024. "The Application of Residual Connection-Based State Normalization Method in GAIL," Mathematics, MDPI, vol. 12(2), pages 1-14, January.
    19. Iwao Maeda & David deGraw & Michiharu Kitano & Hiroyasu Matsushima & Hiroki Sakaji & Kiyoshi Izumi & Atsuo Kato, 2020. "Deep Reinforcement Learning in Agent Based Financial Market Simulation," JRFM, MDPI, vol. 13(4), pages 1-17, April.
    20. Peter Eastman & Jade Shi & Bharath Ramsundar & Vijay S Pande, 2018. "Solving the RNA design problem with reinforcement learning," PLOS Computational Biology, Public Library of Science, vol. 14(6), pages 1-15, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:15:y:2022:i:16:p:5776-:d:883799. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.