IDEAS home Printed from https://ideas.repec.org/a/das/njaigs/v8y2025i02p425-431id465.html

AI Optimized Parallel Storage Orchestration for High Performance Computing Workloads in Hybrid Cloud Environments

Author

Listed:
  • Rahiman Shareef

Abstract

High Performance Computing workloads generate massive parallel input and output operations that place extreme demands on storage subsystems. Modern HPC environments increasingly integrate on-premises clusters with cloud infrastructure, creating hybrid architectures where data must move efficiently across heterogeneous storage tiers. Traditional parallel file systems rely on static striping, predefined caching policies, and manual performance tuning. These approaches are insufficient for dynamic workloads such as distributed AI training, simulation bursts, and checkpoint-intensive applications. This paper proposes an AI optimized parallel storage orchestration framework designed to enhance performance and resource utilization for HPC workloads in hybrid cloud environments. The framework monitors IO patterns, predicts workload behavior using lightweight machine learning models, and dynamically adjusts striping configurations, burst buffer usage, and cloud spillover policies. By introducing an intelligent orchestration layer above existing storage systems, the proposed approach improves adaptability without requiring modification of underlying file system architectures. A case study based evaluation demonstrates improved throughput stability during checkpoint storms, reduced IO contention during multi-tenant workloads, and better utilization of hybrid cloud storage resources compared to static configurations. The results indicate that AI driven orchestration can significantly enhance performance efficiency and scalability in next generation HPC storage systems.

Suggested Citation

  • Rahiman Shareef, 2025. "AI Optimized Parallel Storage Orchestration for High Performance Computing Workloads in Hybrid Cloud Environments," Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023, Open Knowledge, vol. 8(02), pages 425-431.
  • Handle: RePEc:das:njaigs:v:8:y:2025:i:02:p:425-431:id:465
    as

    Download full text from publisher

    File URL: https://newjaigs.com/index.php/JAIGS/article/view/465
    Download Restriction: no
    ---><---

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:das:njaigs:v:8:y:2025:i:02:p:425-431:id:465. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Open Knowledge (email available below). General contact details of provider: https://newjaigs.com/index.php/JAIGS/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.