An Event-Aware Hybrid Data Partitioning Algorithm with Metadata-Indexed Caching for Mitigating Data Skew in Distributed Training on Large-Scale Neural Electrophysiology Datasets

Distributed training on large-scale neural electrophysiology archives faces severe data skew because spike rates, event densities, and trial durations vary substantially across recording sessions and brain regions. Default hash sharding produces partition imbalances that incur GPU idle time, while uniform time-window partitioning fails to align with biologically meaningful events. This paper presents an event-aware hybrid data-partitioning algorithm coupled with a metadata-indexed caching scheme tailored for standardized neural recording archives such as the Distributed Archives for Neurophysiology Data Integration (DANDI), the International Brain Laboratory Brain-Wide Map, and the Allen Brain Observatory. The algorithm controls partition granularity using event-rate statistics extracted from Neurodata Without Borders metadata, combines fixed time-window slicing with event-triggered slicing to reduce skew ratio, and exploits an event-id-to-chunk-offset index for fast cache lookup. An adaptive granularity adjustment mechanism driven by loss-decay feedback is also introduced. Evaluation on three publicly available neural datasets, comprising 459 IBL Neuropixels sessions, the Allen Brain Observatory Visual Coding release, and selected DANDI dandisets, indicates that the proposed scheme reduces skew ratio from 3.41 to 1.27, raises cache hit rate from 41.2% to 78.6%, and improves sustained GPU utilization from 62.4% to 88.9% relative to hash-sharding baselines under an MLPerf-style storage benchmark setup.