Reasoning the Runtime Overhead of Profiling Tools

Modern high-performance computing (HPC) depends on an ever-evolving hardware landscape. Supercomputers, typically composed of hundreds to thousands of heterogeneous computing units, are further complicated by a variety of available memory and storage architectures. Consequently, efficient HPC-oriented parallel data processing and computation are becoming increasingly complex, even for experienced users. To address this important challenge and streamline the deployment of modern HPC resources, academia and industry have created various performance analysis and profiling tools. The aim of these tools is to collect information on program execution, enabling informed decisions and program optimization. A recent study naively applied a variety of profiling tools on well-known MPI-centric HPC proxy applications and compared the generated runtime overhead, memory consumption, and call path data. Results demonstrated that instrumentation-based tools, like Score-P and TAU, have limitations. We reproduced the experiments and identified the causes for these disadvantages. First, we enhanced libunwind, responsible for collecting backtraces from a signal handler, to make it more reliable. These enhancements enabled Score-P to produce results for all experiment configurations, but also improved the overhead for one benchmark. Second, we show that a proxy application exercises a common MPI communication pattern that induces a high overhead for instrumentation-based tools. We re-ran all of the experiments using a version of this proxy benchmark that was adjusted for ORNL’s Frontier system. This adjusted version also reduced the overhead of the instrumentation-centric tools. Finally, if the tools are used as advertised by the tool developers, all tools work with acceptable runtime overhead, regardless of the profiling technique used.