Music Control in an Interactive Conducting System Using Kinect

Music conducting is the art of directing musical ensembles with hand gestures to personalize and diversify a piece of music. Although the ability to successfully perform a musical piece demands intense training and coordination for the conductor and the orchestra, preparing a practice session is expensive and time-consuming. Hence, there is a genuine need for alternatives capable of providing adequate training for conductors at all skill levels. The potential use of virtual and augmented reality technology holds particular promise. The goal of this research is to examine the mechanics of music conducting and to develop a system capable of closely simulating the experience of conducting a piece of music. After extensive discussions with professional and nonprofessional conductors, in addition to wide-ranging research regarding music conducting materials, several key features of conducting were identified. A set of lightweight algorithms exploring these features was developed to enable tempo control, volume adjustment, and instrument emphasis, which are core components of conducting. Such a system would be a helpful training tool for students, an experiential tool allowing professional conductors and composers to shape music at a low cost, or an entertainment tool for nonprofessional music lovers. In this paper, we propose a real-time interactive conducting system using Microsoft Kinect. The proposed system overcomes the limitation of Kinect's design, which is generally designed for large body movements. In this system, delicate conducting signals can be correctly recognized without referencing any prior knowledge. Evaluation of the algorithms in real-world scenarios reveals promising results. The system was evaluated by conductors of all skill levels and provided a high level of accuracy and a low latency. Users of the final system expressed satisfaction with the virtual experience.