Automatic choreography generation with convolutional encoder-decoder network
Juheon Lee, Seohyun Kim, and Kyogu Lee
Seoul National University Music & Audio Research Group
Abstract
Automatic choreography generation is a challenging task because it often requires an understanding of two abstract concepts - music and dance - which are realized in the two different modalities, namely audio and video, respectively. In this paper, we propose a music-driven choreography generation system using an auto-regressive encoder-decoder network. To this end, we first collected a set of multimedia clips that include both music and corresponding dance motion. We then extract the joint coordinates of the dancer from video and the mel-spectrogram of music from audio and train our network using music-choreography pairs as input. Finally, a novel dance motion is generated at the inference time when only music is given as an input. We performed a user study for a qualitative evaluation of the proposed method, and the results show that the proposed model is able to generate musically meaningful and natural dance movements given an unheard song. We also revealed through quantitative evaluation that the network has created a movement that correlates with the beat of music.
Generated Results
Generated choreography 1 ( with music input : save-me, BTS)
Generated choreography 2 ( with music input : playing with fire, Black pink)
Generated choreography 3 ( with music input : knock knock, Twice)
Music and Audio Research Group
443-270, 864-1 Iui-dong, Yeongtong-gu,
Suwon-si, Gyeonggi-do, Republic of Korea
TEL +82 (0)31 888 9150