User-Centric AI-Based Multitrack Music Mixing Informed by Expert Practice

Music Mixing is a multi-faceted task that is technical, creative, and collaborative in nature. Once the audio tracks are recorded, these are handed over to a mixing engineer to create a cohesive mix that is well-balanced and representative of the artist's vision and identity. The task involves collaborative aspects like talking to the artist to understand their vision for the mix and exchanging demo mixes and reference songs, followed by engineering aspects where the engineer makes technical decisions to fix frequency masking, establish balance and panorama, and more. This is followed by more creative tasks that involve further transformations of audio tracks to get the desired sound. As a part of our PhD research, we have begun by understanding the intricacies of the process beyond the engineering aspect, so that we can build tools that are inspired and informed by the professional practice. To achieve this, we have so far conducted interviews and studies with professional and pro-am mixing engineers with significant experience and advanced skills in mixing to understand the workflow of mixing from the collaborative perspective. Our studies have shown that mixing engineers are more inclined towards technology that encourages assistance, collaboration, and easy integration into current workflows. We also found that mix engineers use reference songs and demo mixes as some of the directors to get a better understanding of the client's goals. Further, we are working on a deep learning-based model that uses the available expert knowledge about the engineering aspects of mixing informed by the collaborative standpoint to build a system that takes in stems and reference songs to predict parameters for the various audio effects in the mixing console for stems based on the mixing style of the reference song. This could help mix engineers create boilerplate mixes that they can further fine-tune to create the desired mix.