Music Emotion: Are We There Yet? A Brief Survey of Music Emotion Prediction Datasets, Models and Outstanding Challenges

0:00Music has this unparalleled ability to evoke and transmit emotion. It does. It cuts across all cultural and linguistic divides. And researchers have been captivated by this for decades. I mean, you can trace it back to foundational work in the early 20th century. People such as Meyer and Hefner who are trying to categorize music's emotional impact. Exactly. And our deep dive today is focused on the cutting edge of that very challenge.

0:27Music emotion recognition, or MER. Specifically using deep learning models. That's right. We're looking at a comprehensive survey of the field, the data sets, the evaluation protocols, and crucially, the obstacles that researchers keep running into. The central tension being that even with all this advanced technology, accurate predictions are still quite elusive. It comes down to the subjective nature of human emotion. And that's objectivity, you know, it translates directly into data limitations and problems with benchmarking.

0:57And yet the practical applications, if we can get this right, are just immense. We're talking about things such as personalized music recommendation in health care, perhaps to guide a listener's emotional state. Or training a generative AI to compose music with a specific controlled emotion. Which would be invaluable for, say, interactive media, or even just the composition process itself. This knowledge is essential for designing effective systems.

1:22And to construct any of these robust MER models, the quality of the emotion annotated data sets is paramount. Everything starts with the data. Everything. Everything. We can structure the comparison of these data sets across, well, five critical dimensions. Okay. The emotion representation model they use. The annotation method, whether it's induced versus perceived emotion. The data modalities. And, of course, the size and diversity of the data set.

1:47Let's start with representation, then. As I understand it, we're operating in two primary worlds. First, there are the categorical models. These use discrete labels. You mentioned Heppner's effective ring earlier with its eight categories. Dignified, sad, happy. That's a foundational one. A more modern music-specific framework would be something such as the Geneva Emotional Music Scales, GMS, which has 45 emotion tags grouped into nine categories.

2:14And the alternative to these discrete boxes. The alternative is the dimensional model. The most famous is Russell's Circumplex Model of Affect. Where emotions are mapped on a continuous space. Precisely. Mapped across two main axes. Valence, which is your positive to negative scale. And arousal, which measures the intensity or activation. Russell's original model had a third dimension, didn't it? Dominance. It did, yes.

2:37But that dimension is generally excluded from automated mirror studies. It just proves prohibitively difficult for human annotators to score consistently. So that distinction is vital, but so is how the values are collected. The annotation methods. Yes. First, you have static annotation. Meaning you assign one single emotion label to an entire song or a fragment of a song. Kind of a summary. A summary, yes.

3:00The MTG Jimendo data set is a good example of that. But then the opposite of that would be? Dynamic annotation. This is where the listener continuously indicates the emotion throughout the song. So you get these time-continuous ratings. Often sampled second by second. Yeah. It reflects how our emotional state evolves during the listening experience. The mood swings data set is a classic example of this approach.

3:21Capturing that flow is one thing, but we also have to distinguish between perceived and induced emotion. This seems crucial. It is. Perceived emotions are what the listener recognizes in the music. The composer's intent, you could say. Whereas induced emotions are what the listener actually feels. The personal response they experience. And the challenge there is that induced emotions are highly personalized. They're influenced by context.

3:44By memory. Which means you need spiralized data to capture them. You do. Betasets targeting induced emotion, such as DAP or HQU956, often have to rely on psychophysiological measurements. EEG, heart rate, skin conductance, that sort of thing. Because self-reporting for an immediate internal emotional response can be quite unreliable. Wait, hold on. If capturing induced emotion accurately requires biosensors, doesn't that immediately disqualify the vast majority of music data that's just audio?

4:15It does present a significant data barrier for that specific application, yes. A perhaps insurmountable one. It necessitates a fundamental shift in data collection. While audio is still the backbone, the field is moving toward a diverse range of multimodal sources. So beyond audio. Beyond audio. Yeah. The MIDI format is still important, for example, although only a few datasets focus on it, such as VG MIDI and Amopia.

4:39And those are typically for symbolic music, piano arrangements, and such. Correct. But other multimodal approaches are growing, incorporating lyrics. You see that in datasets such as DMVD and Moody lyrics and also video. There was that study using the movie dataset. A fascinating one. Raiders were presented with music alone, or a muted video, or the combined music video. And the finding was somewhat counterintuitive.

5:01It was. Models trained on the isolated modalities, so just the music or just the video, were sometimes more accurate than models built on the combined feed. What does that suggest? Well, it suggests that when the goal is to isolate the musical affect, the additional visual data might be introducing a type of noise that current models struggle with. And for induced emotion, you mentioned the integration of physiological data is advancing.

5:25Rapidly. Right. Data sets such as SayTunes merge biosensor readings with audio features, and they confirm that these signals contribute significantly to accurate valence classification. The multimodal approach, when it includes physiological data, generally outperforms single modality ones. Given all this variability, it makes sense that there's a push to refine the annotation methodology itself. There is. A specific shift toward relative methods. So moving away from absolute annotation, where you just assign a direct score.

5:54To relative annotation, where you ask annotators to compare pairs of samples. To judge one's emotional position relative to another. And the benefit there. Research shows that relative judgments, which are used in the Misi Save E dataset, can lower the cognitive burden on the annotator. Crucially, this produces more reliable ground truth data. It helps mitigate some of that subjectivity. But even with these refinements, you still have the problem of label distribution.

6:19Human tagging creates a massive data skew. It does. With user-assigned tags, you get high imbalance. In the MTG Gemendo dataset, for instance, common tags such as Happy and Film are among the most frequent. While more nuanced tags are barely represented. Which means the model needs specific techniques, like using focal loss to remain robust. So the data quality is clearly a bottleneck. It's limited, it's noisy, it's imbalanced.

6:45The models processing it have to be quite sophisticated. They do. Which brings us to model architectures. Since 2020, deep learning has cemented its dominance. We're seeing CNNs, recursive networks such as LSTMs, and now transformer architectures. And often a combination of these, the hybrid architectures seem to perform best. They do. A CNN LSTM model, for example, can combine the spatial feature extraction of a CNN with the temporal sequencing of an LSTM.

7:13And that approach achieved an accuracy of 68. And that approach achieved an accuracy of 68.1% for audio classification on a subset of the million-song dataset. A strong result. A strong result. But volume is still a core barrier. These emotion-annotated datasets are often small. Which forces researchers to use transfer learning. Precisely. They use pre-trained audio embeddings. models such as MERT, that's the music understanding model, which have been trained on huge, unlabeled music corpora.

7:40So MERT acts as a high-fidelity feature extractor. It does. It lets researchers overcome the limited size of their specific emotion-annotated datasets. Of course, the viability of any of these systems comes down to rigorous evaluation protocols. And the metrics depend entirely on the task. Is it classification for categorical models or regression for dimensional ones? For classification, you'd use accuracy or area under the RSC curve.

8:05Correct. For regression, though, especially with dynamic temporal predictions, the gold standard is the concordance correlation coefficient, or CCC. Why is the CCC so important for that task? Because it measures not just the average error, but how well the predicted temporal trace aligns with the ground truth trace over the whole song. It ensures the model is tracking the emotional shifts. But benchmarking across studies remains a systemic challenge.

8:31It does. Centralized initiatives, such as MEREX, have stopped their mirror-specific tasks. So direct comparison is very difficult. Different dataset constructions, different emotion representations, it complicates things. Which leads us to the persistent challenges that impede the field. And the first has to be the fundamental subjectivity of emotion itself. Without a doubt. Emotion ratings depend significantly on listener demographics. Things such as musical training, gender, age, familiarity with a song.

9:00And cultural background. And cultural background. The MRRP dataset actually tries to address this by including Rater profile information to enable personalized predictions. And when you delve into culture, the problem becomes pronounced. This is the cultural bias issue. Yes. There are studies showing that when Raters from different countries, say, the U.S. versus India, annotate the same piece of music, you find considerable differences in their annotations.

9:23So a truly global MR system needs a diverse, cross-cultural annotator pool. It's essential. And all this personal variability feeds directly into the problem of noisy labels. The ground truth data is imperfect. It is. Ambiguity in the emotion models, difficulty annotating certain dimensions, it all introduces noise. And this lowers inter-rater reliability, making it a challenge to train accurate models. What are researchers doing about that?

9:51They're exploring advanced techniques, things such as noisy graph cleaning methods designed to filter out the inconsistent outlier annotations caused by that high human subjectivity. It almost sounds as if we are asking machines to solve a psychological problem that humans can barely agree on. In a way, we are. So, given all of this, what are the key trajectories forward? Well, first, there is a pronounced need for standardized benchmarks.

10:16Centralized efforts, such as leaderboards or open data splits, are needed to promote transparent, comparable research. To get away from relying on siloed individual model papers. Exactly. Second, the MIDI format warrants more concentration. We mentioned there are only three established, emotion-annotated data sets for symbolic music. And that limitation restricts the development of generative music systems, where emotion could function as a direct control parameter.

10:41Finally, there's the demand for real-time systems. Yes. Most existing deep learning models require specialized hardware. They can take minutes to process a song. And for applications such as therapeutic emotion detection or mood guidance playlists, you need immediate recognition. You do. So, to summarize. The advancements in deep learning for Mare have been substantial, but the core conclusion is that the field is currently constrained by psychological variability and data scarcity, not by computation.

11:11So, the path forward requires larger, more diverse, culture-aware data sets. Coupled with standardized evaluation protocols and benchmarks. That's the key. So, here is a question we'll leave you with. If we successfully harness physiological sensing, using EEG or heart rate data, to objectively detect a patient's induced emotional state in real-time, does the advent of robust, multimodal M-year systems render traditional, subjective self-reporting in therapeutic settings obsolete?

11:37So, here is a question we'll leave you with.