Anyone interested in this application of machine learning might be interested to read How Emotions are Made by Dr. Lisa Feldman Barrett. She makes a compelling case that emotions cannot be reliably understood through facial expressions alone, and that context must be included to improve our own human accuracy at the task, let alone machine accuracy. While this article is about a textual dataset and so not an exact parallel, I think some of the same principles apply — namely that greater context is often needed to interpret an emotion from a message.