Emotional states – Do not rely on facial expressions for how people are feeling | Science and technology
ARISTOTLE RECKONED the face was a window onto a person’s mind. Cicero agreed. Two millennia on, facial expressions are still commonly thought to be a universally valid way to gauge other people’s feelings, irrespective of age, sex and culture. A raised eyebrow suggests confusion. A smile denotes happiness. A frown indicates sadness.
Or do they? An analysis of hundreds of research papers that examined the relationship between facial expressions and underlying emotions has uncovered a surprising conclusion: there is no good scientific evidence to suggest that there are such things as recognisable facial expressions for basic emotions which are universal across cultures. Just because a person is not smiling, the researchers found, does not mean that person is unhappy.
As Lisa Feldman Barrett, one of the authors of the study, published in Psychological Science in the Public Interest, told the AAAS meeting in Seattle, “We surprised ourselves”. Dr Feldman Barrett is a psychologist at Northeastern University in Boston, Massachusetts, and along with her colleagues she found that, on average, adults in urban cultures scowled when they were angry 30% of the time. Which meant that some 70% of the time they did not scowl when angry. Instead, they did something else with their faces. People also scowled when they were not angry. “They scowl when they’re concentrating, they scowl when someone tells them a bad joke, they scowl when they have gas, they scowl for lots of reasons,” says Dr Feldman Barrett.
A scowl, the researchers concluded, is certainly one expression of anger. But it is not the only way people express that emotion. The ambiguous nature of facial expressions was not restricted to anger, but seemed valid for all six of the emotional categories that they examined: anger, disgust, fear, happiness, sadness and surprise.
All this raises questions about the efforts of information-technology companies to develop artificial-intelligence algorithms which can recognise facial expressions and work out a person’s underlying emotional state. Microsoft, for example, claims its “Emotion API” is able to detect what people are feeling by examining video footage of them. Another of the study’s authors, however, expressed scepticism. Aleix Martinez, a computer engineer at Ohio State University, said that companies attempting to extract emotions from images of faces have failed to understand the importance of context.
For a start, facial expression is but one of a number of non-verbal cues, such as body posture, that people use to communicate with each other. Machine recognition of emotion needs to take account of these as well. But context can reach further than that. Dr Martinez cited an experiment in which participants were shown a close-up picture of a man’s face, which was bright red with his mouth open in a scream.
Based on this alone, most participants said the man was extremely angry. Then the view zoomed out to show a football player with his arms outstretched, celebrating a goal. His angry-looking face was, in fact, a show of pure joy.
Given that people cannot guess each other’s emotional states most of the time, Dr Martinez sees no reason computers would be able to. “There are companies right now claiming to be able to do that and apply this to places I find really scary and dangerous, for example, in hiring people,” he says. “Some companies require you to submit a video CV, and then this is analysed by a machine-learning system. And depending on your facial expressions, they hire you or not, which I find really stunning and not only based on the wrong hypothesis, but a dangerous hypothesis.”■
This article appeared in the Science and technology section of the print edition under the headline “Face blind”