Getting to Know Neuromarketing: Facial Coding
Joy and surprise or fear and disgust: imagine knowing exactly how a consumer feels about an advertisement, website or mobile app moment to moment as they experience it. It's not science fiction, it's Facial Coding.
Facial Coding is a technique that makes it possible to detect and qualify emotions and their intensity based on the observation of the facial expressions of an individual.
How does it work?
Facial Expression Analysis, as the Facial Action Coding System, was originally developed by Carl-Herman Hjortsjö in the late 60’s and early 70’s and pioneered by Paul Ekman and Wallace Friesen in the 70’s, where facial expressions are analyzed, mainly through changes in specific categories of facial muscles, to determine an individual’s emotional response.
In this system, facial muscle contractions or relaxations are broken down into "action units" (AU) identified by a number. Each unit represents the activation of one or more facial muscles. For example, AU 0 represents a neutral face and AU 1 corresponds to the "raising of the inner part of the eyebrows". An expression can therefore correspond to several action units. The intensity of the expression is noted on a scale from A to E, E being the maximum intensity. Thus, for example, 1C shows a marked or pronounced rise in the inner part of the eyebrows, but not very intense.
The facial-feedback theory of emotions, is the backbone emotion theory of how facial expressions are connected to experiencing emotions. Two important proponents, Charles Darwin and William James both noted that physiological responses often had a direct impact on emotion elicitation, rather than simply being a consequence of the emotion. Supporters of this theory have suggested that emotions are directly tied to changes in facial muscles; for example, people who are forced to smile pleasantly during a social function are likely to have a better time at the event than they would if they had frowned or carried a more neutral facial expression. There has been a large body of research undertaken in Facial Expression Analysis over the last four decades, contributing towards diverse fields such as psychology, anthropology, business, marketing and crime.
How does this look?
Today, facial expressions are easily captured using a good-quality camera. A participant will be performing a set task on a website, for example, while a webcam records their facial expressions. The video will then be analysed frame by frame using sophisticated software, usually in conjunction with a recording of whatever the participant was experiencing on-screen.
It can then be determined what emotions the participant was feeling at any point during the task. For example, if your participant was frowning during key moments of a user journey while navigating your website, you might want to rethink those specific elements of your user experience as users may be having difficulty or find those aspects very confusing.
Some Pros and Cons
For facial coding to be accurate, a clear, front-on recording of the participant’s face is needed, with good and even lighting (a change in lighting can influence the readings drastically). This can make it tricky to implement in real world environments where the participant is moving freely, such is during a shopper journey assessment. Some participants also just don’t emote as much as others. They naturally have a dead pan facial expression that gives nothing away, much like a professional poker player concealing his winning hand, thereby not giving researchers any useful data. The software used at this point is also subject to error and may miss the context or more subtle emotional cues in the data it puts out. It is for these reasons (and more) that, of all the neuromarketing techniques available, facial coding has been shown to be the least effective in predicting future consumer behaviour.
However, there are a lot of positive aspects to it. For example, facial coding experiments are non-invasive and, as all that is needed is a good camera, it is possible to gather data from more than one participant at a time. Although an expert is ideal for Complex Facial Expression Analysis studies, less robust data collection can be performed by just about anyone with minimal training. As the data collection is also mostly automated, turnaround time is relatively quick while the cost is fairly low compared to other Neuromarketing methods. Facial expressions are also universal – a smile means the same thing in humans almost everywhere - so cultural differences need not be factored in. Overall, Facial Coding has modest reliability and is a robust measure of emotional expression; especially providing quality and type of emotion expressed.
A Fun Example
Disney recently used AI and facial tracking to gather data from audience sizes of 400 people. They tracked the audience’s facial expressions - and, therefore, their emotional responses – to nine of their movies. The study involved 150 showings and yielded a stunning 16 million facial landmarks from 3179 audience members overall.
With this information fed into an AI, Disney is now able to predict what an audience member’s reaction will be to key points throughout a movie based on their facial expressions in the first ten minutes. Moving forward, this will be a valuable tool for assessing the effectiveness of their new movies in hitting the right emotional beats. This improves their storytelling ability and ultimately the success of their feature films.
