An algorithm to measure social traits like trustworthiness in video interactions

Unmet Need

As humans interact with each other, their emotions and behaviors become coordinated over time. This phenomenon is referred to as “social synchrony”. These types of socially synchronized interactions correlate with how much people like each other, trust each other, and cooperate. Synchronized behavior occurs in customer relations, negotiations, and doctor-patient interactions. A deficit of these social interactions can also be a sign of developmental disorders, such as autism spectrum disorder (ASD) or social anxiety disorder. With high diagnosis rates of social communication disorders and ASD, CDC estimates that 1 in 44 children were diagnosed with ASD in 2018, there is a substantial population suffering from mental health issues and poor quality of life. Despite a high diagnosis rate, there still aren’t tools complex or comprehensive enough to appropriately measure the role of social synchrony in social deficit disorders. Better tools could lead to a quicker diagnosis that ultimately means significant gains in cognition, language, and adaptive behavior. There is a need for a method that appropriately measures the coordination of multiple features and assess their role in human behavior and interconnection.

Technology

Duke inventors have developed a completely automated algorithm to analyze social synchrony between two interacting people. This is intended to be an automated video analysis software that uses machine learning to measure the synchronization of multiple facial features as two people communicate. It has potential applications in the clinic, helping diagnose and treat mental disorders, as well as helping train humans, or robots, in empathy and trustworthiness. Specifically, the algorithm can use conventional or customized software to extract facial action units from a prescribed feature set and weigh each unit to determine the most relevant types of synchronies. Facial action units can be trained on relevant data sets for traits such as likeability, negotiation, and leadership. The algorithm uses a dynamic time warping procedure that allows for time lags and dynamic relationships between signals, instead of using conventional synchrony metrics. This has been demonstrated by predicting, significantly better than chance, if two people will trust each other in a Trust Game, after first analyzing video of them interacting freely.

Other Applications

This technology could also:

Monitor therapy sessions to give therapists feedback on their facial responses to increase patient trust, leading to better mental health outcomes.

Monitor telehealth appointments to give clinicians feedback that optimizes their approach and attitude towards patients, dramatically improving health outcomes, treatment adherence, and pain levels.

Help train customer service representatives, or executives, to show empathy and likeability.

Create social robots with the ability to teach humans.

Create virtual reality and augmented reality characters capable of emotionally connecting with humans.

Advantages

• Completely automated
• Simultaneously analyzes multiple types of behavior synchronization
• Can be trained to evaluate combinations of personality traits
• Algorithm uses a novel measure of synchrony that predicts social behavior better than any previously proposed synchrony metric