Authors: Eric Novotny, Gary Bente
Categories: Original Paper, Motion capture, Nonverbal, Perception, Synchrony
Source: Journal of Nonverbal Behavior
Interpersonal synchrony serves as a subtle, yet powerful bonding mechanism in social interactions. Problematically, the term ‘synchrony’ has been used to label a variety of distinct aspects of interpersonal coordination, such as postural similarities or movement activity entrainment. Accordingly, different algorithms have been suggested to quantify interpersonal synchrony. Yet, it remains unknown whether the different measures of synchrony represent correlated features of the same perceivable core phenomenon. The current study addresses this by comparing the suitability of a set of algorithms with respect to their association with observers’ judgments of dyadic synchrony and leader-followership. One-hundred fifteen observers viewed computer animations of characters portraying the movements of real dyads who performed a repetitive motor task with instruction to move in unison. Animations were based on full-body motion capture data synchronously collected for both partners during the joint exercise. Results showed most synchrony measures significantly correlated with (a) perceived synchrony and (b) the perceived level of balance of leading/following by each dyad member. Phase synchrony and Pearson correlations were associated most strongly with the observer ratings. This might be typical for intentional, structured forms synchrony such as ritualized group activities. It remains open if these findings also apply to spontaneous forms of synchrony as, for instance, occurring in free-running conversations.
Keywords: Synchrony, Nonverbal, Perception, Motion capture
Behavioral coordination, and more specifically ‘interpersonal synchrony,’ is a common means of affiliation and bonding among humans (Hove & Risen, 2009; Launay et al., 2016). Interpersonal synchrony refers to the coordination of body movement rhythms between individuals in an interaction (Bente & Novotny, 2020; Bernieri, 1988). It can occur spontaneously in social interactions, promoting rapport (Bernieri, 1988), trust (Launay et al., 2013), and cooperation (Wiltermuth & Heath, 2009) between the interaction partners; it can alternatively be orchestrated when group members deliberately follow a common external rhythm, as is the case in many cultural rituals that aim to strengthen feelings of belongingness and group entitativity (McNeill, 1995; Wiltermuth & Heath, 2009).
Both spontaneous and deliberate synchrony are evidently experienced as rewarding by the individual, as they benefit groups in terms of collective motivation (Reddish et al., 2013) and effective collaboration (Miles et al., 2017). In this sense, synchrony has been conceptualized as an evolutionarily established principle that facilitates social information processing and joint action (Bente & Novotny, 2020). This evolutionary perspective implies that behavioral synchrony can be perceived, and that the attainment of synchrony is functional and putatively rewarding (Kokal et al., 2011; Oullier et al., 2008).
Though numerous algorithms have been suggested to quantify motor synchrony (Cheong, 2019; Coco & Dale, 2014; Fujiwara & Daibo, 2016; Schmidt et al., 2012), little is known about which of the quantitative measures best relates to human-made global perceptions of synchrony. Numerous measures have been used to assess synchrony of movement activity between interactants, such as Pearson product-moment correlations, rolling window time-lagged cross correlations (Boker et al., 2002), mutual information (Moddemeijer, 1989), dynamic time warping (Pouw & Dixon, 2020), phase synchrony (Schmidt et al., 2012), among others. Understanding which measures relate to perceived synchrony is critical, as for synchrony to induce social outcomes like rapport, it must first be perceived by those involved (Oullier, et al., 2008). In other words, the human experience of synchrony and its social benefits are only possible insofar as the involved parties can perceive it (though this perception need not be consciously evaluated; Richardson et al., 2005).
Observer judgments have shown reliability and validity with respect to the measurement of synchrony (Bernieri, 1988; Bernieri et al., 1994). However, as Cappella (1990) noted, synchrony ratings could possibly be confounded with observers’ impressions of the dyads’ relational quality, or rapport. Specifically, he argues that because observers believe high synchrony and high rapport are equivalent, they may attribute high synchrony to a dyad because they first perceived high rapport. This confound is possible when other factors that contribute to perceived rapport, like smiling and proximity, are visible (Bernieri et al., 1994). Methodologies that obscure physical attributes of the dyads while preserving the movements are thus critical to faithfully assessing synchrony. Motion capture and character animation, in combination, do just this (Bente, 2019; Bente & Novotny, 2020), and we leverage these technologies for the current research.
To address the question of which measures relate to external perceptions of synchrony, we present a study that compares descriptive measurements of full-body motion capture data with observer ratings of synchrony. In this study, we address three methodological concerns associated with past research in this domain (see Bente, 2019). First, appearance-based confounds have plagued interaction observation studies that use videotaped interactions, as the physical characteristics (e.g., age, sex, race, perceived rapport of interactants; Bente, 2019; Cappella, 1990) could sway observer ratings of synchrony. In response to this issue, we overlay motion capture data with neutral (in terms of age, sex, race, etc.) computer avatars, which obscures the identity of participants while realistically preserving the movements.
Second, social scientists have studied synchrony within spontaneous, conversational interactions of dyadic partners (e.g., Bernieri, 1988; Fujiwara & Daibo, 2016). However, stemming from fields such as dynamic systems and physics, synchrony is more rigidly defined as a rhythmic, repetitive phenomenon (Bente & Novotny, 2020). That is, what is called synchrony in daily conversations might refer more to a metaphoric ‘social harmony’ (e.g., feeling in sync) rather than a truly cyclical, oscillatory phenomenon, such as the spontaneous coordination of metronomes affixed to a common support (Bente & Novotny, 2020). As such, participants in the current study performed a repetitive martial arts routine, thus allowing for the appropriate use of available synchrony measures given this strict definition.
Third, previous research on synchrony has based analyses on data protocols that are not conducive to precise measurement of movements of specific body parts. For example, MEA (motion energy analysis; Ramseyer & Tschacher, 2011) is a commonly used technique that quantifies movement activity as changes in video pixels over time. To measure synchrony between dyads, MEA allows a comparison of two videos of interactants in terms of their movement activity across time. Though this procedure has been validated and is well-established, it fails to allow for measurement of more granular movements and specific postural information, thus offering less descriptive data in terms of richness (see Bente, 2019). Instead, we base movement analyses upon a full-body motion capture procedure, which we call the Standardized Animated Motion Capture Protocol (SAMCP). This protocol addresses the stated concerns with MEA by allowing export of rich positional and/or rotational data of individual body parts (e.g., head nods, hand waving) while also allowing aggregation of motor activity across multiple body parts (Bente, 2019).
In sum, the contributions of the current research are (a) to illuminate which measures of synchrony align with observer perceptions (arguably the most important facet of synchrony for inducing social outcomes) while (b) showcasing the advantages of motion capture and character animation methodologies via the SAMCP, which circumvents methodological issues related to physical confounds, within-dyad biases, and imprecise data protocols on which to base synchrony analyses. In the following sections, we discuss interpersonal synchrony broadly, including its associated definitions, functions, key measures, and outcomes. We then present a study that uses a novel character animation/motion capture methodology (SAMCP) to compare objective synchrony measures of full-body motion capture data with observer ratings of synchrony. These measures include Pearson correlations, mutual information, dynamic time warping, phase synchrony, rolling window time-lagged cross correlations, and a new dynamic pose similarity, which compares moment-to-moment changes in the positions of specific body parts between two individuals.
Broadly, the concept of interpersonal or behavioral synchrony has been used to describe the mutual attunement of biological and behavioral rhythms between interactants (Bernieri et al., 1988; Burgoon et al., 2007). Evidence for synchrony is found in the alignment of the amplitude (strength) and frequency (rate) of bio/behavioral cycles such as heart rate (Mitkidis et al., 2015), breathing rate (Müller & Lindenberger, 2011), affect (Rafaeli et al., 2007), speech, and other expressive behaviors (Cappella, 1981), as well as body movements (Wiltermuth & Heath, 2009). Restricting the current research’s consideration of synchrony to the nonverbal domain, interpersonal synchrony is defined as the temporal coordination of motor behavior rhythms between interaction partners (Bente & Novotny, 2020; Bernieri et al., 1988; Delaherche et al., 2012). Beyond timing, the form of interactants’ movements may also be similar, though this is not a requirement for classification as synchrony. Interpersonal coordination types characterized by occasional matching of postures or movements are better subsumed by the term mimicry (Chartrand & Bargh, 1999). A combination of behavioral matching in rhythmic and form has been dubbed ‘perfect synchrony’ (e.g., perfect unison of a marching band), whereas general synchrony only requires a match in timing (e.g., an orchestra; Hale, 2017).
Multiple explanations exist regarding the ubiquity and utility of interpersonal synchrony in human behavior. One perspective holds that engagement in and/or observation of synchrony produces a perceptual phenomenon that enhances social bonding (Hove & Risen, 2009; Lakens et al., 2016). Here, synchronous movement functions to blur self-other perceptual boundaries in the minds of those involved. This means that when one actor witnesses another moving to the same rhythm, the neural representation of ‘self’ and ‘other’ becomes temporarily indistinguishable (Paladino et al., 2010). Moreover, as Aron et al. (1991) “…to the extent a partner is perceived as part of oneself, allocation of resources is communal (because benefiting other is benefiting self)” (p. 242). As such, the self-other merging created from synchrony fosters cooperation and social coordination (Galinsky et al., 2005), positive outcomes that could explain our propensity to synchronize.
A second explanation for synchrony is the brain optimization principle (Koban et al., 2019), which holds that performing or perceiving synchronous actions requires minimal neural energy compared to performing or perceiving asynchronous actions, thus creating an economically optimal neural state. Optimization of brain functionality is based on the free energy principle, which refers to the brain’s tendency to minimize coding costs when predicting and representing environmental stimuli (Friston, 2010). Neural networks have been compared to man-made electronic devices in that they are constructed to facilitate minimization of energy cost (Laughlin & Sejnowski, 2003). The optimization principle proposes that during an interaction where two people’s perceptual systems are linked (i.e., they can see or hear each other) synchronization is likely to develop because the brain requires less effort to represent the other’s behavior if it is similar to that of the self. As such, an implicit desire for less mental energy stimulates synchronized movements, and subsequently, through properties of dynamic systems (Schmidt et al., 1990), a stable state can emerge where interactants’ behaviors remain in synchrony. Moreover, Koban et al. posit that the reduced effort involved in synchrony is experienced as rewarding. The desirable emotional states deriving from synchrony become associated with the interaction partner, leading to positive bonding variables such as rapport and cooperation.
These principles explain spontaneous interpersonal synchrony; however, humans also intentionally leverage these principles in ritual behaviors that lead to synchrony and its associated benefits. From a cultural perspective, synchrony has been noted for its ability to enhance well-being and survival within the human social species (McNeill, 1995; Wiltermuth & Heath, 2009). Historically, many cultures have developed rituals that foster From tribal dances around a campfire, to religious practices involving concurrent bowing and kneeling, to vibrant hopping at modern rave festivals. Such activities are thought to increase cooperation and bonding among group members, as well as identify potential “free-riders,” or members of the group who do not pull their weight in terms of coordinating toward group goals (Wiltermuth & Heath, 2009). In the cultural perspective, movement synchrony is thus a way of enhancing group entitativity, or the degree to which a collection of entities is perceived as a unit (Lakens, 2010).
Throughout these explanations for synchrony winds a common the perception of synchrony, not just synchrony per se, is vital to the experience of synchrony as well as accompanying psychosocial outcomes (e.g., trust, rapport, cooperation, etc.; Bernieri et al., 1988; Tamborini et al., 2018; Wiltermuth & Heath, 2009). Increased attention to the behavior of other participants has been shown to moderate the impact of synchrony on cooperation, for instance (Reddish, 2012). Notably, perception of another person’s movements does not necessarily need to be done consciously for synchrony to occur; for example, visually coupled pairs have demonstrated the tendency to unknowingly synchronize cyclical movements (Richardson et al., 2005). Still, conscious or non-conscious, some level of perception of another’s movements is a requirement for spontaneous synchrony to occur (Oullier et al., 2008) and to produce the related outcomes.
Interpersonal synchrony has spawned many measurement techniques over the course of its study, ranging from the most basic (Gestalt ratings by human observers; Bernieri, 1988) to the most complex measures assessing the intricate dynamics of dyadic interactions. In the following sections, we focus on behavioral rating as a basic measure, followed by Pearson correlations, rolling window time-lagged cross correlations, (see Cheong, 2019), mutual information, dynamic time warping, phase synchrony, and dynamic pose similarity. This range of measures addresses different ways to consider synchrony, from an overall aggregation of similarity to fine pattern recognition.
A basic measure of interpersonal synchrony is conducted through human observation and identification (Bernieri, 1988). Pioneers of interpersonal postural congruence research created coding systems to identify specific body part locations in video film frames (Condon & Ogston, 1966; Scheflen, 1964). More recently, scholars have invented new, less laborious means of rating synchrony. For instance, Bernieri (1988) conceived of synchrony as a Gestalt-level behavior, identifiable not from specific movements per se but from the degree to which an interacting dyad generally shares tempos, meshes behaviors smoothly, performs movements simultaneously, and assumes similar postures. As Bernieri (1988; Bernieri et al., 1994) contends, synchrony can be accurately assessed from observations of dyadic video, thus not requiring rigorous movement coding or computational analyses. It remains to be seen, though, which aspects of synchronous movement drive these perceptions.
This observation technique typically involves observers watching videos of real interactants in conversation or some other dyadic activity. The videos are muted, and observers are instructed to judge synchrony and/or rapport, a dyadic-level construct characterized by a pair of partners really ‘clicking’ with one another during an interaction (Grahe & Bernieri, 1999). Rapport is related to synchrony, as it is partially embodied by the physical expression of motor coordination. In fact, rapport has been conceptualized as primarily a nonverbally derived phenomenon, and is thought to consist of mutual attentiveness, positivity, and coordination of behaviors in interaction (Tickle-Degnen & Rosenthal, 1990). Like synchrony, rapport is characterized as readily observable from nonverbal behavior (Grahe & Bernieri, 1999). Indeed, stable impressions of rapport can be formed from “thin slices” of human behavior (Ambady & Rosenthal, 1992), even within ten seconds (Bernieri et al., 1996).
A limitation of behavioral ratings of synchrony is the subjugation of measurement precision for more abstract assessment. Bernieri argues that synchrony can be observed from an abstract viewpoint, but this approach does not answer questions pertaining to specific movement patterns (in timing or form) that drive perceptions of synchrony or rapport. Thus, the explanatory power of this method regarding parameters that drive synchrony is relatively limited. Further, this method is confounded by appearance-based variables of the stimulus dyads (see Bente, 2019, p. 11). Bernieri et al. (1994) created a video mosaic method to account for a different appearance-based confound (smiling behaviors being linked to positivity, thus disrupting measures of synchrony per se), but it is evident from viewing these stimuli that gender and race are still interpretable (Bente, 2019). Other procedures, such as the SAMCP we introduce in this paper, avoid these confounds and enhance precision.
The Pearson product-moment correlation, or simply Pearson r, is a widely used and simplistic measure that assesses the strength of association between two continuous variables (Puth et al., 2014) without making causal predictions. Though Pearson correlation can also be used to compare the time series made by human observers’ coding of the two dyad members’ movements, in the current research, this correlation was calculated between the two time series measurements of dyad members’ movements as made by the automated procedure software. This measure is easy to interpret but is limited in (a) its susceptibility to outliers and (b) its assumption that data are stationary across a time series (Cheong, 2019). To account for these issues, extensions of the correlation, such as cross correlations and windowed cross-lagged correlations, have been developed (Boker et al., 2002; Coco & Dale, 2014). Still, the basic Pearson r is useful as a straightforward first look at the association between systems.
One prevalent time series analysis is the rolling window time-lagged cross correlation (RWTLCCs; Boker et al., 2002; Cheong, 2019), which provides correlations between two data streams across different time lags. Rather than only calculating the movement similarity between person A and person B at an inter-subject lag of 0 (‘on the spot’), the RWTLCC provides correlations for each of a range of lags specified by the researcher. RWTLCCs improve over correlations alone in that the latter assumes that time series data are stationary—that is, that the mean and variance of a parameter are relatively stable throughout an interaction (Hendry & Juselius, 2000, 2001; Jebb et al., 2015; Moulder et al., 2018). As many unstructured dyadic interactions are not stable in this regard, the RWTLCC’s advantage lies in its flexibility in analyzing non-stationary data. RWTLCC is used over short windows of time (whose measures can later be aggregated), thus allowing for granular inspection of fluctuations in measurement patterns, rather than producing correlations across a lengthy time series (which results in higher measurement reliability at the expense of measurement sensitivity; see Boker et al., 2002, p. 5). Lastly, the different lags can be compared in terms of highest inter-subject correlations, thus providing a measure of how tightly two people were aligned temporally. For instance, if the optimal movement activity similarity correlation occurred at a lag of 0 for two partners, those two participants were most often aligned in movements with no lag between them.
The rationale for using RWTLCCs over measures like standard correlations is that the former tolerates a wider range of possible contingencies between two data streams, whereas the latter is limited to only on-the-spot correlations. Thus, RWTLCC provides a more holistic representation of dynamic data patterns. RWTLCC output can be used to create heat maps, or figures that indicate correlation strength (over time and across different lags) via different colors. Inspecting the heat maps, one can identify at which time lags correlations are strongest throughout an interaction, thereby depicting fluctuations in patterns of leader and follower behavior. This is in contrast to global observer ratings, which might be useful for obtaining a general overview of a dyad’s coordination but less functional for identifying leader and follower fluctuations or onset/offset patterns of synchrony. A disadvantage of RWTLCCs is the difficulty or arbitrariness of selecting values for the required parameters, such as the size of the time window within which correlations are conducted. Researchers should look to prior literature in their respective fields (e.g., Schoenherr et al., 2019; Tschacher et al., 2018) and infer from theory which time lags and window sizes should be used, as well as the how much lag between systems could still be considered ‘synchrony.’ Another potential disadvantage is that the RWTLCC’s assumption of local stationarity (i.e., that the mean and variance are stable throughout a window) can be violated, driving down estimates of correlations.
Mutual information (MI, Moddemeijer, 1989; Shannon, 1948) is a measure of how much the behavior of one discrete or continuous variable can predict that of another (Ince et al., 2017). Derived from Shannon’s mathematical theory of communication, the central measure in MI is entropy (1948), which indicates the amount of information (in Shannons or bits, typically) provided by an event in relation to all other possible events. All else equal, more possibilities in terms of outcomes (i.e., more uncertainty) equals higher entropy. Entropy is, thus, the degree of uncertainty regarding an outcome of an event (Shannon, 1948), and MI is a measure comparing entropy between two variables.
Mutual information has been utilized as a measure of synchrony mainly within the psychophysiological literature as an indicator of the synchrony between humans’ auditory and visual systems (Hershey & Movellan, 2000; Prince et al., 2004). In the case of Hershey and Movellan (2000), MI was calculated for the synchrony between an audio signal and a spatially localized video signal. As Prince et al. (2004) note, “The HM [i.e., Hershey & Movellan] algorithm is relatively general, detecting temporal synchrony between two time-based input streams” (p. 89). Though little research has used mutual information to measure interpersonal motor synchrony, its generality in this respect gives it potential. In sum, MI is a previously established method with possible application to different synchrony scenarios. One could use this measure to provide an aggregate measure of total alignment in time of two motor systems, though it is not useful for uncovering specific dynamic patterns in the data (e.g., leader–follower relationships).
Dynamic time warping (DTW) is a technique that measures similarity between two time series while accounting for time shifts and speed differences (Sakoe & Chiba, 1978). DTW realigns two time series by plotting their data arrays against each other in a matrix and comparing each time series’ data points to those of the other (Mueen & Keogh, 2016; Pouw & Dixon, 2020). It involves calculation of a warp line, or a path through the matrix that connects all the lowest values (i.e., smallest distances between data points). This warp line can be compared to the ideal diagonal to indicate how closely the two time series are aligned and visualizes any temporal differences or time shifts between the two. Data from two people who were perfectly synced would generate a warp line that was very close to the ideal diagonal. A final distance value can be computed that sums all the minimum values, providing an aggregate representation of overall difference between the two time series. DTW can be useful for measuring synchrony between two data signatures that are different time lengths; however, it can be inaccurate and difficult to interpret compared to other measures (Silversides & Melkumyan, 2016).
Derived from dynamic systems research (Rosenblum et al., 2001; Schmidt & O’Brien, 1997; Schmidt et al., 2012), phase synchrony measures the relationship between two time series in terms of their phase. Along with period, frequency, and amplitude, phase is a feature of an oscillating system’s (a system whose parts show a periodic behavior) cycle that defines its dynamic behavior. Phase is the point in a cycle at which the oscillator is located at a given time. A pendulum’s cycle could be thought to start at 0º on the left endpoint, swing to 180 º on the right endpoint, and then restart the cycle at the left again. Phase synchrony, then, represents the relation between phase angles of two oscillating systems that are coupled. In the case of interpersonal synchrony, coupling refers to an interdependent relationship facilitated through a shared visual or auditory space (Oullier et al., 2008; Schmidt et al., 1998). It has been shown that once in action, coupled systems stabilize to either an in-phase (same phase angle) or anti-phase (opposite angles, e.g., 0° and 180°) angle, and remain in this state robustly (Schmidt et al., 1990).
Phase synchrony is a useful measure when researchers are interested in the alignment of rhythms between two systems. It is advantageous in that it can identify synchronous rhythms between even noisy and nonstationary systems (Rosenblum et al., 2001). For example, in a conversation in which movements are not repetitive or cyclical, phase synchrony can still identify interdependencies of phases. However, phase synchrony operates independently of the amplitude of the systems, thus not giving meaningful information about the magnitude of behaviors.
For the current study, we created a new measure known as dynamic pose similarity (DPS) that compares the positions of each of 15 joints over time. Whereas other measures here make use of overall movement activity (i.e., changes in position), this measure serves as a dynamic comparison of the specific locations of two actors’ body parts in a 3-D space. In this way, it can be thought of as a measure of the ‘perfect synchrony’ (rhythmic matching as well as form matching) discussed earlier in this manuscript (Hale, 2017). In addition, the output of this measure gives a lag offset measure similar to that of the RWTLCC.
DPS is useful for any researcher interested in both rhythm and form of synchronous dyads. However, for a researcher who is only interested in rhythm/timing of movements (such as the timing of overall movement activity shifts), this measure would offer superfluous positional information. Given its recent creation, it has not been applied in other synchrony research to date. In the current study, this measure provides the only instance of form-similarity. As such, if only this measure relates to perceived synchrony, these findings would suggest that similar movement form is indeed vital to people’s perceptions of synchrony. Table 1 summarizes the measures covered in this section.
The following study examines how objective measures of synchrony relate to its perception, which we have argued as essential to understanding the experience and effects of synchrony. As a source of full-body motion capture data, we refer to a previous unpublished experiment (Novotny, Tamborini, & Bente, 2019) that measured synchrony in dyads performing a martial arts routine and tested the impact of this synchrony impact on trust toward racial outgroup members. The resulting motion data allowed for (a) calculations of various objective synchrony measures and (b) the creation of stimulus videos depicting the movements via neutral computer characters (i.e., characters whose appearance lacked age, race, gender, or cultural cues, which can confound judgments; see Bente, 2019), via the Standardized Animated Motion Capture Protocol (SAMCP). A series of these stimulus clips was presented to a sample of participant observers, who judged both synchrony and the leader–follower relationship (the perception of which partner’s actions primarily preceded the other’s) of each dyad. The ratings generated from this study provided a comparison measure against which to judge the objective operationalizations (Bernieri et al., 1988; Cappella, 1981). If synchrony is a readily perceivable phenomenon at the Gestalt-level, and currently available measures capture synchrony validly, we should see a high correlation between subjective observer ratings and the various objective synchrony measures.
Attempting to find just this, researchers (Schoenherr et al., 2019) conducted a study to validate various time series analytic methods by comparing them to human coder ratings. Using a therapist-patient context, they found that only in an artificial condition (comparing person A’s movements with a time lagged version of his/her own movements) were time series methods reliably correlated with human ratings. Conversely, in more naturalistic conditions (where person A’s movements were compared with person B’s), the algorithms did not agree highly with raters in terms of identifying synchrony. As the authors “Our study revealed that a lot of algorithms with very high identification quality in the artificial configuration failed in the naturally embedded configuration. This could mean that the algorithms had another synchrony concept than the human raters in our study” (p. 17). This comparison between algorithms and coders will be retested in the current study, though with an improved means of measuring movements. Notably, Schoenherr et al. used motion energy analysis (MEA; Ramseyer & Tschacher, 2011) as the technique to extract time series measures.
Similarly, Fujiwara et al. (2021) recently compared automated and manual coding methods for specific nonverbal behaviors, including gesture, posture, and nodding behavior, also using MEA to quantify movement activity. They found moderate correlations between the two methods. These findings were encouraging in planning the current study. However, as mentioned, MEA evidently lacks precision with respect to analyzing specific body part locations throughout an interaction (Bente, 2019). As Fujiwara et al. (2021) note in their limitations, they failed to find a relationship between manually coded nodding similarity and automatically coded (through cross-wavelet analysis; Fujiwara & Daibo, 2016), stating that “This could be because the targeted region in MEA (i.e., the whole body) was too rough to capture the behaviors” (p. 15). In contrast, the use of full-body motion capture in the current study should further illuminate the relationship between objective synchrony measures of finer-grained behaviors and human observer ratings. Thus, we
Beyond capturing the degree of synchrony, we are also interested in the role of the leader–follower relationship (LFR) in a synchronous interaction. This is often the product of the entrainment of the relationship as outlined earlier. In a strict leader–follower type interaction, one person mimics the behavior of another with some delay, whereas in a reciprocally adaptive interaction, each person synchronizes through mutual prediction and reaction in real-time (Konvalinka et al., 2010). The nature of this relationship has been shown to impact the smoothness or performance of the involved partners (Noy et al., 2011). If LFR is a central defining factor of a synchronous interaction, and synchrony can ostensibly be perceived by observers, then objective measures that can accurately identify leader–follower patterns should align with observers’ ability to detect these same
In a previous unpublished study, Novotny et al. (2019) used Optitrack Motive-based motion capture data from 38 dyads performing a Tai-Chi routine (a martial art characterized by smooth, flowing movements of the whole body), which provided the foundation for stimulus generation in the current study. The details of this procedure are in Appendix A (see Figs. 1 and 2).
Fig. 1 Motion capture to character animation procedure
Fig. 2 Visualization of two participants’ movement data. Figures are constrained at the hips and standardized in size
Using the motion database, we created stimulus videos of dyadic partners performing the Tai-Chi routine side by side. This process involved rendering the motion capture data as standardized virtual characters and producing a video for embedding into the final survey.
We used Motionbuilder animation software (Autodesk Motionbuilder, 2018) to attach each participant’s motion capture data (exported from Motive) to standardized avatars via the SAMCP. These avatars (a) disguise the identities of participants in a controlled manner, (b) preserve the fidelity of the human movement (cf. Bente, 2019), and (c) prevent confounds typically associated with synchrony ratings, such as smiling and eye contact, which could affect observer ratings (Bernieri et al., 1994). Next, these animated avatars of partners were rendered into a single stimulus video for each dyad. Here we ensured that the two actors’ movements generally shifted in the same lateral direction as one another throughout their interaction.1
The resulting files averaged about 2 min and 27 s. Next, we segmented each AVI file into three parts that represented the first three cycles of the Tai-Chi routine (as not all dyads completed all five cycles). This segmentation was done to provide more stimuli for the survey, as well as to provide more appropriate time durations for observers. Segments were created by noting the time frame at which a participant’s Tai-Chi cycle restarted. Because one dyad had an erroneous third segment resulting from a capture error, the final stimulus pool featured 113 videos (38 dyads × 3 segments, minus 1 faulty segment), with an average segment length of 24 s. A screenshot of a stimulus video participants viewed is demonstrated in Fig. 3.
Fig. 3 Still image of a stimulus video from the observer survey
For data analysis, data were exported using a custom Python script (Leuschner, 2010) which allowed us to export movements as a batch file. Before exporting animation data in ASCII format, characters were scaled to a uniform size and the hips were snapped in the global origin and frozen so the positions of joints were relative to this root node (see AMAB procedure by Poppe et al., 2014).
Perceived synchrony was measured as a single item, which used a slider scale from 0 (no synchrony) to 100 (perfect synchrony) for each video. A single item was utilized to reduce participant fatigue (compared to requesting responses to an index after each video). Participants received the following
The perceptions of the leader–follower relationship of the dyad were judged for each video through the following multiple-choice “Was one person leading the interaction, or was it fairly balanced?” Possible responses to this question “Person A (on the left) led mostly,” “Person B (on the right) led mostly,” “It was fairly balanced,” and “Not sure.” The frequency of each response option (e.g., the number of times “Person A led mostly” was selected for a given stimulus) was divided by the total number of responses to that stimulus to provide (1) the proportion A or B led per stimulus and (2) the proportion of balanced ratings per stimulus.
Two variables were used to quantify the similarity of shifts in overall movement activity (used for Pearson correlations, MI, phase synchrony, and RWTLCC) and the similarity of position (used in DPS).
Overall movement activity. For most of the various synchrony measures outlined below, the variable of interest is the overall movement activity exhibited by a single dyad member with respect to their partner. Rather than focusing on the form of movements, for example in behavioral mimicry research (Lakin & Chartrand, 2003), this approach focuses on the timing of general movement activity, the variable more central in the concept of synchrony (Hove & Risen, 2009). First, using data from each dyadic partner in a given dyad, the x, y, and z translation (change in location from time n to time n + 1) of 14 primary body locations joints was targeted as suggested by Poppe et al. (2014). These Chest, left arm, left forearm, left hand, right arm, right forearm, right hand, head, right upper leg, right leg, right foot, left upper leg, left leg, and left foot. Next, an average was conducted across joints to give a single “movement activity” score for each time frame. This score served as the y-axis variable that fluctuates in the various time series measures used in this study (except for DPS).
Positional difference. Rather than implementing the changes in overall movement as a measure of synchrony, this variable represents a comparison of specific positions of two actors’ body parts in a shared global space. Specifically, it is the difference in Euclidean distance (x, y, and z translation) between all 14 joints of two actors. This was conducted by using the difference between a body part of partner A at time 1 and the body part of Partner B at time 1, with the aforementioned normalization in space providing a mutual root from which to measure coordinates. A lower difference of positions indicates higher positional similarity, which, when looked at over time and with different lags, gives us the DPS measure detailed below.
The following measures were collected using a combination of Python codes, based primarily on a synchrony measurement suite created by Cheong (2019). The final program is available upon request from the author.
**Pearson correlation. **To compute a Pearson correlation between two time series, the program calculates the average movement activity score across a given time series and correlates it with the average value of a second time series.
Mutual information. Mutual information (MI) was calculated by a Python program (https://stackoverflow.com/questions/20491028/optimal-way-to-compute-pairwise-mutual-information-using-numpy; McIntosh & Jadzinsky, 2017) that was appended to the original program by Cheong (2019). Using the formula mentioned MI(x,y) = H(x) + H(y)-H(x,y), where MI is the mutual information, and H(x) is the entropy of time series x, H(y) is the entropy of time series y, and H(x,y) is the joint entropy (shared by both systems).
**Dynamic time warping. **Dynamic time warping (DTW) is a measure of interdependency between time series irrespective of overall segment length. It is computed by minimizing the distance between two time series’ data points in a matrix and comparing the resulting diagonal line to an ideal diagonal. The package dtw (https://github.com/pierre-rouanet/dtw) was used to visualize the DTW matrix and provide overall distance scores for each dyad. These distance scores indicate the distance of the diagonal to the ideal line; a smaller distance indicates higher synchrony.
**Phase synchrony. **The phase angles of two time series can be compared for a measure of interpersonal synchrony. First, one must transform the movement data using a Hilbert transform, which separates a time series signal into its phase and power (Zayed, 1998). Then, the phase angles are plotted along a time series and inter-subject comparisons can be made. To obtain a score of phase synchrony, the program compares the phase angles by the
where PS is phase synchrony, a1 is the phase angle of time series A at a given point, and a2 is the phase angle of time series B at a given time point. Finally, this PS score is averaged over a whole time series to give a measure of overall phase synchrony, to be used for correlations with other variables.
Rolling windowed time-lagged cross correlation. The program executes Pearson correlations between two time series over given windows of time and smoothes out this process with a more continuously sliding window. In the current study, we used a window size of 75 frames (3 s) for correlations and a step size of 15 frames (0.6 s). In this way, the resulting time series graph gives a smoothly rolling output that is more visually interpretable.
Dynamic pose similarity. This measure aggregates the position (rather than overall movement activity) of one participant’s joints in x, y, and z directions of translation, and compares these values with those of their dyadic partner. The difference in position is then plotted over a time series with a range of time lags along the y-axis (as with RWTLCCs). The lag offset at which positional differences are the smallest (i.e., most similar) is plotted in a separate graph, and will be used as the aggregate measure of positional similarity for correlation with other variables.
Participants were 115 individuals (MAge = 23.3, SDAge = 10.92, 54% female, 78% White) recruited from two sources. The first set of participants (NGroup1 = 13, MAge = 46.0, SDAge = 17.62, 54% female, 85% White) consisted of acquaintances of the researcher, who were provided a survey link via email. These participants were blind to the research questions of the researcher and received only thanks for participation. The second set (NGroup2 = 102, MAge = 19.89, SDAge = 1.50, 50% female, 74% White, 14% Asian, 12% other races) consisted of undergraduates from a large public university in the midwestern United States. This group participated to fulfill optional research credits for a communication course of their choice. A Welch’s t-test, which tests for two-sample group differences when sample sizes are unequal, was performed to check for differences in mean perceived sync scores. There was no significant difference between groups, MGroup1 = 45.71, SDGroup1 = 28.94, MGroup2 = 45.23, SDGroup2 = 27.53, t(113) = -0.06, p = 0.96. As such, data from Group 1 and Group 2 were combined and will be referred to as a single sample for the remainder of this manuscript. All procedures were approved by the institutional review board at the university from which the second sample was drawn.
A survey was created in Qualtrics survey software. The survey presented to participants a random series of 30 videos to view and rate (with a cap of 30 views set). Resulting from this randomization, the total views of each video ranged between nine and 30. Notably, it was possible due to this randomization that observers could view multiple segments from a single dyad. We chose not to restrict the randomization in this way so as not to limit the number of available videos per observer. The survey asked participants to watch each video until it auto-advanced to the next page, which instructed them to rate both synchrony and LFR. Demographics, which appeared at the end of the survey, included age, race/ethnicity, and gender questions.
A link to an online consent form was distributed to friends and family via email, and to undergraduates through a participant pool management software. Upon consenting to participation, participants were rerouted to the observer survey. Due to coronavirus-related quarantine procedures, participants filled out the survey from a location of their choosing rather than a computer laboratory. The survey guided participants through viewing and rating of 30 stimulus clips.
A custom Python program was adapted from Cheong (2019). In the original code, this program calculates Pearson correlations, phase synchrony, dynamic time warping, and rolling window time-lagged cross correlations. A custom script that computes Z-transformations of selected joints, and which additionally calculates mutual information, was appended to this code. The script begins firstly by importing necessary packages and defining the variables to be measured. Second, one selects the variables for which they would like to view figures and descriptive statistics. Third, once the measures are selected, the program looks for a list of CSV files (described in the previous section) from which to derive data. In the current study, this list includes 38 files for participant As and 38 files for participant Bs. Fourth, the “movement activity” score for each time frame is conducted in the means described in the Overall movement activity section. Fifth, a filter of the user’s choosing is applied. We used a lowpass filter set to 0.5 Hz (1/50th of a second) to smooth the data. Finally, when the program runs, it outputs text files of descriptive data for selected measures (e.g., distance scores for DTW for each dyad) as well as a PDF of all figures.
An average synchrony score (perceived sync) between 0 and 100 was calculated for each of the 113 stimulus M
PerceivedSync = 44.47, SD
PerceivedSync = 16.18, Max
PerceivedSync = 73.84 (Dyad 11), Min
PerceivedSync = 21.20 (Dyad 30). The synchrony means were also broken down by averaging synchrony across the three segments for each MSegment1PerceivedSync = 44.49, SDSegment1PerceivedSync = 16.48; MSegment2PerceivedSync = 44.26, SDSegment2PerceivedSync = 18.23; MSegment3PerceivedSync = 43.64, SDSegment3PerceivedSync = 17.15. The three segments correlated highly in terms of perceived sync scores within dyads, segments 1 and r = 0.78; segments 1 and r = 0.78; segments 2 and r = 0.86, and showed high reliability, Cronbach’s α = 0.93. An ANOVA was conducted to test whether these three segments differed statistically from each other (in other words, to see if there was a change in synchrony ratings throughout the duration of a dyad’s routine). The difference between time segments was non-significant, F(2, 112) = 0.054, p = 0.947, partial eta^2^ = 0.001, suggesting no change in ratings over the course of the three segments of dyadic interaction. Thus, practice effects (i.e., improving in synchronizing over time) were not apparent from our data.
Two average proportions for each stimulus video (N = 113 videos) were (a) the percentage A or B was rated as leading and (b) the percentage of ratings indicating a balance of leadership/followership. Pearson correlations were conducted to check for covariation between perceived sync and the perceived proportion that A led (LFR_AB) or that there was an even LFR (LFR_E). Perceived sync was positively correlated with LFR_E, r = 0.576, p < 0.001, and had the inverse of that correlation with LFR_AB, r = -0.576, p < 0.001. This suggests that the perception of a balanced LFR is highly related to perceived synchrony.
For each of the objective synchrony measures, we provide a comparison of figures between Dyad 11 (the highest LFR_E, or most balanced dyad) and Dyad 38 (the lowest LFR_E, or least balanced dyad). A juxtaposition of these figures demonstrates how each measure showcases the range of synchrony from high to low (see Figs. 4, 5, 6, 7, 8, 9). The raw Motive data and video files are available on https://osf.io/49cwh/.
Fig. 4 a and b Pearson correlations over time for Dyad 11 (a) and 38 (b). Higher correlations indicate higher synchrony, which is indicated by a close matching of the gray (Participant A) and black (Participant B) lines. Dyad 11’s lines overlap temporally more often than Dyad 38
Fig. 5 a and b Rolling window time-lagged cross correlations (RWTLCCs) for Dyads 11 (a) and 38 (b). Darker colors indicate negative correlations, lighter colors indicate positive correlations. The midline for Dyad 11 (0 offset) shows that the highest positive correlation (i.e., synchrony) occurred almost on the spot. This was not the case for Dyad 38
Fig. 6 a and b. Optimal lag from rolling-window time-lagged cross correlations (RWTLCC) for Dyad 11 (a) and Dyad 38 (b). The optimal offset for Dyad 11 was 6 frames and for Dyad 38 was 110 frames (indicated by the dotted lines). Left of the solid black line would indicate that Subject A leads, and right of the solid black line would indicate that Subject B led. The smaller optimal offset for Dyad 11 compared to Dyad 38 suggests a more temporally aligned dyad with respect to overall movement activity. Also note the difference in maximum correlation values on the y-axes of the two graphs
Fig. 7 a and b Dynamic time warping distance matrix for Dyads 11 (a) and 38 (b). The x-axis is one participant’s timeline whereas the y-axis is their partner’s timeline. The white line traces the minimum distance between participants’ movement activity at each time point. A white line more approximating a perfect diagonal represents a smaller distance, or higher synchrony. Dyad 11’s distance is less than that of Dyad 38
Fig. 8 a and b Phase synchrony for Dyad 11 (a) and Dyad 38 (b). The top graph for each dyad shows the angle at each timepoint, with the light gray line representing one actor and black the other. The bottom graph for each dyad shows phase synchrony (from 0 to 1) continuously throughout the interaction. Dyad 11 more often shows phase synchrony scores approximating 1.0, whereas Dyad 38 shows this less often
Fig. 9 a and b. Dynamic pose similarity of Dyads 11 (a) and 38 (b). The x-axis represents the timeline of the interaction, and the y-axis represents the inter-subject lags. The darker coloring around the horizontal midline for Dyad 11 indicates a low difference in positions between the two actors when lag = 0. This is less evident for Dyad 38, whose coloring was less consistent in this regard. This suggests that the positions of Dyad 11’s actors’ body parts were more aligned when lag = 0 compared to those of Dyad 38
The average correlation of movement activity between participant A and B was calculated for each dyad with an inter-subject lag of 0 frames, MPearson = 0.28, SDPearson = 0.27. Though a range of lags could be used for Pearson r, these would fall under the category of a time-lagged cross-correlation, which is discussed in the RWTLCC measurement section. The correlation between participants in Dyad 11 was r = 0.69, p < 0.01. The correlation for Dyad 38 was r = 0.02, p = 0.13. The time series of general movement activity featuring these correlations can be seen in Fig. 4a, b.
This measure indicates the amount of lag between participants at which synchrony of movement activity was the highest. The average lag offset in frames, given by the RWTLCC, was MLag = 21.21, SDLag = 18.85. For Dyad 11, lag = 6 frames, and for Dyad 38, lag = 42 frames. The RWTLCC graphs can be found in Fig. 5a, b, and a comparison of these dyads’ lag offsets can be found in Fig. 6a, b.
MI indicates the information we can predict from one system based on observations of another system. The average MI for all dyads was MMI = 10.64, SDMI = 0.16. For Dyad 11, MI = 10.76, and for Dyad 38, MI = 10.63. This indicates that Dyad 11 had a slightly higher MI score than Dyad 38, aligning with higher synchrony exhibited by Dyad 11.
The average distance (DTW) for all dyads was MDTW = 6221.92, SDDTW = 1132.20. For Dyad 11, DTW = 5674.25, and for Dyad 38, DTW = 6356.17. The lower score for Dyad 11 indicates higher synchrony compared to Dyad 38. A comparison of these dyads’ DTW scores can be found in Fig. 7a, b.
This measure indicated the degree to which the phase angles of two participants’ overall movement activity were aligned. The average phase synchrony was MPhase Sync = 0.0.75, SDPhase Sync = 0.06. For Dyad 11, Phase Sync = 0.83, and for Dyad 38, Phase Sync = 0.65. A comparison of these dyads’ phase synchrony can be found in Fig. 8a, b. The higher Phase Sync score of Dyad 11 indicates that Dyad 11 was more temporally aligned in the cycles of their movements compared to Dyad 38.
Dynamic pose similarity represents the similarity of positions of two actors’ joints over time. Here we show calculations of DPS for Dyads 11 and 38 (see Fig. 9a, b). Further, the optimal offset of DPS (i.e., the time lag value at which positional differences were smallest) was calculated (see Fig. 10a, b). The average lag offset in frames, given by the DPS, was MLag = 36.79, SDLag = 41.60. For Dyad 11, lag = 7 frames; for Dyad 38, lag = 21 frames. The absolute values of these lags were used in correlations, as the sign should not impact the strength of association.
Fig. 10 a and b. Offset of dynamic pose similarity for Dyads 11 (a) and 38 (b). The solid black line is on-the-spot, and the dashed line is the lag at which peak similarity was exhibited for the dyad. Dyad 11 shows a lag closer to 0 compared to Dyad 38, indicating that for Dyad 11 movement similarity was highest when there was a smaller delay, compared to Dyad 38 (for which movement similarity was highest when there was a large lag)
To address the research questions, first, correlations were run among the various objective and subjective synchrony measures as well as LFR_E. The results of these correlations can be seen in Table 2. For ease of understanding, we inverted the scoring of the DTW, RWTLCC, and DPS variables so that higher scores indicate higher synchrony for all variables in the correlation matrix.
In response to RQ1, which asked which measures of synchrony relate to perceived synchrony, perceived sync correlated significantly with all measures, though most strongly with Pearson r, r = 0.85, p < 0.001, and phase synchrony, r = 0.77, p < 0.001.
Regarding RQ2, which asked which objective measures of synchrony relate to observer ratings of leader–follower relationships (LFRs), the proportion of equal (i.e., balanced) leader–follower relationship ratings was correlated significantly with phase synchrony (r = 0.56, p < 0.01), Pearson r (r = 0.46, p < 0.01), and RWTLCC offset (r = 0.39, p < 0.05).
The goal of this research was to illuminate which objective measures of interpersonal synchrony best relate with global perceptions of synchrony. Results indicated that numerous measures including phase synchrony, Pearson correlations, rolling window time-lagged cross correlations, dynamic time warping, mutual information, and dynamic pose similarity are all linked to global synchrony perceptions for this interaction type. Next, a balance in LFR was related to phase synchrony, Pearson r, and RWTLCC offset.
The first research question inquired which measures of synchrony would relate most highly to the subjective measure perceived sync. In order of correlation strength from strongest to weakest, Pearson r, phase synchrony, DPS, RWTLCC, and MI all were significantly related to perceived sync. Beginning with phase synchrony, the strength of this measure’s association with perceived sync may stem from the repetitive nature of this study’s interaction routine. The phase of an interaction is a feature of its periodic or cyclic nature; the more aligned two systems’ phases are, the more rhythmic they can be said to be. Despite the fact that the interaction type in the current study was not regular (in the sense that it did not feature a standard rate of movements), the repetitive and scripted nature of the Tai-Chi routine likely improved participants’ ability to achieve phase synchrony. In more spontaneous interactions, such as free-flowing conversations, it might be more difficult for the phase synchrony measure to identify rhythmic regularities like these. Accordingly, this measure is often used for scripted or regular interactions (Ouwehand & Peper, 2015; see for exceptions Fujiwara & Daibo, 2016; Schmidt et al., 2012).
Next, DPS was associated with perceived sync. This shows that the position of the limbs in space, not just the timing alone, could be related to perceptions that a dyad is in synchrony. The strength of this measure’s correlation with perceived sync shows that perhaps people look for ‘perfect synchrony’ (timing and form matching; Hale, 2017) when making judgments. In sum, in the type of synchronous interaction shown in this study, the form of the movements evidently played some role in shaping judgments.
Moving to Pearson r and MI, these aggregate measures were also associated with perceived sync. For this type of interaction, these measures serve as strong indicators of global synchrony, and are a good starting point for synchrony research involving relatively stationary data. The fact that they showed association with perceived synchrony in a highly complex dataset such as this one points to their robustness in identifying synchrony. However, for researchers interested in (a) non-stationary data types or (b) the dynamic patterns in a dataset, these measures simply will not suffice. As we saw from this study, the leader–follower relationship in a synchronous interaction ties in closely to perceptions of global synchrony, so researchers interested in the LFR would require more dynamic measures. Further, examination of figures produced by dynamic measures, such as the RWTLCC chart, can reveal patterns in the data that may be otherwise missed by aggregate measures. For instance, imagine a dyad that was—visually—highly coordinated in their movement dynamics, but that had one participant leading the other by five frames. If correlations were conducted only ‘on the spot’ (i.e., with no inter-subject lag), the result may indicate that there was an absence of synchrony. By looking at the patterns throughout the range of time lags, a strong association could be found beyond the on-the-spot portion of the interaction graph. Regardless of definition of synchrony as simultaneous or simply coordinated, many researchers would likely still be interested in the alignment of this dyad. As such, aggregate measures are advisable, but not sufficient in cases where dynamics are of interest.
The second research question asked which measures would correlate with a balance in leader–follower relationship, as measured by the item LFR_E. A balanced LFR correlated with several measures including phase synchrony, Pearson r, and lag offset (RWTLCC). Many synchrony ratings and measures thus seem to be inextricably related to a balance of leadership and followership in an interaction, even though there are types of synchrony in which leader and follower roles are not balanced (i.e., unilateral synchrony). When leader and follower roles are fixed, and there is an accompanying delay in the follower’s movements (i.e., mimicry), LFR is not balanced—though the movements themselves are still somehow coordinated in timing. Future studies should continue to investigate the role of balanced LFR in perceptions of synchrony—is it an essential component, or just something that enhances the synchronous experience?
The first major implication of this research is that it established and compared the validity of different measurement techniques for assessing interpersonal synchrony. Many synchrony measures, perhaps predictably, were related to global perceptions of synchrony. This study thus demonstrated the convergent validity between several of the available synchrony measures and observers’ subjective impressions of synchrony. These results suggest that in terms of global perceptions of synchrony, judgments made by human raters can be as accurate as state-of-the-art objective measures. However, if a researcher is interested in more specific movements (e.g., judgments of which body parts were most synchronized during an interaction), automated methods might provide more accuracy as well as reducing the time and effort required by human raters.
Moreover, our findings may be peculiar to the type of interaction used in the present research. We used a practiced, cyclical movement routine as a basis for synchronization among dyadic partners. Other researchers may be more interested in spontaneous synchrony that does not feature a cyclical aspect, but instead is linked by mutual adjustments in the timing of movements generally (e.g., Fujiwara & Daibo, 2016). Different interaction types could then lead to differences among the available synchrony measures. For instance, in a dyadic conversation where both participants are standing, dynamic pose similarity could be low, whereas phase synchrony could be high (if the participants’ exact body parts are not similarly postured, but their movement timing is aligned). We thus encourage other synchrony researchers to consider their studies’ interaction types, and to justify their use of measures over others accordingly.
Another implication is our methodological advancement in measuring synchrony. Several aspects of the SAMCP methodology render it an improvement over other extant methods. First, the use of character animation allows researchers to either alter or control the appearance of stimuli, while preserving the fidelity of the real human movements (Bente, 2019). This balance between control and realism is ideal. Second, the use of full-body motion capture is relatively rare in synchrony research. Many studies in this domain rely on motion energy analysis (Ramseyer & Tschacher, 2011), which leverages changes in video pixels as a measure of broad movement activity shifts. As noted earlier, this technique lacks the precision and granularity of the current method that locates the movements of specific joints on the human body, which can subsequently be aggregated. Thus, this research is a showcase of the power of combining character animation and motion capture in nonverbal communication research involving observations of movement parameters (see Bente, 2019).
Lastly, given the relative ubiquity of findings stating that synchrony improves social outcomes, it remains to be seen which types/qualities of synchrony drive these improvements. Do spontaneous interactions induce different outcomes than planned ones? Does the form of movements matter, or just the rhythm? These questions cannot be ignored by lumping all interaction types together and dubbing them ‘synchrony.’ The current research brought these issues to the forefront so they may be addressed going forward. Future research would ideally compare these aspects of synchrony in terms of their outcomes; for instance, one might expect perfect synchrony, compared to general synchrony, to produce stronger social effects, given that shared timing and form have been shown to contribute independently to social outcomes.
The first limitation of this study was that it did not compare multiple types of synchronous interactions. A direct comparison between reciprocal and unilateral interactions, or between regular versus irregular routines, for example, could be useful in further uncovering the utility of the various available measures. Still, this study was a first step toward establishing the need for further research on this topic. We urge future researchers to examine relationships between objective measures and global synchrony perceptions during different types of synchronous routines.
A second limitation was the exploratory nature of this research. Strong theoretical background warranting the use of certain measures over others is lacking in the communication science literature, as well as in other domains that study synchrony. As such, addressing research questions instead of hypotheses seemed more appropriate for the current study. As differences among measures and their relationships to qualities of synchrony continue to be discovered, the grounding for theoretical advancement will become more plausible.
A third limitation was that this study did not encapsulate all available measures of synchrony. Other methods have emerged, such as cross-recurrence quantification analysis (Coco & Dale, 2014; Shockley et al., 2002) and spectral approaches like the cross-wavelet analysis (Fujiwara & Daibo, 2016; Schmidt et al., 2014). Moreover, additional factors comprising perceived synchrony, compared to our single item assessing movement similarity, could reveal more nuanced findings. Facets of synchrony such as tempo similarity and simultaneous movement (see Bernieri & Rosenthal, 1991), if measured, might lead to distinct judgments. Future studies should incorporate these alternative measurements to observe how they align with the current findings.
Lastly, the scripted routine we used in the current study does not reflect the types of spontaneous interpersonal synchrony one might witness in the real world. Instead, the advantage of our routine was to facilitate a type of synchrony that observers could easily recognize from body movements alone. Constraining the type of routine across dyads also enables observers to compare the level of synchrony achieved by dyads without the influence of movement routine differences (e.g., one dyad is jumping up and down while another is having a laid-back conversation). As such, we chose the route of control over broad generalizability to all types of dyadic movement interaction.
Interpersonal synchrony can be found in different shapes and scopes throughout the natural world. Disentangling how a metronome differs from a human, how a religious ritual differs from a conversation, and how instruction differs from spontaneity are all key questions for synchrony researchers. For humans, at least, perception appears to be a common thread linking synchrony to outcomes. Our research used a state-of-the-art methodology to showcase how movement data can be assessed free from confounds while preserving precision, and compared these assessments to human perceptions. Several measures were able to detect synchrony differences that corresponded with variation in general perceptions. Future research may find that more unintentional and spontaneous interactions show different results with respect to synchrony measures. Indeed, we may wonder, does synchrony in a free-flowing conversation even exist in the same vein as two partners rocking back in forth in chairs stably? Answers to such questions must wait for the next wave of synchrony research.
We would like to thank Ralf Schmaelzle, Ron Tamborini, and Rene Weber for their invaluable input in refining this paper with their feedback.
The OptiTrack Motion Capture system was used to collect full-body motion data from 38 dyads. Motion capture took place in two divided square cells (15’ × 15’) in a laboratory. Twelve optical cameras were suspended from a truss system in each cell. These cameras detect motion through transmission of infrared light from reflective markers on the participants’ body suits. The suits are composed of tight-fitting black Nylon, and feature 37 passive Velcro markers placed throughout the participant’s body. Motive, the software that operates the OptiTrack system, enabled recording and storage of the motion tracked time series data.
The motion capture procedure was divided into four phases. In phase one, participant dyad members entered separate rooms in a laboratory and donned motion capture outfits before completing a pre-test outgroup trust measure. Next, (phase two), they separately learned and mimicked a Tai-Chi routine from a virtual avatar appearing as a gender- and race-neutral wooden mannequin (‘Woody'). This instructor, who appeared on a large wall projection, performed five repetitions of a 30-s routine, thus providing the training necessary for the next phase. In phase three, participants were instructed to perform the same routine they just learned, but now with a Black or White virtual avatar (the avatar’s ostensible race being the main manipulation; this race manipulation did not appear in the current study) appearing on the wall – who was in fact embodied by their dyadic partner. Specifically, the movements of the phase three avatars were generated in real-time by relaying the live movement data of one partner to an animation software (Autodesk Motionbuilder, 2018) that displayed a Black or White avatar onto the partner’s wall projected screen. Importantly, unlike phase two, where participants simply followed along to a pre-recorded routine, phase three did not have strict leader-follower roles; rather, the two partners’ task was to co-construct the routine using each other as references, allowing a mutually constructed synchrony to develop. Figure 1 demonstrates the routine in phase three. Notably, this was the stage in which the participants’ movement data (body part locations in 3-D space at each time frame) were collected via motion capture for the current study. Finally, in stage four participants completed a post-test outgroup trust measure to assess the effect of partner group and synchrony on this outcome.
A spatial normalization procedure of motion capture data was performed as recommended by Poppe et al. (2014). This is advised for comparing motion capture data between actors of different sizes and with different starting positions. To begin, we merged the motion capture files of two dyadic partners using Motionbuilder. We then applied a pre-rendered character to each actor’s motion capture data for visualization purposes. Once characterized, we scaled uniformly each character according to the average size of a male (1.75 m or 5′9″) and female (1.62 m or 5′4″) in the United States. After scaling, we translated each actor’s root node (the hip joint) to the origin of the the point where x, y, and z are all set to 0 m in Motionbuilder’s viewer window. Next, we ‘snapped’ the two actors’ hips to this origin; that is, throughout the scene, the translation of both actors’ hips were constrained to the origin point while the rest of their bodies moved freely as in real life. The last step was to set the starting orientation (at frame 0) of each actor to the front of the scene by rotating the Woody’s hip joint to 0º around the y-axis. The resulting scene shows two identically sized characters, both facing forward, and their hips fixed together. See Fig. 2 for a still image of this result.
The movement data were exported, with one data file per dyadic partner, via the tool Export Global Data (Leuschner, 2010) for Motionbuilder. This tool outputs the movement data as a spreadsheet in which the rows are time frames (at 25 Hz) and the columns are the movement translation in x, y, and z dimensions of 15 key body parts as advised by Poppe et al. (2014). Given a dyadic routine lasting 2.5 min, this would result in a rich dataset of 168,750 cells (3750 frames × 45 body part translation columns) per partner.