I integrated RocSpeak into an iterative, online training module where users receive their subjective feedback from others within a community. All users receive automated feedback on their nonverbal behavior features. There are 5 prompts, users can only advance through each prompt after giving feedback to at least three other users. This creates a kind of marketplace for users to exchange feedback.
Here's an overview:
The system is evaluated and validated in a 10 day study. The study revealed a lot of interesting insights.
Ru Zhao, Vivian Li , Hugo Barbosa, Gourab Ghoshal, M. Ehsan Hoque, A Semi-Automated, Collaborative, & Online Training Module for Communication Skills, PACM on Interactive, Mobile, Wearable and Ubiquitous Technologies.
Exploratory Work:
Quote from participant: "it was really fun for me both recording and leaving comments to help others. Everyone seemed so nice, some of them I really looked forward to seeing posts from each day. In fact, someone posted on my final video they thought I was charming and I've been smiling all day over it. When you have crippling social anxiety, it just feels really nice to know you still can come off as friendly and not like you are about to shit yourself."
While I did not initially plan to analyze sub-community patterns, I saw a lot of feedback that shares a theme as the quote above from many participants. This got me really interested in how people exchange information, influence each other, and learn together. I conducted preliminary network analysis on this data that leads to ongoing work with Dr. Hugo Barbosa and Dr. Gourab Ghoshal. The following excerpt is not included in the submitted paper due to inconcrete evidence with such small community structures, but remains a very interesting work that I would love to explore further.
Figure Description: Visualization of feedback exchange between participants. Nodes represent participants, directed edges represent the feedback given by one participant to another. Red indicates top performers that received an extra reward after the study. The layout is produced with the Force Atlas for directed algorithm using the Gephi Toolkit. The Force Atlas Algorithm takes edge weights into account while computing repulsive forces and distances between nodes.
"The nodes in the resulting networks were clustered using the Louvain Method for community detection. We factored in edge weights to identify the most connected nodes and optimize modularity.
The clustering algorithm classified 3 major sub-communities for the baseline group and 6 sub-communities with 3 major sub-communities for the treatment group. The baseline group had 7 out of 10 of its top performers (indicated in red) fall into one of the major communities (B3), and the remaining 3 in another community (B2). One community in baseline (B1) with 7 users had no top performers. The treatment group top performers were spread out more evenly into its sub-communities, with 4 in the 2 major sub-communities (T1,T2) and 3 in another major sub-community (T3). We theorize that these clustering dynamics may have potentially played a role in the greater relative improvements observed in the treatment group. The top performers, who are relatively more cognizant of areas for improvement, are consequentially able to provide more beneficial feedback to others. This attribute of the top performers should be consistent in both groups. Despite the anonymity of feedback, the baseline group formed a more homogenous community where top performers decided to mainly give feedback to one another, spreading knowledge that is possibly already known between users. We observed a negative overall change in performance in the subgroup with the highest density of top performers (B3) and thus speculate that the sub-community as a whole has reached a ceiling effect where they can no longer improve by learning from other members of their sub-community. We also observe a similar effect in the T3 subgroup, where average change in performance is lower than other sub-communities in the treatment group. We observed that sub-communities with participants that varied in performance - B2, T1, T2 – demonstrated the greatest increase in performance over other sub-communities in their respective groups.
Our initial exploratory observations introduce interesting questions for future iterations of our work. The potential of these social dynamics can possibly be harvested to create a truly synergistic system, which combines the human element and automated features to create an effective training module. Although community bonding appeared to be beneficial, the patterned formation of bonding can be detrimental to the improvement of users. For instance, some naturally formed sub-communities may lack guidance or exchange information that is already known by the majority. The ultimate goal of our tool is to improve public speaking, thus we seek an effective strategy for community bonding that will optimally benefit each user. With our next iteration, we can enforce the formation of communities after our first speech screening. This could be done by changing the video feed so that users who received high performance scores from the first video will see users with a previously lower score video at the top of the feed. While the patterns that we have observed on how communities are formed between the groups are very interesting, further analysis with larger communities are required to make conclusive results."