Over the last month billions of people have been unwilling participants in the largest unintentional social experiment ever run – testing how video conferencing replaced face-to-face communication.
While we’ve discovered that in many cases it can, more importantly we’ve discovered that, regardless of bandwidth and video resolution, these apps are missing the cues humans use when they communicate. While we might be spending the same amount of time in meetings, we’re finding we’re less productive, social interactions are less satisfying and distance learning is less effective. And we’re frustrated that we don’t know why.
Here’s why video conferencing apps don’t capture the complexity of human interaction.
All of us sheltering at home have used video conferencing apps for virtual business meetings, virtual coffees with friends, family meetings, online classes, etc. And while the technology allows us to conduct business, see friends and transfer information one-on-one and one-to-many from our homes, there’s something missing. It’s just not the same as connecting live at the conference room table, the classroom or local coffee shop. And it seems more exhausting. Why?
What’s missing?
It turns out that today’s video conferencing technology doesn’t emulate how people interact with others in person. Every one of these video applications has ignored a half-century of research on how people communicate.
Meeting Location
In the physical world the space and context give you cues and reinforcement. Are you meeting on the 47th floor boardroom with a great view? Are you surrounded by other animated conversations in a coffee shop or sitting with other classmates in a lecture hall? With people working from home you can’t tell where the meeting is or how important the location or setting is. In a video conference all the contextual clues are homogenized. You look the same whether you are playing poker or making a sales call, in a suit or without pants. (And with video conferences people are seeing your private space. Now you need to check if there’s anything embarrassing lying around. Or your kids are screaming and interrupting meetings. It’s fatiguing trying to keep business and home life separate.)
In the real world you just don’t teleport into a meeting. Video conferencing misses the transitions as you enter a building, find the room and sit down. The same transitions are missing when you leave a video conference. There is no in and out. The conference is just over.
Physical Contact
Second, most business and social gatherings start with physical contact – a handshake or a hug. There’s something about that first physical interaction that communicates trust and connection through touch. In business meetings there’s also the formal ritual of exchanging business cards. Those all are preambles to establish a connection for the meeting which follows.
Meeting Space Context
In person we visually take in much more information than just looking at someone’s face. If we’re in a business meeting, we’ll scan the room, rapidly changing our gaze. We can see what’s on desks or hanging on the walls, what’s in bookshelves or in cubicles. If we’re in a conference or classroom, we’ll see who we’re sitting next to, notice what they’re wearing, carrying, reading, etc. We can see relationships between people and notice deference, hierarchy, side glances and other subtle cues. And we use all of this to build a context and make assumptions—often unconsciously —about personalities, positions, social status and hierarchy.
Looking in a Mirror While Having A Meeting
Before meeting in person, you may do a quick check of your appearance, but you definitely don’t hold up a mirror in the middle of a meeting constantly seeing how you look. Yet with the focus on us as much as on the attendees, most video apps seem designed to make us self conscious and distract from watching who’s speaking.
Non-Verbal Cues
Most importantly, researchers have known for at least fifty years that at least half of how we communicate is through non-verbal cues. In conversation we watch other’s hands, follow their gestures, focus on their facial expressions and their tone of voice. We make eye contact and notice whether they do. And we are constantly following their body language (posture, body orientation, how they stand or sit, etc.)
In a group meeting it’s not only following the cues of the speaker, but it’s often the side glances, eye rolls and shrugs between our peers and other participants that offer direction and nuance to the tenor of a meeting. On a computer screen, all that cross person interaction is lost.
The sum of these non verbal cues is the (again often unconscious) background of every conversation.
But video conferencing apps just offer a fixed gaze from one camera. Everyone is relegated to a one-dimensional square on the screen. It’s the equivalent of having your head in a vise, having been wheeled into a meeting wearing blinders while tied to a chair.
Are Olfactory Cues Another Missing Piece?
There’s one more set of communication cues we may be missing over video. Scientists have discovered that in animals, including mammals and primates, communication not only travels through words, gestures, body language and facial expressions but also through smells via the exchange of chemicals and hormones called pheromones. These are not odors that consciously register, but nevertheless are picked up by the olfactory bulb in our nose. Pheromones send signals to the brain about sexual status, danger and social organization. It’s hypothesized odors and pheromones control some of our social behaviors and regulate hormone levels. Could these olfactory cues be one additional piece of what we’re missing when we try to communicate over video? If so, emulating these clues digitally will be a real challenge.
Why Zoom and Video Teleconferencing is Exhausting
If you’ve spent any extended period using video for a social or business meeting during the pandemic, you’ve likely found it exhausting. Or if you’re using video for learning, you may realize it’s affecting your learning by reducing your ability to process and retain information.
We’re exhausted because of the extra cognitive processing (fancy word for having to consciously do extra thinking) to fill in the missing 50% of the conversation that we’d normally get from non-verbal and olfactory cues. It’s the accumulation of all these missing signals that’s causing mental fatigue.
Turning Winners Into Losers
And there’s one more thing that makes video apps taxing. While they save a lot of time for initial meetings and screening prospects, salespeople are discovering that closing complex deals via video is difficult. Even factoring out the economy, the reason is that in person, great salespeople know to “read” a meeting. For example, they can tell when someone who was nodding yes to deal actually meant “no way.” Or they can pick up the “tell me more signal” when someone leans forward. In Zoom all those cues are gone. As a result, deals that should be easy to close will take longer, and those that are hard won’t happen. You’re investing the same or more time getting the meetings, but frustrated that little or no forward progress occurs. It’s a productivity killer for sales.
In social situations a feel for body language may help us sense that a friend who’s smiling and saying everything is fine is actually have a hard time in their personal life. Without these physical cues—and the loss of physical contact—may lead to a greater distance between our family and friends. Video can bridge the distance but lacks the empathy a hug communicates.
An Opportunity for Innovators to Take Video Conferencing to the Next Level
This billion person science experiment replacing face-to face communication with digital has convinced me of a few things:
- The current generation of video conferencing applications ignore how humans communicate
- They don’t help us capture the non-verbal communication cues – touch, gestures, postures, glances, odors, etc.
- They haven’t done their homework in understanding how important each of these cues is and how they interact with each other. (What is the rank order of the importance of each cue?)
- Nor do they know which of these cues is important in different settings. For example, what are the right cues to signal empathy in social settings, sincerity, trustworthiness and rapport in business settings or attention and understanding in education?
- There’s a real opportunity for a next generation of video conference applications to fill these holes. These new products will begin to address issues such as: How do you shake hands? Exchange business cards? Pick up on the environment around the speaker? Notice the non-verbal cues?
- There are already startups offering emotion detection and analytics software that measure speech patterns and facial cues to infer feelings and attention levels. Currently none of these tools are integrated into broadly used video conferencing apps. And none of them are yet context sensitive to particular meeting types. Perhaps an augmented reality overlay with non verbal cues for business users might be a first step as powerful additions.
Lessons Learned
- Today’s video conferencing applications are a one-note technical solution to the complexity of human interaction
- Without the missing non-verbal cues, business is less productive, social interactions are less satisfying and distance learning is less effective
- There’s an opportunity for someone to build the next generation of video conference applications that can recognize key cues in the appropriate context
- This time with psychologists and cognitive researchers leading the team
Share this:
Filed under: Covid-19/Recovery, Family/Career/Culture |
Speaking of Zoom…
Sent from my iPad
>
Wow, yes, a huge opportunity to go from 2D video conferencing to many higher dimensions! I wonder if VR or AR will provide a lot of that, although we’ll need a disruption to headsets as well. Being stuck in a headset all day would be even more tiring, not to mention nauseating.
Excellent observations, analysis, and commentary. Thank you!
Tough to get the olfactory sense virtual. Have to add sensors to Zoom that trigger when we raise our hand!
Ignoring the known is what we do with Agile. Most users don’t know the relevant theories, so you can’t ask them. And, if they studied it, they have forgotten all the implicit things that they learned. They can’t tell you those things. So we iterate towards a solution that turns out to be ignorant and amatuer. It should come as no surprise that Zoom is an example of these problems. We were never good at eliciting requirements. It is the unsolved problem of software development. We don’t even try to solve the elicitation problem these days.
Steve your on it!
I pivoted a dissertation subject due to this market and some interesting key pointers here I’ll use for my research.
Thank you 🙏
Stay safe, Steven x
Excluding that which applies to David Mamet’s House of Games and The Spanish Prisoner, the biggest issue, for working via ZOOM, may simply be that people have greater confidence, that those with whom they are communicating, are people who actually exist, and are not like the “sponsored” Facebook advertisements that are total scams, by Contrepreneurs, running under a flag of legitimacy, like that provided by the FT and Bloomberg, etc. 🙂 On the plus side: For those who loaded up on ZOOM stock, in January, they have doubled their money. Also, the claims over sexual harassment, ought to be hugely less.
my olfactory interactions are often noisy and embarrassing. better on mute than in a conference room 🙂
interesting, thoughtful post. thanks. hope you and the ladies are all safe and quarantined. we seven dorfs(kids, grandkids and nanny) are entering quarantine week 7 in stamford.
saw the first book with a wiley logo on it. congratulations on no longer needing to listen to erika.
be well Bob Dorf startup guy bobdorf@gmail.com us cell (203)253-3656 http://www.bobdorf.nyc
>
I’ve been video conferencing since 2008, and it is not the same as in person meetings. It’s been known for a long time that communication is not only what is said, but is also through how it is said, body language, tone of voice, facial expression.
video conferencing can extend communication among people who have worked together before, but it’s pretty good for establishing a working relationship with a new person.
However, if it is all you have, you can make it work. I have led a 200 person distributed software development program for 10 different software applications for the past 10 years. We were developing physics-based software applications for the DoD. As a result of the GSA convention debacle in 2012, the DoD prohibited all travel for almost 2 years. Our program suffered, but it survived. It would have fallen apart if we hadn’t had a high quality video and audio vtc system. Most recently we have been using a CISCO Tandberg system which has high definition audio and video. It works pretty well but is expensive and requires a very high bandwidth and a dedicated computer system to handle the video processing. The VTC terminals were $20k each, and the servers were also expensive. You can supplement the information transfer with email, a central document repository such as confluence, a user forum, an issue tracker, etc.
One of the most important “missing elements” are the personal and importantly, private, side conversations that take place with trusted colleagues. The opportunity to share ideas, needs, concerns with several trusted or new colleagues in 1:1 discussions quickly and efficiently helps accelerate collaborations, deals, contracts and innovations. In this quarantined state 1:1 interactions are significantly more time consuming and effort intensive (phone calls, 1:1) web meetings vs personal interactions at the conference while walking thru the halls or meeting for meals or drinks. I’d love to hear ideas on how this job-to-be done could be accomplished.
Andy, I don’t think this would be hard at all. In the Zoom world, you can have breakout rooms, but the users must be assigned (at least as best I can figure out). However, if two people could go off into a private breakout room or even an empty breakout room without being assigned to it, it would allow these off-line communications.
Marc, thanks for the comments. The hosts of a meeting control the breakout rooms – so if you’re not a host you’re not easily going to be able to go off with someone for the communication I mentioned. (BTW, Zoom breakout rooms can be either assigned by the hosts or random). It would be great if one or more of the technology providers could develop functionality that would work for this challenge. I believe it’s one of the big benefits of attending conferences. For me at least, the 1:1 discussions have almost always been more beneficial than sitting in the large room hearing (often) generic presentations or panel discussions
The opportunity is to actually move away from an over-reliance on meetings as a format for sharing information/ideas/resolving issues. That is actually the “old tech” and there are many better ways that don’t require “video” to solve those problems. Meetings have been broken for a long time now maybe the disruption is to develop new ways of working more aligned to what we see in modern research on teams. i.e. building up team safety and agreeing on ways of working – the obvious but uncommon ways of managing teams. There is lots of research out there (google/Woolley/etc) – you can download a summary from here https://dydx.digital/remote_teams/
Beyond communication to collaboration using shared spaces.
Hello Steve, Thank you for taking the time to write this article. I really liked what you say until you come to the recommendations 2.and 3. (Under Video Conferening, Innovators, Next Level).
Your described opportunities for innovation devalue, in my view, the deep insight you exposed in your introductionary text. Disappointingly, you revert to just another technical fix.
There are many opportunities to “invent” without needing to include stuff like “augmented reality overlays”.
Why not innovate with boldness? For example by reinventing good old fashioned local meetings, small local schools, local producers, local suppliers, local services, enhancing these with a wise choice of technology. Of course, this means challenging the economic and social system at its root.
I really don’t think we need more speech patterns and facial cues to solve the human issues you describe.
Going back to shaking hands in one same place, physically, may just be the boldest innovation we could make at this stage.
And, yet, it is not even slightly exhausting to some of us. What is exhausting is a one hour commute each way and then having to deal with the constant walk-by interruptions to work. I’m more productive and in a much better mood now.
Stop trying to put everyone in the same box.
Some of the disadvantages of videoconferencing you noted are made even more problematic by poorly designed virtual meeting processes. In “Journey to Group Power” we teach a variety of virtual group discussion and decision-making techniques. We thought previously that we had to be in the same room to teach and learn how to use these techniques; we now know that participants can learn how to use them virtually. It opens up so many opportunities for effective virtual communication.
I love that you’re thinking about this stuff. We’ve been experimenting with these questions and exploring the future of facilitation in this crazy new world.
The questions I’ve been specifically exploring is: “How might we recreate the interaction of a whisper that is so common within in-person workshops?”
I also echo Nevos’ comments about the over-reliance on meetings and the need for more asynchronous collaboration.
Check out the roundup that we created with lots of virtual tools, Figma is especially great for asynchronous co-creation: https://voltagecontrol.com/blog/the-best-tools-practices-for-remote-teams/
Good morning Steve. This is spot on! Those of us who consult, sell, teach, and work with others know this very well. Thank you for your thought leadership in this area. I look forward to advances in technology that can address some of the issues, although I’m convinced it will NEVER address the importance and necessity of human interaction. Quite frankly it is the uniqueness of this gift in some people that creates the difference that makes a difference, and their unique value to their organizations, and society at large. Thanks, Steve!
I think that this entire season is going to transform business. On one hand it will increase our commitment to human interaction. On the other hand, I believe it is going to make businesses consider how to be vastly more flexible to accommodate the changes in our society and culture.
Thanks for this article, Steve. I have definitely experienced a lot of what you mention above during this quarantine. Specifically, I have struggled with non-verbal cues and body language. I have realized that I rely on body language quite a bit when in a meeting. It is really hard to tell whether or not someone is excited about a project or working with you over Zoom. I’ve been thinking a lot about the future of our work environments and how things will change as a result of this virus. While I think people will definitely see the convenience of working from home, I also think that people are beginning to realize the great importance of in-person interaction. Thanks, again, for a great read!
Hi Steve. Thanks so much for this article — it resonates a great deal with me as someone who has always preferred face-to-face interaction in real life for business meetings. I agree with many of your points – for one, that you have to keep your focus on one person’s camera in the Zoom call, thereby missing a sense of the other people in the meeting, not just the speaker. I also agree with your point about the setting of the meeting – Cafe, meeting room etc providing extra meaning and a separation from home life. I do think Zoom is still useful for remote working though I hope we’ll be able to get back more real life face-to-face meetings in the future when this pandemic is over.