Beyond hearing: sensing when conversation becomes difficult

An NSSN Grand Challenge Fund project has revealed how our eyes, head movements and even our hand gestures change when conversation becomes difficult, findings that could ultimately help hearing devices respond to what their users need.

About six years ago, hearing researcher Dr Kelly Miles had a stroke, just a week after having a baby.

The medical emergency left her unable to speak or read.

While her language recovered relatively quickly, she continued to struggle to follow conversations in noisy environments. Not because of peripheral hearing loss, but because of the way her brain was processing sound and communication after the stroke.

Dr Kelly Miles. Credit: Supplied

“I really started to understand the problem of people who have difficulties hearing,” says Dr Miles, an MQ-Sonova Research Fellow at Macquarie University Hearing.

“Before I had the stroke, I was really thinking about it from an academic perspective, but afterwards I really felt the impact of struggling to communicate with people.”

It also made her question some of the conventional advice given to people who find communication difficult, such as socialising one-on-one in quiet environments.

“I’m 40. I need to be out with my friends, not one friend at a time,” she says.

“Everywhere we go to socialise is noisy. Even if you go to a quiet place, the very fact that you’re there talking with your group of friends means you’re going to make noise.”

That experience gave added personal significance to the question at the heart of the Motion Optic and Sound sensors to Assess Interactive Communication (MOSAIC) project: can we detect when a person is struggling to communicate during a real conversation?

Looking beyond traditional hearing tests

Led by Macquarie University, MOSAIC was supported by the NSW Smart Sensing Network Grand Challenge Fund and included researchers from Western Sydney University, and industry partners Cochlear and Google Research Australia.

Dr Miles says most traditional hearing research has focused on listening.

“You listen to something and then you track performance,” she says. “But when we’re listening in the real world, we’re usually communicating. There are two or more people involved.”

The team created a multisensor laboratory capable of capturing participants’ movements, eye behaviour, pupil dilation and speech as they spoke face-to-face, while background noise was gradually increased to make communication more challenging.

The research was carried out in a multisensor laboratory at Macquarie University. Credit: Supplied

The result was a huge dataset, almost a terabyte of data, allowing researchers to explore how people adapt as communicating becomes harder.

The signals we give when conversation gets difficult

Pupils can dilate when people are under greater cognitive load, making pupil response a useful indicator of listening effort. The researchers also tracked where people looked during conversations.

Previous research suggested people look towards a speaker’s mouth in noisy conditions to gain visual cues from lip movements. MOSAIC found this was true for participants who spoke English as a second language.

But among first-language English speakers, researchers saw a different pattern: as conversation became harder, people increasingly looked towards their partner’s eyes.

Dr Miles said this could reflect the social side of communication.

“You’re giving this really strong social cue that you can hear them, that you’re engaged and what they’re saying is interesting,” she said.

The researchers also found that people changed the way they moved as conversations became harder.

“Just by looking at the magnitude and velocity of arm movement, we can tell if somebody’s listening in noise or in quiet,” Dr Miles said. “People gesture bigger and use their hands more in a noisy environment.”

Changes in head movement were also detected as background noise increased.

Research Professor Jeesun Kim from The MARCS Institute at Western Sydney University says her team brought expertise in the behavioural mechanisms underpinning face-to-face communication, particularly “how gaze, gestures and speech adapt when conversation becomes difficult and critically, and how those mechanisms change with age and hearing status.”

“In analysing the MOSAIC data, what struck us most was that these adaptations to noise are graded and multimodal,” Professor Kim says.

Professor Jeesun Kim. Credit: Supplied

The MARCS team found that people don't simply talk louder; they shift their posture, change where they look, and alter the complexity of their gestures.

“The promise is that if it can be detected that someone's gaze has shifted or their gestures have changed, a hearing device could respond to the moment communication is breaking down and, in this way, help people stay in the conversation rather than withdraw from it,” Professor Kim says.

Hearing what the listener wants to hear

These behavioural signals could eventually help solve one of the major challenges facing hearing technology: determining listening intent.

Dr Miles said modern hearing aids and cochlear implants can perform extremely well in favourable conditions, but noisy environments involving several people remain difficult.

In a noisy restaurant, for example, a hearing device can detect the surrounding sounds but determining which conversation its wearer wants to hear is much harder.

“You know that because you don’t look over at the other tables very often,” Dr Miles says. “Your eyes aren’t going that way, your head’s not turning that way. You’re very focused on what’s going on in a certain space.”

Combining signals such as gaze, head movement, gestures and other physiological measures could therefore help future hearing technologies recognise both who someone wants to listen to and when they are struggling.

That information could potentially be used to activate processing algorithms only when they are needed, improving communication while also conserving the limited battery power available in small wearable hearing devices.

Taking MOSAIC into the real world

NSSN Human Health Lead Catherine Oates Smith. Credit: Supplied

One of the most important outcomes of MOSAIC is that it provides a foundation for taking this research beyond controlled laboratory listening tests.

The project has produced a rich dataset and new methods for analysing communication behaviour, with several research papers published or progressing through peer review.

The work is continuing through the Australian Future Hearing Initiative, which is developing a platform combining wearable and mobile sensing with real-world assessments of communication difficulty.

NSSN Human Health Theme Lead Catherine Oates Smith says MOSAIC demonstrates how smart sensing can help address one of the most important challenges in hearing health.

“Being able to detect when someone is struggling to communicate, rather than simply measuring what they can hear, could transform the way hearing technologies support people in their everyday lives.

“This project is a great example of how collaboration between researchers and industry can translate smart sensing research into technologies that could help people stay connected to their families, friends and communities.”

Ultimately, researchers want to understand individual differences well enough that hearing technology can adapt to the person wearing it.

“The ultimate aim is to personalise a hearing aid, so it works for that person,” Dr Miles says.

The multisensor lab at Macquarie University was capable of capturing participants’ movements, eye behaviour, pupil dilation and speech as they spoke face-to-face. Credit: Supplied

The hearing researcher is also investigating communication difficulties following stroke and acquired brain injury, supported by a five-year fellowship examining the intersection between stroke and hearing.

MOSAIC, she says, marked an important shift in her career.

“This grant was the turning point for me to look at communication,” she says.

“It gave me the opportunity to see what happens when people communicate as opposed to just listen, which I know doesn’t sound novel, but it’s pretty novel.”

“Hearing is a gateway to community and social participation,” Dr Miles says.

Understanding the signals people give when they are struggling could ultimately help hearing technologies do more than improve sound. It could help people remain part of the conversation.