Listening While Reading

How the ear, eye and mind collaborate to strengthen reading and comprehension.

If you've tried a language learning app recently, or listened to a podcast, you may notice that you have choices. In a reading task, you might see the option to hear the content as you read it. Podcasts offer AI-generated transcripts of conversations. News articles online have images, audio and text all embedded. What does this do to a language learner's mind? How does it impact the psychological processes of comprehension and learning? An interdisciplinary research program at the University of Maryland is exploring these questions.

This research is led by Jon Malone, director of the Maryland English Institute in the College of Education, Bronson Hui, assistant professor of second language acquisition in the School of Languages, Literatures, and Cultures, Tetiana Tytko, a doctoral candidate in second language acquisition in SLLC, and Nick Pandža, associate research scientist for intelligent human-machine systems at the Applied Research Laboratory for Intelligence and Security.

"A growing number of studies have begun examining the process of learning a new language alongside comprehension and learning outcomes and UMD has been at the forefront of this work," Malone said. “Previous research, both from our own collaborations and elsewhere has mostly looked at learning gains from reading while listening or watching videos but we now have the ability to track the reader’s attention through recording their eye movements. It allows for us to examine new insights into the reading process that previous multimodal studies couldn’t account for in their designs.” 

Initial research in these areas focused on learning and comprehension outcomes. An early UMD study led by Malone indicated that readers picked up more information about the form and meaning of new words during reading while listening, compared to reading alone and a subsequent study by Hui further illustrated that learners rely on memory skills more when balancing multiple types of information during reading. Two other recent papers by Hui—one on the role of reading speed and text complexity while listening and the other on the comprehension benefits of reading while listening—found that comprehension of text was improved by reading while listening, compared with listening alone, facilitating language fluency in reading without detriment to reading comprehension. 

Newly-published research by the team has taken the next step in examining real-time eye movements during reading while listening, compared with reading alone, finding that language learners look fewer times at unfamiliar words when they are simultaneously hearing the words read aloud, but longer each time, with stronger learning gains from reading while listening than reading only. These research projects have been funded through grants from the Language Learning journal, The International Research Foundation for English Language Education (TIRF) and Duolingo. 

"These findings are very exciting, not only in the outcome data but also the differences in the way learners read based on whether audio is available," stated Malone.

Reading while listening could help struggling readers improve their reading skills and help all readers increase their comprehension of the subject matter. 

“It is amazingly complex and relatively simple; anyone who reads and learns another language understands the challenges of learning and yet so many learners achieve strong reading skills. We hope to continue this work in helping to provide diagnostic data for struggling and developing readers as literacy unfolds across domains, languages and individual ability levels," shared Malone.  

Photo of eye tracking lab provided by Jon Malone.