Malaga University students develop AI lip-reading tool to give voice to silent hospital patients

To overcome this communication barrier, students at Malaga University (UMA) have harnessed artificial intelligence to develop Conecta2 - a system that reads lip movements, interprets non-verbal speech, and translates it into synthetic voice output.

Behind this project are Yeray Peralta, Enrique Pérez, Sofía Casermeiro and Juan Diego Bermúdez, third-year students on the Cybersecurity and Artificial Intelligence degree programme.

The tool they have developed combines computer vision and neural networks to identify what a person is saying without making a sound. The result appears in text form on a screen and can also be heard through a device's speaker.

The idea came about a month and a half ago, when the four of them decided to take part in a hackathon: a computer competition in which they had to develop a technological solution with a social purpose. The challenge was to apply AI to help people with disabilities, improve their independence or alleviate some of the difficulties they face in their daily lives.

"The original idea came from Yeray," Juan Diego Bermúdez said. The rest of the team joined in developing a proposal which, from the outset, they considered "complex" but technically feasible. Their efforts paid off and they ended up winning the competition at the La Noria centre for social innovation, organised by the Malaga provincial authority.

The tool they have devised involves a camera that records mouth movements. The system focuses solely on the lips and analyses how they move while the user attempts to utter a sentence. "A neural network processes these images and suggests words that might correspond to the identified movements," Bermúdez said.

The problem is that many sounds involve similar lip movements. A single sequence of images can give rise to different interpretations and some combinations of words may not make sense.

To solve this, the project's developers incorporated another AI model tasked with examining the possible phrases and selecting the one that makes the most sense.

"There are many words, many vowels that sound the same," Bermúdez explained. That is why they needed a second system capable of distinguishing between the different options. "This has enabled us to improve accuracy because, without that filtering model in place, the output from the neural network is sometimes incorrect."

There is still room for improvement. As the student said, in one of the tests carried out, they managed to reproduce 22 out of 40 sentences correctly, word for word. The requirement of this test is that all the terms must match exactly those spoken. "However, an isolated mistake does not always prevent the message from being understood," Bermúdez stated.

One of the biggest obstacles they have encountered relates to language. In English, there are more advanced lip-reading models that have been trained using large amounts of data, while the available Spanish resources are more limited. Much of their work over the past few weeks has focused on improving the data used to train the system and increasing its accuracy.

To check that their tool works, the students presented it during the La Noria competition. Yeray Peralta had to state the exact time at that moment without making a sound and the programme had to read his lips. By using the actual time, including the minutes, they were able to demonstrate that the result had not been prepared in advance.

The test went well. The device recognised what he was saying and reproduced the message correctly.

At that point, the tool was already functional, although it still lacked an interface to make it easy for anyone to use. The team's priority had been to demonstrate that the technology could work, before spending time on its appearance or simplifying its use.

They highlighted the help of a group of older people from the Arrabal-AID association, who provided them with some ideas and helped to test the app.

The ultimate aim is for it to become an app that can be installed on a mobile phone. All you would need to do is hold the device in front of your face, open the camera and move your lips. The programme would interpret the message, display it in writing and play it back through the speaker.

The students have also taken the protection of personal data into account. Their training in cybersecurity has incited them to pay particular attention to the processing of images, especially as the tool is designed for situations involving people who are ill or particularly vulnerable.

The information sent to the cloud for analysis is limited to the text of potential phrases, with no need to transmit the images captured by the camera. "We must be careful in situations like this," Bermúdez said.

They are now looking to secure the support of a company or organisation that would enable them to develop a project that could eventually become a communication tool accessible from any mobile phone.

Read comprehensive regional education and school news