Breaking barriers with a sign language AI project

At the end of February, Yulong Li flew to the Mohamed bin Zayed University of Artificial Intelligence, ?United Arab Emirates, for his PhD exchange programme. He leads a subproject under Pheno AI, the human phenotype project.

Pheno AI, following on from the Human Genome Project, aims to construct a high-precision dynamic map of health and disease by analysing global health data, driving innovation in pharmaceuticals, healthcare, and life sciences.

Li got his start in the AI-driven health care field when he was an undergraduate student at Xi¡¯an Jiaotong-Liverpool University¡¯s Entrepreneur College (Taicang), where he led a sign language interpretation project called Limitless Mind. A few months ago, Limitless Mind won the grand prize at the finals of China¡¯s Innovation Challenge on Artificial Intelligence Application Scene (CICAS).

The presentation about Limitless Mind at the finals of CICAS 2024

Limitless Mind translates written text to and from sign language, and aims to break down communication barriers between those who are deaf or hard of hearing and those who aren¡¯t.

Bring niche academic interests to society

When he was a Year One student, Li volunteered at the China Disabled Persons¡¯ Federation which gradually drew him to the hearing-impaired community.

¡°Many people ask why we don¡¯t just type to communicate with the hearing-impaired,¡± Li says. But after meeting some of the hearing-impaired individuals, he realised not everyone can read and write ¨C it requires both money and effort that not every family can afford.

Yulong Li in an interview

Believing in the significance of sign language and its two-way technological system, Li began his research into sign language generation and translation in the summer of 2023.

Bridge the gap between two communities: from text to virtual sign language

In the early stages, Li¡¯s team faced an open-ended challenge: how could AI enable seamless communication?

To bridge the gap between hearing-impaired individuals and those unfamiliar with sign language, the team used AI to convert text into animated sign language via virtual avatars.

The team started with frameworks like MediaPipe, to extract skeletal data from hands and body movements enabling AI models to interpret sign language and generate text.

As the algorithm improved, the team developed a large-scale dataset mapping Chinese to sign language, enhancing the accuracy.

The two-way sign language generation and translation framework

Learn as they research

Dr Jionglong Su from the School of AI and Advanced Computing was the supervisor for this project, and he offered guidance to Li¡¯s research direction without micromanaging.

Dr Su believes students should learn as they research. When students hit bottlenecks, he encouraged them to explore the issue independently.

Interesting discoveries and emerging challenges

¡°We struggled with issues like algorithm optimisation and data annotation during the project,¡± Li recalls.

For instance, the team found MediaPipe had a high error rate in real-world applications.

Like Chinese dialects, the sign language system is highly diverse. A single MediaPipe framework couldn¡¯t fully capture all sign language. To address this, Li expanded the 3D skeletal dataset to enable the model to recognise more sign language expressions.

An evaluation of sign language recognition

Now, Limitless Mind¡¯s dataset contains 12,000 professionally interpreted sign language samples.

¡°Enhancing recognition wasn¡¯t enough,¡± says Li. To improve both grammatical and semantic accuracy during translation, the team integrated a text correction network, which acts like a language inspector, to ensure fast, accurate output.

China¡¯s largest sign language translation dataset

The core innovation of Limitless Mind addresses a critical gap: while academic research often focuses on accuracy, it often overlooks real-world constraints like users¡¯ hardware limitations.

To bridge this, the team applied knowledge distillation to sign language research compressing complex datasets into smaller, more efficient versions.

This approach not only optimised data efficiency and reduced hardware reliance but also improved accuracy, positioning Limitless Mind as a technological leader in China.

The team now holds China¡¯s largest sign language translation dataset, the most comprehensive sign language video vocabulary and 3D skeletal point collection.

The generation of sign language videos

When asked about long-term goals, Li emphasises focusing on technological development that make research outcomes more accessible.

¡°If no one uses our developments, what¡¯s the point?¡± he says.

Hence, Limitless Mind was designed as a socially responsible, freely accessible academic initiative.

Profit from elsewhere

While Limitless Mind prioritises real-world deployment of sign language generation, its true commercial value lies in intelligent systems and medical AI models.

Take the team¡¯s latest large model, CauseMotion, as an example. It focuses on emotional causality in human-machine interaction, mapping how changes in tone, speed, and facial expression relate to user behaviour. The model has outperformed GPT-4o and GPT-o1 in this area and was designed to empower Chinese AI applications.

Lacking business experience, the engineering-led team joined Xi¡¯an Jiaotong-Liverpool University¡¯s Entrepreneur College (Taicang) X³ Co-Venture in October 2024.

Since then, X³ Co-Venture has guided them on market expansion, target audience analysis, and funding strategy.

Their industry mentor, Dr Mikhail Zenchenkov, further supports investor matchmaking.

Now, the team is venturing into brain-to-text translation. By decoding rich information from EEG signals, the team hopes to open up new interactive possibilities for the hearing-impaired community.

By Jiayan Ji

Translated by Xiangyin Han

Photos by Yulong Li, Zuofu Wang

Edited by Angelina Yang and Patricia Pieterse