Sign Language Technologies

Overview

The Australian Deaf community is a vibrant minority-language community who use Auslan (Australian Sign Language). Students joining this project will work on an aspect of sign language technologies, with potential directions including:

• Working with Motion Capture recordings of native signers to control the movements of an avatar
• Working with a sign language encoding system, such as SiGML, to automate avatar movements for translations
• Exploring how we can use AI technologies, such as ML/Deep learning models to recognise, translate, or generate (elements of) signs
• Developing technologies to support sign language learning, for children and/or adults
• We are also open to proposals from students for

This topic is interdisciplinary, and attached to 2 ongoing projects (Breaking the Communication Barrier for the Australian Deaf Community, Google Award for Inclusion Research; and Auslan Assist, Sydney Trains)

Research engagement

Students will have the opportunity to engage with different aspects of a research project depending on which branch of the project they are most interested in. All students will have the opportunity to undertake development work. Some students will undertake literature review work. Some students may have the opportunity to undertake supervised field work.

Research activities

Option 1: Motion Capture avatar work

Use Motion Capture data from native signers to control Epic Metahuman avatars in Unreal Engine, creating sequences of signs that can be used in translating English messages to Auslan.

Possibly: attend Motion Capture sessions with native signers.

Write research report.

This work will be primarily with Maria Zelenskaya

Option 2: Sign encoding

Comparison of sign encoding systems and tools.

Selection or development of a sign encoding tool.

Connect sign encoding tool with our existing signing avatar.

Possibly: develop automatic translation system from English to encoded sign.

Write research report.

This work will be with both Jessica Korte and Maria Zelenskaya

Option 3: Auslan AI

Literature review of relevant ML/Deep Learning/other AI models/approaches.

Comparison and selection of sign language dataset.

Development/adaptation and training of models on high performance computing infrastructure.

Write research report.

This work will be primarily with Jessica Korte

Option 4: Auslan learning technologies

Literature review of educational approaches.

Comparison of existing sign language learning tools and technologies.

Selection of educational and/or technology approach.
Development of prototype

Possibly: evaluation with users or proxies.

Write research report

Option 5: Your idea

Probably: Literature review, “market” comparison, prototype development

Possibly: data collection with potential users or proxies.

Write research report

Outcomes

Option 1: Develop a library of signs/phrases for the avatar.

Option 2: Develop a system for sign encoding (and possibly translation).

Option 3: Develop an AI model for some element of sign recognition, translation or generation.

Option 4: Develop a prototype of a sign language learning technology.

Skills and experience

Option 1: Motion Capture avatar work
Required: Experience with 3D animation and Unreal Engine

Option 2: Sign language encoding
Required: Experience with programming AND/OR Unreal Engine

Option 3: Auslan AI
Strongly recommended: Experience developing ML systems

Option 4: Auslan learning technologies
Strongly recommended: Programming

All: Sign language knowledge desired, not required.

Start date

22 November, 2024

End date

14 February, 2025

Location

This research can be undertaken at Gardens Point or Kelvin Grove. Virtual attendance may be available by arrangement.

Additional information

Appropriate computing resources will be available (e.g. remote access to QUT HPC, or computer suitable for animation work).

An existing avatar and sign language resources will be available for students’ use.

Keywords

• Auslan
• sign language
• AI
• ML
• Deep learning
• avatar
• animation
• education
• language acquisition
• programming
• development
• language

Contact

Jessica Korte

Jessica.korte@qut.edu.au