LUDI
About
Contact
Search
Register
Log in
Back
The Effect of Embodiment in Sign Language Tutoring with Assistive Humanoid Robots
Summary
Play Experience by Assistive Technology for play
Devices, services & contexts for play with children with disabilities
Name of Play Experience by AT
The Effect of Embodiment in Sign Language Tutoring with Assistive Humanoid Robots
Type of project
Intervention
Finished research project
Ongoing research project
Summary
(~150 words: target group, aims, kind of activities, Play Experience by AT (devices, services and contexts) used, play experiences and results)
This study presents interactive games for sign language tutoring assisted by humanoid robots. The games are specially designed for children with communication impairments. In this study, different robot platforms such as a Nao H25 and a Robovie R3 humanoid robots are used to express a set of chosen signs in Turkish Sign Language (TSL) using hand and arm movements. Two games involving physically and virtually embodied robots are designed. In the game involving physically embodied robot, the robot is able to communicate with the participant by recognizing colored flashcards through a camera based system and generating a selected subset of signs including motivational facial gestures, in return. A mobile version of the game is also implemented to be used as part of children’s education and therapy for the purpose of teaching signs. The humanoid robot acts as a social peer and assistant in the games to motivate the child, teach a selected set of signs, evaluate the child’s effort, and give appropriate feedback to improve the learning and recognition rate of children. Current study presents results from the preliminary study with different test groups, where children played with the physical robot platform, R3, and a mobile game incorporating the videos of the robot performing the signs, thus the effect of assistive robot’s embodiment is analyzed within these games. The results indicate that the physical embodiment plays a significant role on improving the children’s performance, engagement and motivation.
Play Experience by Assistive technology
Low-tech, high-tech products, services and contexts for play
Description, intended use(s), intended target group(s), manufacturer/developer, reference
The humanoid robot used in the study is a specially modified version of the Robovie R3 robot. The standard R3 platform is 1.08 m tall and weighs 35 kg and has 17 DOF (2*arms*4, neck*3, 2*eyes*2, wheels*2) but the modified version of R3 robot has additional DOF in wrists, and fingers, 29 DOFs in total; and also a LED mouth to imitate and/or express some basic emotions (such as a happy, sad or neutral facial expressions). The robot has five-fingered hands with independent fingers and longer limbs providing the opportunity to implement signs more accurately. The robot has also a small platform on the chest with an ASUS RGB-D camera which is used for gesture recognition. This platform provides also some flexibility on the interaction level and with respect to game scenario e.g. the depth camera attached to it can be easily replaced by a touch pad tablet according to game requirements in future studies. The other humanoid robot is Nao H25 robot. It has a height of 0.57 m, a weight of 4.5 kg, and is a system with 25 DOF, two cameras, sonar sensors, and force sensitive resistors. Furthermore, Nao robot provides two loudspeakers and programmable LEDs around the eyes. In this study, eye LEDs are used for giving nonverbal feedback to children. The Nao H25 robots have hands with three dependent and movable fingers to implement most sign language words. They are suitable to be used in interaction games due to their expressive face, small size, compact shape and toy-like appearance. For virtual games, two different sign game applications have been developed for Android platform. First application consists of a beginner version of an interaction game. This application includes little number of signs and a training session with the aim of teaching signs from TSL. It also gathers the subjective evaluations of the user for different robot platforms used to generate the signs within the project. This game is designed to be played by the users with no prior sign language experience or just beginners level sign language knowledge, especially children of early age group. The second application was for the users with advanced sign language knowledge. This application was used to verify the recognition rate of the signs generated by different robot platforms. This game had no training session and consisted of more questions. The flashcards/visual icons used in the physical game were replaced with the multiple choices for the tablet/web based test, where the participant watched the video of the physical robot’s signing.
Picture
Download
Context
The context of use
(home, school, rehab center or other environments)
The presented platforms (robots and Android applications) can provide assistance to deaf or hard of hearing children with different levels of TSL knowledge in learning new signs or reinforcing their prior knowledge by playing different games. Therefore, the robots and the apps may be used in different environments such as homes or classrooms.
Type of Play
Type of play in this play system
LUDI Classification of types of play
Cognitive Dimension
Practice
Symbolic
Constructive
Rule play (including videogames)
Social Dimension
Solitary
Parallel
Associative
Cooperative
Objectives
Objectives related to play according to ICF-CY
International Classification of Functioning, Disability and Health - Children & Youth Version (ICF-CY)
Play for the sake of play
Major life areas - d880 engagement in play
d8800 solitary play
d8801 onlooker play
d8802 parallel play
d8803 shared cooperative play
d8808 engagement in play, other specified
d8809 engagement in play, unspecified
Community social and civic life - d920 recreation and leisure time
d9200 play
Play-like activities
Therapeutic and educational objectives
b1 Mental functions
b2 Sensory functions and pain
b3 Voice and speech functions
b4 Functions of cardiovascular, hematological, immunological and respiratory systems
b5 Functions of digestive, metabolic and endocrine systems
b7 Neuromusculoskeletal and movement related functions
d1 Learning and applying knowledge (learning through symbolic play, learning through pretend play)
d2 General tasks and demands
d3 Communication
d4 Mobility
d7 Interpersonal interactions and relationships
Participant
Participant(s)
Chronological Age
0-3 years
3-6 years
6-12 years
12-18 years
Development Age
0-3 years
3-6 years
6-12 years
12-18 years
LUDI Categories of disabilities
Mental/intellectual impairments:
mild
moderate
severe
profound
Hearing impairments
Partially hearing impaired
Deaf
Visual impairments
Partially sighted
Blind
Communication disorders (speech and language disorders)
Physical impairments
Mild
Moderate
Severe
Autism Spectrum Disorders
Multiple disabilities
Explanation
Explanation on the use of low-tech, high-tech devices, services or contexts
Explanation
No instruction, self-discovery of the participant/subject
Verbal instruction, language and communication fitting to chronological age
Verbal instruction, language and communication is adapted
Visual and/or verbal instruction with AAC (Aumentative and Alternative Communication)
Visual instruction with written language
Visual instruction with pictures or drawings
Modeling by therapist/researcher
Hand over hand: therapist/researcher leads the actions of the participant
Prompting: therapist/researcher touches the participant as a key for further actions
Guided discovery: therapist/researcher coaches the participant so s/he discovers how to use the assistive technology
Modeling by peer
Visual instruction by peer
Verbal instruction by peer
Involvement
Adult: therapist/educator/researcher
Parent or significant others
Peer with disabilities
Peer without disabilities
Role
Non-participatory observer
Participatory observer
Providing instruction
After the instruction, providing supervision during play
Evaluation
Information about availability of outcome measure: publisher, website, contact person
Participants’ teachers and TSL instructors were interviewed before and after the studies. The children participants have been handed questionnaires with pictures to evaluate the designed system. Their recognition abilities per sign, per robot platform, and subjective evaluations of the robot’s signing performances are gathered and analyzed.
Achievements
Summary of achieved effects
The virtual games were tested with 16 adults and 5 children having no sign information and no hearing problems, as well as two groups of 31 hearing-impaired children with different levels of sign knowledge (beginner and advanced). Although the children preferred to play with Nao robot more, their success rate using the R3 robot was higher. The results verified our hypotheses and research questions: (1) the age of the participants, who did not have sign language acquaintance, did not influence their preference about the human or robot signer; (2) physical differences of the robot platforms did not matter when their videos appear in virtual signing game; (3) physically embodied robots improve the recognition rate of the signs drastically compared to virtually embodied robots; (4) the level of sign language knowledge changes the fact that the recognition performance gets better with the physically embodied robots in comparison to the virtually embedded robots.
References
References to the intervention or research project
Contact Person
Hatice Kose, hatice.kose@itu.edu.tr
Website
http://humanoid.itu.edu.tr
Publication
Kose, Hatice; Uluer, Pinar; Akalin, Neziha; Yorganci, Rabia; Ozkul, Ahmet; Ince, Gokhan; The Effect of Embodiment in Sign Language Tutoring with Assistive Humanoid Robots, International Journal of Social Robotics, 7(4), p. 537-548, 2015, Springer Netherlands
Keywords
Keywords
Example: Robots, Virtual Games, Mobile Phone, Public Playground, Adapted Toys
Humanoid robots, Interaction games, Embodiment, Non-verbal communication, Sign language tutoring
Play Systems with similar keywords
Motion sensing devices, humanoid and non-humanoid robots for enhancing motor and social skills of children with developmental problems
Socially Interactive Robot Sign Language Tutors
Back
