SiSi (Say it, Sign it): signing avatars

The other recent event that impelled me to start my internal blog was last week’s Extreme Blue European Expo at Hursley. Extreme Blue is a student internship program that IBM runs every summer. It lasts 12 weeks. The projects are proposed by IBMers but are implemented by students. The Expos are held in different locations each year, I think, but this year the European one was held in Hursley, UK.

I’d heard a bit about the Hursley-based SiSi project from a friend who was mentoring the team, so I moseyed on down to Hursley House and spent a good hour-and-a-half visiting the Expo stands and hearing about those and other projects from around IBM sites in Europe.

I’ve been learning British Sign Language (BSL) for about a year and, having learnt just the basics about how to communicate in BSL (that is, it’s not just hand signs or fingerspelling but also facial expressions, lip shapes, and the spatial location of the signs that matter), I couldn’t imagine how an avatar could convincingly sign – especially not translated in real time from speech, which is what the SiSi project aimed to do.

The SiSi team’s demo blew me away. They use a third-party piece of software to convert speech input to text. The text is then sent to the client machine (I think) where an avatar signs the text in BSL or American Sign Language (ASL), depending on the language you selected. I can’t remember any more of the technical details than that but the demo text they tried was translated to BSL at a reasonable speed, I thought (probably as fast as a human interpreter). The demo was on a local system but the students reckoned it did okay over remote systems.

The project was done with the University of East Anglia and the RNID (Royal National Institute for Deaf people) who supplied the database of signs (which I guess is probably a database of video clips and associated labels). I’m not sure who marked up the signs in Sign Language Markup Language (SLML), a form of XML, but I expect that’s the most intensive part of it.

The great thing for IBM and the Extreme Blue scheme is that, like last year’s LAMA project the SiSi project has attracted loads of press coverage, here and around the world.

SiSi aside, there were loads of other cool projects including (you can probably spot a theme in my interests here!) the Accessibility in Virtual Worlds project. For a change, the virtual world concerned was not Second Life but, instead, Active Worlds. Active Worlds enabled the project team to devise a way to mark up objects around the world using XML so that blind people can walk through the virtual worlds using sonar. The user wears headphones (or has speakers set up) and the nearer something is, I think, the louder the sound (or something like that).

I came away from the Expo with a handful of really professional-looking Moo cards and leaflets from the stands I had time to visit. I think the most amazing thing that occurred to me about the Expo was the amount and quality of work that the students were able to produce in just 12 weeks.

Leave a Reply

Your email address will not be published. Required fields are marked *