How can I use 3d animation to create a character that uses sign language?

Question

I performed a quick search on programs that translate natural language to sign language (through a 3D animation character). Here are examples:

• IBM SISI
• ProDeaf (which is an android app)

My question is:

• how can I make my own character that uses sign language?
• Which platforms are preferred for 3D-modeling and animation(e.g. blender, maya, unity, ... etc.)?
• Is there specific algorithm(s) to make the character learn the sign language, or it is just a dictionary?

Answer 1

This is quite a broad topic, so I can only give a general overview. If you've never done any interactive animation before, this might be too big a challenge to start with, and maybe you should think of something smaller to do first.

You first need to model and rig your character, or try to find an existing model online that's rigged with enough control over the fingers. You then need to define a "dictionary" of animations or poses for each item of vocabulary in the language. To create the poses, it is probably easiest to use motion-capture technology to record a proficient sign-language speaker.

All of the above needs to be done in a 3D modelling and animation tool. Maya is very popular for character animation.

To get natural sign-language, you'll also need to write an algorithm to understand the way places in space are used as pronouns, and a fall-back that spells things out using letters when you don't have a vocab item for them. Based on next to no knowledge of sign language, I'd guess the algorithm will end up a little more complicated than a normal text-to-speech algorithm.

You've already mentioned Unity 3D in your question, and that might be a good choice for the real-time part of the system (assuming you want to render the character in real-time). If you use that, you can write the algorithm in C#. You would import the animation/pose dictionary into your system in Unity, and have your algorithm compute the transitions from one pose or animation to the next. In Unity itself, you can define the camera and lighting layout for your model and the backdrop to draw it in front of.