# Estimating the position of vertexes in a 3D model

Introduction to my project:

We are machine learning scientists and working on a biomedical system for tracking tongue for speech pathology using a technology called EMA (Electromagnetic articulography). In EMA a coil is attached to tongue surface using a glue and its 3D pos and orientation (roll and pitch) are read by EMA measurement system. So, you can track the coil and thus a point on the surface of tongue while speech is made. Also let's assume we have a low density 3D model of tongue for the patient is obtained by 3D MRI and we know exactly that which vertex in 3D model corresponds to the point that coil is attached.

It would be very valuable for our research if we can visualize the tongue for the patient. But the issue is that we just track 1 or 2 points on the tongue but our 3D model has more than 100 vertices which are not tracked. Now, we think it might be possible to estimate the position of those point given these two points.

Here are some of the facts that we think it should be possible:

1. Although tongue is not a solid object, it has a degree of rigidity thus the position of these 100 vertexes are correlated by physics For example how much they can be stretched or bent (Something like spring and mass model in physics).
2. by looking at a training set of 3D models for tongue during production of different sounds, we might say that the position of vertexes are not pure random and there is a distribution for them which might be estimated using the training dataset (sequence of MRI 3D models).
3. we might assume that in our presentation the tongue can have just a bent toward the palate to simplify the modeling problem. or a 2D profile of tongue (like the line at the center) is enough for modeling. or any simplification that we can make.

Now my question:

What do you know in computer graphics literature which resembles to our problem (any model, method, tool, software, library,....)? What is your suggestion for us to do this task in more efficient way?

• This is a difficult, under-determined problem. Perhaps look at minimizing the Willmore energy of a closed surface that approximates the tongue. The literature on this topic might give you ideas. Commented Sep 5, 2019 at 12:40
• 2 points are not enough to accurately represent a tongue model, even if you know that they are points on some manifold with rigidity constraint. The closest thing I can think of is Sumner's paper on deformation transfer. Commented Sep 5, 2019 at 15:00

If you have a good understanding of how the tongue deforms while speaking (presumably some part of it remains relatively fixed and other parts get a curvature), you might design a simple empirical model with a few degrees of freedom that maps any point in 3D space to the corresponding point after deformation.

For instance, you might map a line segment on a symmetry axis along the resting tongue to a parabola, so the horizontal plane by this segment to a parabolic cylindre, and all parallel planes to "parallel" parabolic cylinders. (This is just a simple example, you know better than me.)

I guess that you can focus on a median section, so that the problem is 2D and easier. From the 2D modelling, the shape of the transverse sections might be estimated from the position of the section along the tongue and the thickness after deformation.

Once you have this deformation model, you can use the tracked point(s) to determine the free parameters of the model (the number of degrees of freedom needs to match the number of independent measurements that you make).

Then when you have the parameters, you can compute the deformed positions of any vertex of the mesh.

If you can construct a mesh that is representative of the tongue, like a spring mass system, then you can attach 'hooks' at your two control points and get a close approximation of the deformation of the tongue.

I would be tempted to solve this problem using sets and subsets, and combine that with the symmetry that tongues tend to exhibit to get an approximation of the overall shape.

The first premise is: I can't imagine a tongue curled up on the left side and down on the right side, whatever the left side is doing the right side is almost always mirroring. So if we can figure out what one side of the tongue is doing then we can mirror that to the opposite side. This greatly reduces our search space for a solution.

The second premise is: Create a set of "major" positions the tongue can be in. Then inside each of the major positions create a subset of "minor" positions that are essentially refinements of the major position.

For a first pass at the solution I might just go for the major positions only to see if the overall approach was worth pursuing.

I would further limit the major sets to only track one half the tongue and assume that the opposite half is mirroring the behavior of the side that is being tracked. (for a first pass solution this seems reasonable)

The sensor data can then be used to filter the major sets into a limited group of sets of tongue shapes that are possible given the sensor position.

If we start with a known position of the tongue and we know the rate that the data is being collected then this further limits the positions the tongue can be in since it can't jump from one deformation to the next without going through other deformations first.

With one sensor I think the data would only be able to give major positions, but with two sensors it may be possible to get a fairly good approximation of the final shape of the tongue.

One way to increase the amount of data would be to have the person with the sensors repeat a sentence over and over, and then move the sensor to a new position and get them to do it again, and keep doing that until multiple positions have been tracked. Take the repetitions and average the sensor data for a single sensor tracking session, then combine multiple sessions of data to get an approximation of having many sensors on the tongue at once.

Once the tongue positions have been estimated for each sensor reading standard animation techniques could be used to show the tongues movement. With two sensors it might be possible to get a descent real time (or near real time) image of the tongue in action.