How are textures projected onto 3d models in texture painting applications

In most modeling software you can texture paint onto 3d models using so called stencil textures. They basically project a texture from the viewport view onto the model as seen for instance in the image below from Blender.

What is the general technique used to project textures like this onto 3d models?

I don't know if there is a smarter/more efficient way to do it, but a possible approach is the following:

Step 1 - Projection volume

First, you calculate the "projection volume" where its near plane corners are identical to the texture corners on your screen. You can do this by transforming the screen coordinates of the textures' corners into the range [-1, 1]. Then you use those coordinates to create a quad in NDC by combining the transformed x-y coordinates with z=-1 (near plane) and z=1 (far plane).

Step 2 - Collision test

Now you need to have both, your model and your "projection volume", in the same space. You can either transform the "projection volume" into your model space or your model into NDC. I think the second choice makes the next step a little bit easier.

Now you have to find all triangles that are fully or partially inside the "projection volume". This is basically a collision detection check (triangle vs. quad or frustum) and I won't go into the details here. For all triangles that are fully inside the volume, you now have a connection between each vertex and its position inside the texture you want to project. Simply transform the vertex position into screen space and from there to the textures pixel coordinates. The second transformation depends on where you positioned your texture on the screen and how you have scaled it.

For all partially covered triangles, the basic procedure is the same, but you have to introduce extra "helper vertices" and to transform them too to separate the covered from the uncovered areas. Doesn't mean that you have to really split your triangles into multiple smaller ones, but at least you need the extra vertices as intermediate data for the final step.

Step 3 - Adjust model data

In the last step, you have to bake this information somehow into your model. This totally depends on your model's data structure and layout. You can for example just split partially covered triangles into new ones and then set the texCoord of every vertex that is inside the "projection volume" accordingly.

If you have some kind of model-specific UV-mapped textures, you would now use the information of the previous step to interpolate and copy the data of your projected texture to your model texture. For each vertex, you know where it is located inside the projected texture and inside its models UV texture. Now you basically cut out each triangle from your projected texture, and transform it into the corresponding triangle in your UV texture. Notice that this is not trivial since the triangle usually gets distorted and scaled. However, you can use the graphics pipeline for that by rendering directly to your UV-texture.

There are probably a lot of other data representations that might require a different approach for this final step. However, the key here is the second step which establishes the connection between your vertices and the texture. Once you have this, it shouldn't be too hard to come up with a method to add the projected texture to your model.

Further information

During the second step, you might also need to perform some kind of depth testing, so that only the closest area gets affected and not all sides that are inside the projection volume.

• Hey thanks for a very detailed answer! Regarding the collision detection, does it have to be performed on the CPU? Do we cut the partially covered triangles, create a temporary mesh including the new vertices and pass it into vertex buffer? Jul 17, 2020 at 14:01
• Also do you know if projective texture mapping is a viable technique? Jul 17, 2020 at 19:27
• It should also be possible to do the collision detection in a compute shader on the GPU, but it depends on the complexity of your model if it is worth it. Regarding the second part of your question, that depends on how you are going to store the results in your model. In the first case, I mentioned you need to update your whole mesh with the new vertices. If you use a UV texture for your model, you can create a temporary mesh to render the triangles to the UV texture. Jul 17, 2020 at 21:36
• "Also do you know if projective texture mapping is a viable technique?" --- Yes, why shouldn't it be? However, keep in mind that you should use the approach I described only if you want to include the projected texture directly into your model. If you just want to project a texture somewhere into your scene, there are easier, more effective ways to do that since you do not need to find a connection between model surface coordinates and texture coordinates. Just search for "Projective texture mapping" and you should find some material on how to implement it. It's similar to shadow mapping. Jul 17, 2020 at 21:50
• Oh okay, I need the projected texture to be directly painted onto the model and its associated texture. So I guess projective texture mapping wouldn't work for me. When it comes to drawing onto models with brushes, I imagine the same technique (the original one you proposed) could work. However not sure if it would be too slow with continuous brush strokes. Something like this imgur.com/a/McpJy7b Jul 17, 2020 at 22:33