How to paint onto 3d models

Question

I have searched this topic for very long time and have found basically no information on it.

If we have a 3d model with a uv layout. What is the basic technique for 3d painting on said model with a brush? A brush stroke consists of the brush pixel data/image laid out in succession. So do we project each individual brush image onto the surface of the object? (Although in actuality to do this we would be projecting the object onto the image space instead). I have managed to project textures onto objects before. And projecting each individual brush image definitely seems feasible, but I'm not sure if it's practical from performance standpoint and whether there're better techniques.

Answer 1

I can't give you a "better" approach here, but I think that the approach to

project each individual brush image onto the surface of the object

is not really a performance issue if you implement that using the GPU. Especially in an editor environment, where not much is going on in comparison to a computer game. All you basically want to do is to render a single image per frame onto a limited set of textures. I can give you a draft of how I would approach the subject if I needed this feature. However, since I haven't implemented it myself yet, there might be some complications and problems I haven't thought of so let me know in the comments if something doesn't work as expected.

1. Find the affected surfaces

   • Use the corners of your brush texture and generate a reduced brush projection frustum
   • Check all objects against it and find the closest surfaces -> If you already use some kind of culling for your scene, reuse the already culled object list, since the brush can only "see" what the camera sees
   • Identify the subsurfaces/triangles affected by the brush

2. Gather all affected textures. You should know for each surface that is affected, which texture it possesses. Optimally, your whole object has just a single UV texture.

3. Create a shader program to render to your texture.

   • be careful, the vertex shader has to calculate the normalized UV coordinates and pass them down the pipeline as the vertex position, since they are the position in your texture. Check what additional information you need to pass down the pipeline.
   • In the fragment shader, overwrite each pixel that is under the brush. You might need to use blending here for non-rectangular brushes.
   • The brush position is a little bit tricky. You need it in the textures UV space in the fragment shader and I am not sure if you can calculate it in the vertex shader. So you might need to do that in the fragment shader or alternatively in a geometry shader. The problem I see is that the angle in which the brush projection hits the surface can't be calculated in the vertex shader and it is necessary to determine the covered area. The geometry shader has enough information to calculate the angle and the covered area. In the fragment shader, you can't calculate the angle either, but there you can simply check if and where it is located in the brush projection frustum.

4. Now simply render to your textures.

   • Bind your brush texture
   • For each affected texture
     • Bind the texture as render target
     • Render all affected objects and let the GPU sort out which pixels need to be overwritten.

5. (Optional) If you want to save your modifications, you need to read back the textures from the GPU and save it.

Since the main work is done by the GPU, it should be quite fast. However, I can see some problems with the described approach when the brush hits multiple objects/textures that are partially overlapping. Sorting out which pixel of the brush affects which texture/object can get messy. There might be some stencil/depth buffer tricks one can apply, but I think this is a topic for another question.

Answer 2

As long as no pixel on the texture is used twice, you can render the geometry and display the uv-coordinates (texture coordinates). Usually they are a combination of red and green (2d). When using a brush, you only need to read out the uv-coordinates you've hit and color the texture at position (uv) in the desired color.

Hint: paint onto a second texture, so you'll not destroy the original texture color.

Drawbacks: Sometimes the texture resolution of different triangles within a geometry vary. So at some faces it will look nice, and on others it will be pixelated.

As said before, if the mesh (geometry) uses the same part of a texture twice (two faces aiming at the same texture coordinates), and you draw on one of them, the other will be painted as well.

Answer 3

You could do it in different ways.

But the most logical way for me is that there are points sampled along the path, then splines/benzier curves are used to make it smooth (additonal points can be interpolated), thereafter it is projected and saved on the texture. So no "every" point has to be sampled.