I have figure out the math with use legacy checked. but I don't know what is the math ps use with use legacy unchecked.
1 Answer
If you are using OpenGL and if you are referring to Photoshop's layer blend functions. This may be a complete answer for you
https://mouaif.wordpress.com/2009/01/05/photoshop-math-with-glsl-shaders/
I found that link from this one
https://stackoverflow.com/questions/818230/photoshop-blending-mode-to-opengl-es-without-shaders?rq=3
The rest here is not EXACTLY an answer to your question, because I do not fully understand your question. Do please add more information about how you need to apply this math for more practical answers.
While I don't know what the exact equations are, the blend options in Photoshop have been available for a very long time. My guess is that each layer (legacy) blend mode represents a fairly simple mathematical equation.
Given what photoshop is and who adobe is and how advanced their functionality has always been I would not be at all surprised if they were writing custom GFX card code way back when. However with the shear amount of GFX card code that was written for very specific hardware for specific engines like DirectX and OpenGL historically(and currently), chances are that they leveraged capabilities that have been directly available in these engines for a long time.
My guess is that you can simulate all of the layer blend modes with something like OpenGL's "blendEquation" and "blendFunc", which amounts to modifying how the values between two layers are added together. https://learnopengl.com/Advanced-OpenGL/Blending#Blending