If I want to use my own Ray Tracer in a program like 3DS Max to render geometry, how could I do this? Can someone explain to me what I need to do and where I can learn to use an SDK to make this happen? Is this worth it to render test scenes to compare and test my ray tracer?
You can do this by writing a renderer plugin for 3ds Max. This gives you the same kind of access as the built-in renderers, including a settings dialog, a preview window, and an interface where you can write the final frame of a full render (and 3ds Max will write it to the output file). The renderer plugin gets full access to the scene graph when the user hits "Render" (or when the preview window is in use). Any complex geometry (such as subdivision surfaces) will already have been converted to a triangle mesh for you.
PaulHK already linked the official Programmers' Guide online. You'll also find it in the help files that are installed with 3ds Max, along with sample source code for a trivial renderer plugin. Because your plugin needs to link against the 3ds Max SDK, you'll need to compile it with the same version of Microsoft Visual Studio that was used to compile the particular version of 3ds Max you're targeting. (This means that if you want to target multiple versions of 3ds Max, you might need multiple builds.) Again, how to do this is described in the Programmers' Guide.