I think that it is important to classify between different types of animations here. One rough classification could be
- Rigid body animations
- Vertex skinning animations
Rigid body animations
This refers to animations where the actual geometry data does not change. For example, you could have a geometric object, like a unit cube, and apply (usually affine) transformations to it: You can rotate, translate or scale this object. This is usually accomplished by some sort of scene graph: Each node of this scene graph can have information about the rotation, translation and scaling that should be applied to the objects. In general, this is something that can be expressed with a 4x4 transformation matrix.
The key point is that you don't have to store "animated geometry data". You only have to store the geometry data once, in its original state. In addition to that, you store the key frames of the transformation.
In your case, if you wanted to do a rigid body animation, you could store the geometry in an OBJ file, and additionally store the key frame information in an arbitrary form, for example
0.0 ( 0, 0, 0)
0.1 (10, 0, 0)
0.2 (10,10, 0)
0.3 ( 0,10, 0)
0.4 ( 0, 0, 0)
This could describe a movement of the object along a rectangular path, taking 0.4 seconds. Fortunately, you don't have to create your own file format for that. There already are solutions for storing this sort of animation data. For example, the COLLADA format already supports animations like this. A more compact representation of such an animation can be stored in a glTF asset. For example, it can represent animations like this one, from the glTF sample models repository:
Vertex skinning animations
This technique refers, roughly speaking, to cases where you have an "initial" geometry, and you can define "rules" of how this geometry should be deformed based on the pose of a skeleton. As the name suggests, this is mainly intended for character animations, where you define a skeleton consisting of the "bones" of the character, and the geometry defines the "skin" of the character. Vertex skinning is also supported by different file formats like COLLADA or glTF. The effect is shown in this image, again, from the glTF sample models:
As already mentioned by joojaa in his answer, implementing vertex skinning according to the specifications of these file formats may be a bit tricky. Additionally, a considerable amount of the skinning computations is usually done by the vertex shader. So for a ray-tracer, you would have to move these computations into your application code.
This refers to what you already alluded to in the question: There may be cases where the animation is not a simple transformation of rigid objects, and where you cannot define any "rules" of how the vertices should be transformed. For example, when the vertex positions are computed by a simulator, like a fluid simulator or a soft body simulator.
In doubt, it may simply be necessary to actually store the positions of all vertices, frame for frame. You could achieve this as a sequence of OBJ files. But note that OBJ files contain ASCII data, and thus, are usually large and the geometry data has to be parsed before it can be used. Storing the data in a binary form that can be used by the application directly may be beneficial here.
There are some possible approaches for reducing the amount of data that has to be stored. For example, one could extend the concept of "key frames" here. For example, one could store the initial configuration of al vertices, and then only store the movement of vertices that acutally have changed between two frames. But of course, if (nearly) all vertices changed, the savings may be diminishing.
I'm not aware of any universally applicable, "standardized" solutions for this. Maybe some of the solutions that joojaa referred to in his answer could be helpful here.