If I were to create a chainmail that has realistic reflections, where would I start learning about what mechanisms to use? Normal cloth seems too simple, and the reflections look bad (they have to be in the texture, or the cloth would have to be metalic). If I did it as a series of toroids, physics would have problems with colisions, or they would just overlap weirdly. Maybe there is a hybrid approach? Where can I read about this problem?
It seems that you're mixing physics/engine programming with graphics programming. I wouldn't say that dealing with collisions is related to rendering.
Also, I don't really understand your problem with toroids, but in case, I found one paper related to this term - "3D ChainMail: A Fast Algorithm for Deforming Volumetric Objects" by [Gibson1996]. I would say that if you're interested in creating clothing then you'll have to deal with deformations and animations and not only reflections. And, in case if you'd like to just work with reflections/rendering, then you don't need to deal with physics in any way. Static mesh would be enough, but I'm just speaking based on informations that you've provided until this point.
Normal cloth seems too simple, and the reflections look bad (they have to be in the texture, or the cloth would have to be metalic)
I wouldn't agree with that and I don't really understand what you've meant by that. In many cases, reflections are indeed in the texture. Or more specifically - cubemap, local environment map, irradiance volume and many other names, used for explaining GI voxel-based methods. https://www.kostasvardis.com/files/research/PhD_Thesis_Kostas_Vardis_book_online_lowres.pdf - [Vardis2016] PhD thesis is nice place to check for methods that you might be interested in. I wouldn't say that normal cloths are too simple. I was never really interested in that area but there are number of papers and overall, creating realistic clothing in not an easy task.
Normal cloth seems too simple, and the reflections look bad
I can only guess that by "bad", you've meant that there are little to none reflections. That's because most of clothing is rather rough and you've got little materials with big specular term in everyday life. That's why there is little reflection to it.
I don't know what is your current level of knowledge. Also, I don't know if you're going to creating real-time interactive application or if you're going to create offline renderer. In any case - if you have Turing graphics card or newer, I would suggest using Ray Tracing for most realisting reflections. You can start with cwyman tutorials - http://cwyman.org/code/dxrTutors/dxr_tutors.md.html also you should read microsoft's and nvidia's introduction to DXR. Also, here is a relatively easy path-tracer, created by Matt Petineo - https://github.com/TheRealMJP/DXRPathTracer
If you get through above tutorials, then you can start reading about denoising and state of the arts methods introduced in recent years - https://alain.xyz/blog/raytracing-denoising Raytracing is a subject, which exists since the begining of computer graphics, so there are a lot of papers related to that. But I think that MJP's example is good enough for start. Also, read related papers that he's stating in github and his code. They're mostly related to avoiding artifacts and fireflies.