I have a set of non-convex polygons and I want to merge them. I am able to find the connection between two polygons. If there are more than two polygons, one connection can intersect another polygon within the set. I do not want to check for a possible intersection. So I have to find the nearest neighbor of a single polygon. Is there an existing algorithm? I can't find anything in my literature.
You're looking for a nearest neighbor search: https://en.wikipedia.org/wiki/Nearest_neighbor_search We've solved similar problems in 3D space using spatial subdivision (kd tree). Depending on polygon count and uniformity, a linear search or hash grid would be options too. A linear search, while O(N^2) can still be faster than setting up extra data structures for small a small number of polygons/vertices. A hash grid should be faster to update than a kd tree if your scene is dynamic, and I would expect it to be more efficient when the scene is uniform in vertex density.