I suppose you want an arc of C0 and C1 continuity between the line and an arc.
As illustrated above, you already have a vertex A which is the intersection of an edge and an arc of which the center positioned at O and radius equal to R. The question is thus pure mathematical: given A,O,R, edge direction BA, and a corner radius r, find C,B, and T.
For simplicity, here let's assume that the black dotted line has the form of $C_y=kC_x+b$, where k is the same as edge BA while b is easily derived by combining k and rounding radius r. Therefore, three constraint equations can be listed to solve $C_x$:
- The distance TC is r: $(T_x-C_x)^2+(T_y-C_y)^2=r^2$
- The big arc equation: $(T_x-O_x)^2+(T_y-O_y)^2=R^2$
- Point C is on OT: $\frac{T_y-C_y}{T_x-C_x}=\frac{T_y-O_y}{T_x-O_x}$
Solve for Cx and you'll obtain four solutions (I used Mathematica):
$C_x=\frac{-bk+O_x+kO_y\pm\sqrt{-(O_y-kO_x-b)^2+(1+k^2)(R-r)^2}}{1+k^2}$
$C_x=\frac{-bk+O_x+kO_y\pm\sqrt{-(O_y-kO_x-b)^2+(1+k^2)(R+r)^2}}{1+k^2}$
The second expression is out of the arc, so you should use the first solutions. Given the two Cx values, they can be used to derive Cy values to make a candidate list for point C. Pick the correct C using manual check.
Now you have the center C of the round corner arc as well as its radius r. It should be relatively easy to determine point B and point T, and draw an arc through the two points.
