You could try clipping the polygons in code before you send them off to the renderer. Then you could avoid the stencil buffer entirely. To do this you could clip against all sides of the rectangle one at a time. I can't say for sure if this is faster than what you were doing before but here goes:
For this algorithm you need to represent a polygon as a list of points
How to clip against one side:
# careful of array out of bounds access
k = -1
# find first instance where a vertex is outside followed by one that is inside
for(i = 0; i < verts.length; ++i)
if(isOutside(verts[i]) && !isOutside(verts[i + 1]))
k = i
# if no such instance found
if(k == -1)
if(isOutside(verts[0])) # entire thing outside
return empty array
else # entire thing inside
return verts # return unchanged
# if instance was found
else
newVerts = empty array
# add first vertex in new shape
newVerts.add(intersect(verts[k], verts[k + 1]))
# walk the part of the polygon that is inside
while(!isOutside(verts[++k]))
newVerts.add(verts[k])
# add last vertex in new shape
newVerts.add(intersect(verts[k - 1], verts[k]))
return newVerts
Then you could do something like:
poly = clipLeftEdge(poly)
poly = clipRightEdge(poly)
poly = clipTopEdge(poly)
poly = clipBottomEdge(poly)
If the the original polygon was convex then the new one should be as well which means you could triangulate it or whatever you need before you draw.
A line which forms an edge of the clipping rectangle (if you extend it to be infinitely long) splits the entire space into 2 half spaces. You will need to provide a function to check whether a vertex is in the "inside" half space or the "outside" half space. Ex: left edge of rectangle is at x = -15. To check if the point is outside, do (point.x < -15).
You will also need to provide a function to intersect 2 vertices. You have one vertex on the outside and one on the inside and somewhere in between there is a line that you want to clip against. For example if the left edge of the rectangle is at x = -15:
factor = (-15 - p1.x)/(p2.x - p1.x)
newVertex.x = -15
newVertex.y = (p2.y - p1.y)*factor + p1.y
You will also need to linearly interpolate all other attributes you care about (like texture coordinates).
If you want to be really slick, you can write the clipping algorithm once and provide an appropriate isOutside and intersect function for each run.