Raycasting wall slice height issue

I am having an issue with wall heights in my 2D raycasting playground I made for practice.

Problem

Test code in P5.js editor: https://editor.p5js.org/tomekp/sketches/6fKPIAHNx (WASD to move, Left/Right arrow to turn)
In the code above, I have an implementation of 2D raycasting, it is the best I have so far in that the walls are mostly the correct height depending on the player's position. The core of the logic is in sketch.js (you may need to open the sidebar menu if it doesn't open by default).
However, the problem is that when you are standing directly in front of a wall (or walls), just far back enough to see the top and bottom of the wall slices you are looking at, the top and bottom of the lines that are being drawn vary in size just enough to make the wall look sort of concave.
I've also attached an image at the bottom from the P5 link above to illustrate the issue further.

What I've tried

I've read the Lodev's tutorial (https://lodev.org/cgtutor/raycasting.html), and a number of other articles online that explain the maths behind raycasting. I learned that I should be using cos(rayAngle) * rayLength to get the perpendicular distance from wall to player. And after trying various code changes & studying up on my maths I still can't seem to grasp what's going wrong.

I think the issue is that despite the perpendicular distance being used there are still small variations in that distance causing the concave effect on the walls that you are directly perpendicular to, but not sure.

Any help/pointers would be appreciated, for context, I've been been mainly focusing on trigonometry when brushing up on my maths so I wonder if I am maybe just missing some required bit of maths to understand what I am doing wrong or if it's a case of misunderstanding how the computer renders rects/lines at specific canvas sizes. I think I basically don't know what I don't know to solve this problem.

Image of issue

You've made a classic mistake, not primarily in your raycasting, but in how you compute the rays. You wrote:

const rayAngleIncrement = FOV / width;
...
let angle = -FOV/2;
...
let rayDirection = createVector(
);
...
angle += rayAngleIncrement;


This math is consistent in itself, but it does not create a perspective projection — instead, it creates a cylindrical projection, as if the world is first projected onto a cylinder surrounding your eye, then the cylinder is cut and unrolled onto the flat screen. This process distorts straight lines into curves.

In a perspective projection, every part of the transformation is linear, and this preserves straight lines. Here is a standalone piece of JavaScript showing how to compute rays in a perspective projection:

const FOV = 60;
// Note this is tan() rather than sin(). We could use sin() here, but then
// we'd also need to change the length of ray_direction_y, which is more complex.
const rayXAtImageEdge = Math.tan(FOV / 2 * (Math.PI / 180));

const width = 100;
for (let pixelX = 0; pixelX < width; pixelX++) {
// This value ranges from -1 to +1 across the image, regardless of pixel width
const normalizedX = -1 + (pixelX + 0.5) * (2 / width);

// This is the vector for the pixel/slice.
const rayX = normalizedX * rayXAtImageEdge;
const rayY = 1;

console.log(normalizedX, rayX);
// This vector is in "eye space", so the next step would be to rotate the vector
// according to playerHeading, then cast it.
}


Notice that this ray is not of constant length, since rayY is always 1 — this actually makes the math simpler. You can think of it as if, if the image were in front of you in 3D space like a window pane, these were the coordinates of a specific pixel of the image, and the 1 is how far the image is in front of you.

Now in order to actually put that in your raycaster, get rid of the angle variable entirely, and replace your rayDirection calculation with this:

    let rayDirection = createVector(
normalizedX * rayXAtImageEdge,
-1,
);


(The depth (forward) direction got flipped to -1 to make it work with your existing choice of coordinate system. This is not unusual — different computer graphics systems have different choices of whether a depth coordinate increases when going “into the screen” or “out of the screen”.)

Finally, you'll find that the calculation of the height of walls is wrong. But we actually have the figure we need already:

// Change the raycast loop to let iter be visible outside:
let iter = 0;
for (; iter < RAY_LENGTH; iter++) {
...
}

...


This works because rayDirection's length in the image depth direction is always 1, and so iter is equal to the distance in the depth axis only to the hit, which is exactly the quantity that the sizes of objects should be divided by to make a perspective projection.
It will not be exactly the right height, because it doesn't account for the exact depth of the intersection between the ray and the hit cell, only the number of steps. Fixing that (with a ray-line intersection calculation, or a different raycasting algorithm) is another topic entirely. For now, your edges will be ragged by an amount that scales with CELL_SIZE, but in the large scale they'll be straight, which is what you wanted.