The mathematical approach to this is to represent the other colors/light sources in terms of your standard primaries in the color space. Represent your target color and available light sources as vectors with, say, the RGB color space as the basis. Figuring out how to display your target color using your available sources then becomes a change of basis, a standard linear algebra operation, where the target basis consists of your LEDs/nonstandard primaries.
If the dimensions of the source and destination bases are different, the change of basis matrix won't be square and thus won't be invertible. In that case, you need to solve a system of linear equations, expressed in matrix form:
$$c_{RGB} = [r, y, b, v] \cdot c_{RYBV}$$
Where $c_{RGB}$ and $c_{RYBV}$ are the coordinate vectors of your desired color expressed in terms of RGB and custom primaries, respectively, and $r$, $y$, $b$ and $v$ are your custom primaries (basis vectors) in RGB coordinates.
Solving this system is another common linear algebra operation, and there are a variety of numerical and symbolical algorithms to do so. The system is over-constrained, so solutions will likely have a free variable, however since we can't physically produce negative light, nor do the primaries have infinite dynamic range, for most practical purposes each coefficient will have to be limited to be within the $[0, 1]$ range. This will render many colors unrepresentable under certain sets of primaries.
