However, I can't find photodiodes that have narrow wavelengths, they all seem to be made for a broad spectrum.
Interesting project - I will look forward to see what you find.
One approach could be to use multiple low cost LDR's, but add colored filter caps on each one. E.g. Red, Green, Blue or whatever wavelength equivalents you would need.
I also remember seeing (could not find it again) what I recall as a low cost color sensor that would give you the relative mix of wavelengths and through this allow you to determine color. This in combination with a broad spectrum LDR (for intensity) can perhaps also be of value.