but if the LEDs have to be matched for brightness it won't work.
No so.
There will be very little difference in brightness with the same LED running at 20mA controlled by a resistor from 5V. Lower than a JND (just noticeable difference).
Any brightness variation will come from the LEDs themselves not the variation in current. As such constant current control will not help.