So is it safe to just power the drivers from 12v & hook them up directly to the LEDs?
The current is an exponential function of voltage (and temperature dependent). This means that any attempt to power the LEDs through a pure voltage source is going to produce a current that could vary be many orders of magnitude depending on the temperature, the particular LED and batch etc - ie its not going to work - it'll could be far too dim one day and melt the next.
A non-pure voltage source (such as a power supply with a limit to the output current) might produce the result you want if the current limit is what you want, but you won't be able to run several different LEDs from it.
Individual current control is needed. Often this is a series resistor - gives crude current control, but good enough for most LED applications (except perhaps semiconductor lasers). A constant-current source is the ideal, you will have a brightness unaffected by variations in supply as a bonus.