Current matters. Same way you can't start a car with a bunch of AA's in series.
Button cells have high internal resistance, low output current availability, else they'd expend in no time flat for your project. So capacity (mAh) and current (mA) are the other traits you need to consider along with voltage.