Measuring voltage during use, without disconnecting, is not very useful. You don't know the complete circuit so you don't know what you're measuring. Disconnect the motor while it's powered and you can get a much more sensible measurement. If possible, do also measure the current the motors take - another important parameter for your circuit design.
2.5V is a very low voltage for even a toy motor. It's possible, but it's even less than the output of two AA batteries. The product info page doesn't give any info on what power it uses, how many batteries or so. The manual controller I suppose is doing on/off for two motors in two directions, and that's what you're trying to replace.
In general, relays are not the best option for switching low voltages. They make noise and are bulky. I'd be thinking of MOSFET switches. As added advantage, you may be able to regulate the speed of the motors by PWM, something a relay can't. To get the motors run in two directions you'd need an H-bridge type of circuit - four MOSFETS or relays per motor.