What sort of modem, each different baud rate used different ways of mapping the data signal to the audio signal.
Transmission is OK but reception might be a lot more tricky.
The other vital thing is that the free running oscillator clock is unlikely to be accurate enough.