It sends the values sampled by the ADC encoded in one of the three formats, so, as you said, it basically sends a series of numbers on the serial line. The baud rate depends on the sampling rate chosen and the number of bits per sample; With four bits and 7.2 kHz sampling rate a baudrate of 38400 baud/s is enough and so it's what I am using.
The program I am using for test is just the TwoPortsSerial sketch. I manually send commands to the modem through the Serial monitor.
The sequence of commands I am using (and that works when using my PC's serial port) is:
AT#CLS=8 (set voice mode)
AT#VSM=129, 7200 (G.721 encoding, 7.2 kHz sample rate)
AT#BDR=16 (turn off autobaud and set baud rate to 38400 bps)
AT#VRX (voice receive)
@jregminton The sample rate is 7.2 kHz, not as I said 8 kHz (I wrote 8 kHz by mistake because i got confused by the fact that the modem reports 8000 even if the selected baudrate is 7.2 kHz)
That said, one sample can use less than eight bits (it can be selected by the user out of two or three values, the default is four bits per sample) so that explains how it is possible. There is a table in the voice section of the document I sent that explains the relationship between sample rate and minimum baud rate.
P.S: Even setting the baudrate higher than the reported minimum of 38400 baud per second does not solve the problem.