Ah-HA! I don't need delays throughout the program, but I do need a big delay right after I open the /dev/ttyACM0 device and do the tcsetattr calls. Once I get past that with a big enough delay I can run full speed. Must be something in the linux ACM driver since I don't need any of the delays for talking to /dev/rfcomm0.