The USB-Serial adapter seems to be working if I test it with the Arduino Nano. Also, the LED-flashing frequency/duty cycle seems to be different. With the Nano it's almost constant-on while with the Atmega32 I can see it blink, although fast, still slow enough to see that they are individual pulses.
So I'm guessing somehow that 9600 isn't 9600, but how much slower I can't tell. The scope I have is an analog one, so, no memory to capture the bittrain.