Here are a few guesses for things you can try. Hopefully someone with experience doing what you want can pitch in if the following doesn't work.
I am not sure what state the interrupt mask is in when you set it but you could try the following:
// enable rx and interrupt on complete reception of a byte
UCSRC = (1<<URSEL)|(3<<UCSZ0)|(1<<USBS); // these were|=
UCSRB = (1<<RXEN)|(1<<RXCIE);
If that doesn't help, you may want to check the ATmega8 data sheet to verify the mask parameters
Also, although it probably compiles to the same thing, the Arduino code uses:
SIGNAL(SIG_UART_RECV)
where you have
ISR (UART_RX_vect)
If the Interrupt handler is not being called its worth trying the latter form. And you did say you tested with an led in the handler to verify it was not being called? If you connect the UART to a serial port it may be worth setting a standard baud rate and testing to see if that will trigger Interrupts.
You could also start with the initialization and ISR code in wiring_serial.c, validate it works with a standard serial port, then modify that code for DMX use.
Good luck!