Issue with serial communication. Lost bytes.

Dear Arduino users

I want to implement DMX receiver on atmega328p. I decided to start with code from this thread. I used ft232 to send DMX data from QLC+ on my computer to arduino board. For test purposes i used NeoPixel to indicate values of first channels.

Hardware connections:
Arduino tx <-> ft232 rx
Arduino rx <-> ft232 tx
Arduino D9 <-> NeoPixel data
Arduino Gnd <-> ft232 Gnd <-> Neopixel Gnd
Arduino 5v <-> ft232 5v <-> Neopixel 5v

Code:

#include <Adafruit_NeoPixel.h>
volatile uint8_t  buffer[513]; //array of DMX vals (raw)
volatile uint16_t DmxAddress;    //start address
enum {IDLE, BREAK, STARTB, STARTADR}; //DMX states
volatile uint8_t gDmxState;

#define PIXELS 30
#define RX_STATUS_PIN 13

Adafruit_NeoPixel pixels = Adafruit_NeoPixel(PIXELS, 9, NEO_GRB + NEO_KHZ800);
void setup () {
  pixels.begin();
  //TIMSK0 = 0;
  UCSR0B=(1<<RXCIE0)|(1<<RXEN0)|(1<<TXEN0);//enable tx,rx and rx_interrupt
  UCSR0C=0x0E;//8N2
  uint16_t baud_setting = (F_CPU / 16 / 250000 - 1);
  UBRR0H = baud_setting >> 8;
  UBRR0L = baud_setting;
  
  pinMode(RX_STATUS_PIN, OUTPUT);
  
  for(int i=0;i<100;i++)
    buffer[i]=40; 
}

void uart_putchar(char c) {
    loop_until_bit_is_set(UCSR0A, UDRE0); /* Wait until data register empty. */
    UDR0 = c;
}

void loop() {
   //digitalWrite(13,buffer[1]==67);
   //for(int i=0;i<30;i++){
       //pixels.setPixelColor(i, pixels.Color(buffer[1+i*3+0],buffer[1+i*3+1],buffer[1+i*3+2])); 
   //}
   pixels.setPixelColor(0,pixels.Color(buffer[0],buffer[1],buffer[2]));
   pixels.show();
}

volatile static  uint16_t DmxCount;
ISR(USART_RX_vect) {
    digitalWrite(RX_STATUS_PIN, HIGH);
    
    uint8_t  USARTstate= UCSR0A;    //get state before data!
    uint8_t  DmxByte   = UDR0;          //get data
    uint8_t  DmxState  = gDmxState; //just load once from SRAM to increase speed
    //uart_putchar(DmxByte);
    if (USARTstate &(1<<FE0)){               //check for break
            DmxCount =  DmxAddress;         //reset channel counter (count channels before start address)
            gDmxState= BREAK;
    }

    else if (DmxState == BREAK){
            if (DmxByte == 0) gDmxState= STARTB;  //normal start code detected
            else                      gDmxState= IDLE;
    }

    else if (DmxState == STARTB){
            if (--DmxCount == 0){    //start address reached?
                    DmxCount= 1;            //set up counter for required channels                  
                    buffer[0]= DmxByte; //get 1st DMX channel of device
                    gDmxState= STARTADR;
            }
    }
    else if (DmxState == STARTADR){
            buffer[DmxCount++]= DmxByte;    //get channel
            if (DmxCount >= sizeof(buffer)) //all ch received?
                    gDmxState= IDLE;        //wait for next break
    }
    
    digitalWrite(RX_STATUS_PIN, LOW);                                                
}

Result: Neopixel flashes almost randomly. From what i see it behaves like it skips some bytes. Using logic analyzer @16Mhz i found out that arduino doesn't enter interrupt from time to time and ignores parts of message. For example is triggers interrupt for bytes 1-15 and 27-39 etc.

I attached screenshot from PulseView app. D1 i Arduino rx and D2 is Arduino D13.

I spend several days tinkering with this setup and trying to solve this issue. I wish to hear any suggestions on debugging this.

I'm looking forward for replies! Best regards!

You can't use Neopixels with serial, this is because when writing to the Neopixels the interrupts must be turned off in order to achieve the correct timing. This causes bytes to be missed from the serial port.

This link refers to servos, but serial is an other thing that will show this problem.

why not use Serial?

Thank You for this suggestion. I'm going to get rid of NeoPixel and use normal LED. I will see how it goes.Also do You think that atmega timer interrupts might also lead to lost bytes?

why not use Serial?

I'm not using Serial, because i must detect frame errors and couldn't get it to work. With avr registers I at least got some success.

Change to Dotstar LEDs. They can coexist with interrupts.

You don't even have to change to a different library.

PlaneG:
Thank You for this suggestion. I'm going to get rid of NeoPixel and use normal LED. I will see how it goes.Also do You think that atmega timer interrupts might also lead to lost bytes?
I'm not using Serial, because i must detect frame errors and couldn't get it to work. With avr registers I at least got some success.

The issue is that to feed neopixels their correct timing all interrupts must be completely disabled until all
the LED data has been streamed out making interrupts useless for other purposes.
Timer interrupts aren't antisocial like this unless you abuse them!

However to receive serial bytes every 4µs to support DMX512 means you need a fast processor to
allow any other interrupts enough cycles - ATmega based Arduinos probably cannot do this as
there is no hardware serial buffer.

Update: After removing NeoPixel from my setup i catch all interrupts. However I still can't get int working. LED flashes randomly. From what i see it's some issue in QLC+ or ft232. Arduino catches frame error and resets DMXcounter which results in false readings.

volatile uint8_t  buffer[513]; //array of DMX vals (raw)
volatile uint16_t DmxAddress;    //start address
enum {IDLE, BREAK, STARTB, STARTADR}; //DMX states
volatile uint8_t gDmxState;

#define PIXELS 30
#define RX_STATUS_PIN 13

void setup () {
  DmxAddress=1;
  //TIMSK0 = 0;
  UCSR0B=(1<<RXCIE0)|(1<<RXEN0)|(1<<TXEN0);//enable tx,rx and rx_interrupt
  UCSR0C=0x0E;//8N2
  uint16_t baud_setting = (F_CPU / 16 / 250000 - 1);
  UBRR0H = baud_setting >> 8;
  UBRR0L = baud_setting;
  
  pinMode(RX_STATUS_PIN, OUTPUT);
  
  for(int i=0;i<100;i++)
    buffer[i]=40; 
}

void uart_putchar(char c) {
    loop_until_bit_is_set(UCSR0A, UDRE0); /* Wait until data register empty. */
    UDR0 = c;
}

void loop() {
   digitalWrite(13,buffer[1]>=40);
}


ISR(USART_RX_vect) {
    //digitalWrite(RX_STATUS_PIN, HIGH);
    static  uint16_t DmxCount;
    uint8_t  USARTstate= UCSR0A;    //get state before data!
    uint8_t  DmxByte   = UDR0;          //get data
    uint8_t  DmxState  = gDmxState; //just load once from SRAM to increase speed
    //uart_putchar((byte)DmxCount);
    if (USARTstate &(1<<FE0)){               //check for break
            DmxCount =  DmxAddress;         //reset channel counter (count channels before start address)
            gDmxState= BREAK;
    }

    else if (DmxState == BREAK){
            if (DmxByte == 0) gDmxState= STARTB;  //normal start code detected
            else                      gDmxState= IDLE;
    }

    else if (DmxState == STARTB){
            if (--DmxCount == 0){    //start address reached?
                    DmxCount= 1;            //set up counter for required channels                  
                    buffer[0]= DmxByte; //get 1st DMX channel of device
                    gDmxState= STARTADR;
            }
    }
    else if (DmxState == STARTADR){
            buffer[DmxCount++]= DmxByte;    //get channel
            if (DmxCount >= sizeof(buffer)) //all ch received?
                    gDmxState= IDLE;        //wait for next break
    }
    
    //digitalWrite(RX_STATUS_PIN, LOW);                                                
}

I attach weird framing error screenshot (sending 64 on 2nd ch and 50 on 6th ch) and PulseView capture file.

I have no idea how to fix this now. Probably I will post issue on QLC+ forum.

Change to Dotstar LEDs. They can coexist with interrupts.

Also Thanks for suggesting NeoPixels alternatives. I'm not going to use them in final project and just wanted to use then for debugging purposes (and it turned out they introduced some issues). In the final project i'm going to send some channel data using NRF24L01 radio.

However to receive serial bytes every 4µs to support DMX512 means you need a fast processor to
allow any other interrupts enough cycles - ATmega based Arduinos probably cannot do this as
there is no hardware serial buffer.

If atmega turns out to be too slow then my options are either to implement my own QLC+ output device or use different processor. Would using 20Mhz cristal help with anything?

Thank You for Your Help!

If only 25% faster fixes the problem then you could use the 20MHz crystal. Or optimise the inner loop of the program a bit better.

There are faster Arduinos. I like the Teensy series a lot. While Teensy is not "official Arduino" it is just as easy to program with the same IDE.

OP, it looks like you don't use all 513 bytes of the DMX messaging. In your setup() you seem to initialize the first hundred bytes to 40, so I'll assume you're only interested in a subset of the message. If you need more and this works, there might be a way to expand it.

You need to offload how much you're doing in the ISR. In the code below the idea is to receive the status and rx'd byte in the ISR and bump a pointer and that's it; the receive processing is done in the mainline.

Compiles but not tested.

volatile uint16_t DmxAddress;    //start address

enum {IDLE, BREAK, STARTB, STARTADR}; //DMX states
volatile uint8_t gDmxState;

#define PIXELS          30
#define RX_STATUS_PIN   13

//add receive buffers into which the ISR can dump status and rx'd chars 
#define RX_BUFFER_SIZE  128      //use a power of 2 for easy size masking using &
uint8_t 
    buffer[RX_BUFFER_SIZE];     //array of DMX vals (raw)
volatile uint8_t
    rxStatus[RX_BUFFER_SIZE];
volatile uint8_t
    rxBuffer[RX_BUFFER_SIZE];
uint8_t
    rxHead,                     //head and tail pointers; ISR bumps head, process chases with tail
    rxTail;
bool
    bOverrun;
    
void setup () 
{
    DmxAddress = 1;
    gDmxState = IDLE;
    rxHead = 0;
    rxTail = 0;
    bOverrun = false;
    
    //TIMSK0 = 0;
    UCSR0B=(1<<RXCIE0)|(1<<RXEN0)|(1<<TXEN0);   //enable tx,rx and rx_interrupt
    UCSR0C=0x0E;                                //8N2
    
    uint16_t baud_setting = (F_CPU / 16 / 250000 - 1);
    UBRR0H = baud_setting >> 8;
    UBRR0L = baud_setting;
        
    pinMode(RX_STATUS_PIN, OUTPUT);
  
    for(int i=0;i<100;i++)
        buffer[i]=40; 
}

void uart_putchar(char c) 
{
    loop_until_bit_is_set(UCSR0A, UDRE0); /* Wait until data register empty. */
    UDR0 = c;
}

void loop() 
{
    RxProcess();
    //add some indication if RX ISR overruns the rxprocess()
    //if( bOverrun )
    //{
    //    ...
    //    
    //}//if
    
    digitalWrite( 13, buffer[1] >= 40 );
    
}//loop

void RxProcess( void )
{
    static  
        uint16_t DmxCount;
    byte
        rxst,
        rxch,
        tempHead;

    tempHead = rxHead;
    
    while( rxTail != tempHead )
    {
        rxst = rxStatus[rxTail];
        rxch = rxBuffer[rxTail];
        switch( gDmxState )
        {
            case    IDLE:
                if( rxst & (1<<FE0) )
                {
                    DmxCount =  DmxAddress;
                    gDmxState = BREAK;
                    
                }//if

            case    BREAK:
                if( rxch == 0) 
                    gDmxState = STARTB;
                else                      
                    gDmxState = IDLE;
                
            break;

            case    STARTB:
                if( --DmxCount == 0 )
                {    
                    //start address reached?
                    DmxCount = 1;               //set up counter for required channels                  
                    buffer[0]= rxch;            //get 1st DMX channel of device
                    gDmxState= STARTADR;
                    
                }//if
                
            break;

            case    STARTADR:
                buffer[DmxCount++]= rxch;       //get channel
                if( DmxCount >= RX_BUFFER_SIZE ) //all ch received?
                    gDmxState= IDLE;            //wait for next break
            break;
            
        }//switch

        rxTail = (rxTail + 1) & (RX_BUFFER_SIZE-1);
        
    }//while
        
}//RxProcess

ISR(USART_RX_vect) 
{
    rxStatus[rxHead] = UCSR0A;
    rxBuffer[rxHead] = UDR0;

    rxHead = rxHead + 1 & (RX_BUFFER_SIZE-1);
    if( rxHead == rxTail )
        bOverrun = true;
       
}//ISR

The speed of the ISR is not the issue. Making it more efficient won't solve the problem.

The problem is the amount of time that NeoPixel keeps interrupts disabled. If that is longer than 2 characters, then one will be dropped. DMX is 250000baud, so characters come pretty fast.

MorganS:
The speed of the ISR is not the issue. Making it more efficient won't solve the problem.

The problem is the amount of time that NeoPixel keeps interrupts disabled. If that is longer than 2 characters, then one will be dropped. DMX is 250000baud, so characters come pretty fast.

re Neopixels OP said I'm not going to use them in final project and just wanted to use then for debugging purposes up in reply #6.

In my code for him to try there's no reference to Neopixels. Just wanted to see if the thing could manage to keep up with a 250kbps serial input and not give goofy result with a tidier ISR and moving some processing into the mainline.