Serial comms dropping characters on mega but not leonardo with same sketch

I’m building a neopixel project that uses serial comms to control some of the functions of the LED’s, sent from processing. The first iteration of the prototype was built on a Leonardo based platform (https://www.dfrobot.com/wiki/index.php/Arduino_Shield_for_Raspberry_Pi_2B_and_3B_SKU:DFR0327) using 40 neopixels (4 strands of 10 LEDs on seperate PWM pins, 5,6,9,10 on leonardo, but they are 2,3,4,5 on the Mega). It’s worth noting that I’m using the off-brand arduino mega by Elegoo and I have tried multiple USB cables to make sure it wasn’t a cable issue.
The serial comms between Processing and the Leonardo board works smoothly and flawlessly, but the same code (only code difference is which pins the LEDS are strung off of) on the Mega drops characters, usually the comma, upon receipt. This behavior is consistent across baud rates up to 250000. I’m using a modified example # 5 provided by Robin2 here: http://forum.arduino.cc/index.php?topic=396450.0 for the serial packet receiving with the same structure <0,0,0> Here is the exact code I’m using for that function:

  void recvWithStartEndMarkers() {
    
    static boolean recvInProgress = false;
    static byte ndx = 0;
    char startMarker = '<';
    char endMarker = '>';
    char rc;

    while (Serial.available() > 0 && newData == false) {
        rc = Serial.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                if(rc == ','){numArgs++;} // Counts commas to determine # of arguements
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                Serial.println(receivedChars);  //debugging
                ndx = 0;
                numArgs++; //There is one more arguement than there are commas.
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
            numArgs = 0;
        }
    }
  }

and I’m using this function to parse the data

  int parseData() {      // split the serial data into its parts
    char * strtokIndx; // this is used by strtok() as an index
    int args[numArgs]; 
    memset(args, 0, sizeof(args));   //Zero out the array to keep garbage out
    uint8_t argIndex = 0;
    strtokIndx = strtok(tempChars,",");      // get the first part - Serial Data Address
    args[argIndex] = atoi(strtokIndx);    // copy it to the first argument
    while (argIndex < numArgs){           //args[argIndex] != NULL &&
      argIndex ++;
      strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
      args[argIndex] = atoi(strtokIndx);     // convert this part to an integer 
                                             // and place in next argument  
    }
    #ifdef DEBUG
      for(byte i=0;i<numArgs;i++){
        Serial.print(args[i]); Serial.print(" ");
      }
      Serial.println("");
    #endif
    return args;
  }

Which passes the arguments into a packet handler which is just a switch statement:

void SerPktHndlng(int f_args[]){
// Packet reciever for Processing serial communications.
  // Recieves a packet converting packet arguements
  // strings to int's and arranging them into variables 
  // to populate parameters for the functions from the form:
  //    '<'[FunctionIndex]','[Argument1]','[Argument2]','[Arguement3]'>' 


      // Case switch to figure out where to put the data   
      switch (f_args[0]){
        case 1:
        // Brightness
          globalBright = f_args[1];
          FastLED.setBrightness(globalBright);     
          #ifdef DEBUG
            Serial.print("Brightness! "); Serial.println(f_args[1]);
          #endif  
        break;
        case 2:
        // Select sequence
          //Fade to black first?
          if(f_args[1] > 0){
            cycleShows = false;
            gCurrentPatternNumber = f_args[1] - 1; // set the index back to account for <functionIndex>
          }else{cycleShows = true;}
          #ifdef DEBUG
            Serial.print("Select Show! "); Serial.println(f_args[1]);Serial.println(cycleShows);
          #endif
        break;
        case 3:
        // Change the frame rate of the refresh (Not speed of sequence)
          frameRate = f_args[1]; 
          #ifdef DEBUG
            Serial.print("Frame Rate Change! "); Serial.println(frameRate);
          #endif
        break;
        case 4:
        // Change the speed of the show 
          beatSpeed = f_args[1];
          #ifdef DEBUG
            Serial.print("Speed Change! "); Serial.println(beatSpeed);
          #endif
        break;
        case 5:
        // Play/pause       
            isPaused = (bool)f_args[1];
            #ifdef DEBUG
              Serial.println("Show State! "); Serial.println(bool(f_args[1]));
            #endif
        break;
        case 6:
        // Get new color data
          if(!f_args[1]){ // First value selects between 0(HSV), and 1(RGB)
            curColorRGB.r = f_args[2];
            curColorRGB.g = f_args[3];
            curColorRGB.b = f_args[4];
            CHSV temp = rgbToHsv(curColorRGB.r,curColorRGB.g,curColorRGB.b);
            gHue = temp.hue;
            #ifdef DEBUG
              Serial.print("RGB Color Change!\tR:"); Serial.print(curColorRGB.r);
              Serial.print("\tG:"); Serial.print(curColorRGB.g);
              Serial.print("\tB:");Serial.println(curColorRGB.b);
            #endif           
          }else{
            curColorHSV.hue         = f_args[2];
            curColorHSV.saturation  = f_args[3];
            curColorHSV.value       = f_args[4];
            gHue = curColorHSV.hue;
            #ifdef DEBUG
              Serial.print("HSV Color Change!\tH:"); Serial.print(curColorHSV.hue);
              Serial.print("\tS:"); Serial.print(curColorHSV.saturation);
              Serial.print("\tV:");Serial.println(curColorHSV.value);
            #endif
            }
        break;
        case 7:
            clrCngRt = f_args[1];
            #ifdef DEBUG
              Serial.print("Color Change Rate Change! "); Serial.println(clrCngRt);
            #endif
        break;
        default:
          // Out of bounds
          #ifdef DEBUG
            Serial.println("!!!!! ERROR on switch case input!!!!!!");
          #endif
        break;
      } // end switch(paramAddy)
      
    } // end SerPktHndlng()

I can see in the output on processing that the serial messages have dropped characters, which usually seems to be the comma (which may just be coincidence) For reference, the messages coming back from the arduino are preceded with a single "* ", everything else is output of processing.

===========Sending (in hex):=================
<1,55>
***brightnessSet - GKnob >> GEvent.VALUE_STEADY @ 6512


===========Sending (in hex):=================
<1,49>
* 1

***brightnessSet - GKnob >> GEvent.VALUE_STEADY @ 6531


===========Sending (in hex):=================
<1,46>
* 1,

***brightnessSet - GKnob >> GEvent.VALUE_STEADY @ 6548


===========Sending (in hex):=================
<1,39>
* 1,39

***brightnessSet - GKnob >> GEvent.VALUE_STEADY @ 6565


===========Sending (in hex):=================
<1,33>
* 1,33

***brightnessSet - GKnob >> GEvent.VALUE_STEADY @ 120209


===========Sending (in hex):=================
<1,36>
* 1

***brightnessSet - GKnob >> GEvent.VALUE_STEADY @ 120242


===========Sending (in hex):=================
<1,39>
* 1

***brightnessSet - GKnob >> GEvent.VALUE_STEADY @ 120309


===========Sending (in hex):=================
<1,44>
* 14

***brightnessSet - GKnob >> GEvent.VALUE_STEADY @ 120426


===========Sending (in hex):=================
<1,48>
* 1,48

***brightnessSet - GKnob >> GEvent.VALUE_STEADY @ 120459


===========Sending (in hex):=================
<1,59>
* 1

***brightnessSet - GKnob >> GEvent.VALUE_STEADY @ 120493


===========Sending (in hex):=================
<1,65>
* 1,65

***brightnessSet - GKnob >> GEvent.VALUE_STEADY @ 120509


===========Sending (in hex):=================
<1,72>
***brightnessSet - GKnob >> GEvent.VALUE_STEADY @ 120526


===========Sending (in hex):=================
<1,78>
* 1,72

* 1,78

If the receive buffer in the Arduino is full, it will start dropping data.

The Leonardo uses a different mechanism purely based on USB while the Mega uses a real uart.

Solution is not to send too much at a time; you can add a handshake were a sent message needs to be acknowledged by the receiver before the transmitter can send the next bunch of data.

sterretje:
If the receive buffer in the Arduino is full, it will start dropping data.

The Leonardo uses a different mechanism purely based on USB while the Mega uses a real uart.

Solution is not to send too much at a time; you can add a handshake were a sent message needs to be acknowledged by the receiver before the transmitter can send the next bunch of data.

That's kinda what I was thinking, the problem lies in how the dedicated USB handling of the leonardo differs from the Mega. However, the serial buffer is limited to 32 characters by the recvWithStartEndMarkers() function. Also, you can see in the output that it is dropping characters after only 3 or 4 have been sent.

The handshake approach has been on my mind, but I've been intentionally trying to limit the serial communication to allow the LED's to change functionality based on the serial control as smoothly as possible. I'll try and build in a handshake and see if that will help.

Another thing I have tried is inserted a 10ms delay in various points surrounding the recvWithStartEndMarkers() to hopefully settle things down, but it hasn't seemed to help at all.

So someone told you that your receive buffer is filling up because you are sending data faster than it can process it and you thought the answer would be to slow down the receiver some more?

The sender is the one going too fast.

I've toyed with the Mega's USART vis-a-vis programming direct (without Serial class). For what it's worth, USART0 functioned fine, fully up to 1Mbps. The other three however, seemed only to be able to process characters up to 200K. I never went further with trying to find out why.

RoBob:
However, the serial buffer is limited to 32 characters by the recvWithStartEndMarkers() function.

You can change the 32 to any number you want. If you have enough SRAM then make it a bit bigger than your biggest message. If you don't have enough SRAM then my system is probably not appropriate.

I would not try to count commas in the function recvWithStartEndMarkers()

...R

The Leonardo differs from other typical boards in that the ATmega32u4 has built-in USB communication, eliminating the need for a secondary processor

the NeoPixel library temporarily disables all interrupts when issuing data, then re-enables them when done. That might be an explanation of your serial challenge on a mega

Delta_G:
So someone told you that your receive buffer is filling up because you are sending data faster than it can process it and you thought the answer would be to slow down the receiver some more?

The sender is the one going too fast.

Delta, the snark is totally useless in a learning environment such as this forum. I made very sure I posted according to the stickies to try and avoid being an "annoying noob", but it seems to have been too little.
No, no one "told " that. I tried it before begrudgingly coming here to ask for help, fully knowing some self-proclaimed C++ guru would inject snark to bolster their own ego by attempting to make me feel stupid. I remembered having a similar problem a long while back with serial comms and I recollect (possibly erroneously) the issue was solved by inserting a small delay on the receiver side. And as per the stickies at the top of the thread, I wanted to provide as much information as possible to the kind hearted folks who would give me advice.

Some of you guys need to remember how when one is so immersed in a problem, one sometimes overlook obvious issues. Lord knows you have most likely done it yourself at least as much as people coming here asking for help. Also, the majority of people coming here are total beginners and it would be very unfortunate if someone's curiosity was smashed by some "veteran coder's" snark.

I get it, believe me. I have lurked these forums for years and have seen all of the repetitively annoying and egregious violations of the forum rules. I know how patience for cognitive laziness wears thin. But remember that all those stars behind your screen name make you somewhat of an authority figure on the subject, at least here.

What you do, what you say, and how you act is totally contagious. Lead by example, not ridicule. We need more people who are not afraid to dive into this complicated subject matter. Fear of ridicule is one of the most powerful influencers of behavior in humans. A good example is how there are purportedly more people afraid of public speaking than of death. (Anecdotal? Somewhat. I've C++ bugs to ferret out, no time for researching psychological statistics)

All that rant being said, you are totally correct and I agree that the delay should be on the sender. I'll give this a try and appreciate your advice despite my rant above.

DKWatson:
I've toyed with the Mega's USART vis-a-vis programming direct (without Serial class). For what it's worth, USART0 functioned fine, fully up to 1Mbps. The other three however, seemed only to be able to process characters up to 200K. I never went further with trying to find out why.

Is USART0 the one connected to digital pins 0 and 1 (tx/rx) ?

Robin2:
You can change the 32 to any number you want. If you have enough SRAM then make it a bit bigger than your biggest message. If you don't have enough SRAM then my system is probably not appropriate.

I would not try to count commas in the function recvWithStartEndMarkers()

...R

Hmmm, I was doing that because the messages are of different lengths and I was getting compile errors from leaving the length of the array uninitialized. I'll comb back through the code and see if I can get rid of that instruction. Thanks! Also, thanks for your Serial write-up! It saved me from parseInt().

J-M-L:
The Leonardo differs from other typical boards in that the ATmega32u4 has built-in USB communication, eliminating the need for a secondary processor

the NeoPixel library temporarily disables all interrupts when issuing data, then re-enables them when done. That might be an explanation of your serial challenge on a mega

I didn't even consider that. However, I'm using the FastLED library. This is not to say that library doesn't do the same thing (I suppose I'll find out here in a few hours). Thanks for the advice! I'll check it out and report back with what I find.

You can have a look at this

The single-wire control protocol used by NeoPixels requires a very steady data stream at 800 kilobits per second. Bits must be issued at a precisely controlled rate, putting heavy requirements on any other interrupt based system

May be you can try to suspend updating the neopixels when reading from Serial.

J-M-L:
You can have a look at this

The single-wire control protocol used by NeoPixels requires a very steady data stream at 800 kilobits per second. Bits must be issued at a precisely controlled rate, putting heavy requirements on any other interrupt based system

May be you can try to suspend updating the neopixels when reading from Serial.

Oh, wow! Okay! This looks promising! As much as I've been on that wiki, I'm kinda disappointed in myself that I didn't read that page. I'll give this a try and let you know how it went. Thanks!

Is USART0 the one connected to digital pins 0 and 1 (tx/rx) ?

Yes.