Arduino Serial communication and ram.

I am having a hard time receiving more than 64 bytes of data through serial. I have tried to read it in as fast as its being written to avoid using the entire buffer. However I have had no luck. One end is using C++ code to write to serial. Thearduino end is somewhat like this: While(!Serial.available()); //wait for input While(serial.available()>o){ rmagByteMap[byteIndex] = serial.read(); byteIndex++; }

Also I want to be able to receive upwards of 2kb. Is this just not possible with the 2k of memory on the atmega328? What I am doing is making a scrolling matrix display. Myax message size is 256 characters and each character can be an 8*8 matrix font do that's a 2048 byte array...and I have two colors. is thus just not possible?

hardware and software serial have hard coded 64 byte buffers in the Arduino core code

You could change this, but I doubt it would solve your overall problem, which is lack of memory

Your simplest option is to use an SD card as memory. You can just do "raw" writes and "raw" reads, you don't need to use anything fancy like the Fat file system, if you are only using the SD as slow ram for the Arduino

I suspect that your other issue is that when you finish reading the data, your code goes off to do something else, like update the display, and while its doing that, the Arduino gets sent more data, which will overflow the 64 byte buffer

(I have the same issue on an RS232 based wifi board that I'm using at the moment - in fact I just posted about my issue to the networking section ;-)

If you are loosing data because your code is not ready to receive it. I think the options are either to use flow control if its available (probably not) Or to use the Serial Event See http://arduino.cc/en/Tutorial/SerialEvent

In which case you need to write code that separately handles and stores the incoming serial data, and other code that monitors the status of the stored incoming data and takes the appropriate action

kurtruk: I am having a hard time receiving more than 64 bytes of data through serial. I have tried to read it in as fast as its being written to avoid using the entire buffer. However I have had no luck. One end is using C++ code to write to serial. Thearduino end is somewhat like this: While(!Serial.available()); //wait for input While(serial.available()>o){ rmagByteMap[byteIndex] = serial.read(); byteIndex++; }

In isolation it's impossible to say what's wrong with that fragment, but the Arduino would be capable of reading from the serial port far faster than new data can arrive, if you do it right. If the serial buffer is overflowing, that suggests that your sketch isn't reading from the serial port for an extended period. The most likely reason for that is a design mistake in the sketch.

I notice that the code above doesn't have anything to stop it overflowing the bounds of the buffer.

kurtruk: Also I want to be able to receive upwards of 2kb. Is this just not possible with the 2k of memory on the atmega328? What I am doing is making a scrolling matrix display. Myax message size is 256 characters and each character can be an 8*8 matrix font do that's a 2048 byte array...and I have two colors. is thus just not possible?

I suspect that your memory requirements could be reduced substantially if you encode the data correctly. Do you really need to be able to control each pixel in a 256x8x8 message independently? If your message is going to consist of text characters, it would be possible to store it much more efficiently by defining it using a font.

Here’s my entire code if you would like to take a look:

/*50 X 8 Scrolling Matrix
 Author: Kurt
 Date 2/27/14
 Version 3.0 */
 
 #include "DisplayArduinoClass.h"
 
 void setup(){
   
   Serial.begin(115200);
   
   DisplayArduinoClass obj;
   obj.Setup(); //Setup Pins and Set as OUTPUTS
   for(int i=0;i<50;i++){obj.rmsgByteMap[i] = 0;}   //set 1st and last fifty to zero
   for(int i=0;i<50;i++){obj.gmsgByteMap[i] = 0;}   //set 1st and last fifty to zero
   obj.refresh();
   

  
}

 void loop(){
    
   DisplayArduinoClass obj;
   obj.Setup();
   while(!Serial.available());   //wait for input
   delay(10);
   int byteIndex =0;
   char incomingString[6] = {0,0,0,0,0,0};
   while (Serial.available()>0){    //read incoming string    
      incomingString[byteIndex] = Serial.read(); 
      byteIndex++;
   }
   /* Confirm Connection*/
  
 
  if (incomingString [0] == 'H' && incomingString [1] == 'e' && incomingString [2] == 'l' && incomingString [3] == 'l' && incomingString [4] == 'o' && incomingString [5] == '?'){
    Serial.write(72); Serial.write(101);  Serial.write(108);  Serial.write(108);  Serial.write(111);  Serial.write(33); 
     
  }
  else Serial.print(incomingString);
  
  while(!Serial.available());   //wait for input  
  obj.Speed = Serial.read(); 
  
  
  

  
  
   while(1){
     while(!Serial.available());   //wait for input  //check if input is new? if not continue
       obj.numberOfCols = Serial.read(); 
      // byte numberOfPackets = ((obj.numberOfCols/4)+1);
     
     
     
     
     for(int i=0;i<obj.numberOfCols+100;i++){obj.rmsgByteMap[i] = 0;} //clear rmsgByteMap   
     for(int i=0;i<obj.numberOfCols+100;i++){obj.gmsgByteMap[i] = 0;} //clear rmsgByteMap   
      int rIndex = 50;//+(64*Packets);
     while(!Serial.available()); //wait for input
     //for(int Packets=0; Packets<((obj.numberOfCols/64)+1); Packets++){
       while(Serial.available()){
         delay(2);
        
         if (Serial.available()>0){
        
           obj.rmsgByteMap[rIndex] = Serial.read();  //write gmsgByteMap
           rIndex++;
         }
       }
         
   //}
     
       
     /*while(!Serial.available()); //wait for input
       delay(200);
       int gIndex = 50;
       while (Serial.available()>0){
         obj.gmsgByteMap[gIndex] = Serial.read();  //write gmsgByteMap
         gIndex++;
       }*/
       
    do{
      obj.Shift = 0;
      for(int j=0;j<obj.numberOfCols+51;j++){  //scroll
      
        for(int i=0;i<obj.Speed;i++){  //speed
          obj.refresh();
        }  
        obj.Shift++;
      }
   } while(!Serial.available());
    
    
   

  

     
   }
}

What I have been doing is sending the byte array that has been defined by a font(on the computer side) to the arduino.

I don’t quite understand what you mean about serial.event.

Here’s the C++ code that’s sending the data:

        SP->WriteData(dfObj.rmsgByteMap, dfObj.numberOfCol[0]);

bool Serial::WriteData(unsigned char *buffer, unsigned int nbChar){
    DWORD bytesSend;
    if(!WriteFile(this->hSerial, (void *)buffer, nbChar, &bytesSend, 0)){ //Try to write the buffer on the Serial port
        ClearCommError(this->hSerial, &this->errors, &this->status);  //In case it don't work get comm error and return false
        return false;
    }
    else
        return true;
}

rmsgByteMap is the byte map of the characters defined by a font. ie.
1 0 0 0 1 0 0 0
1 0 0 1 0 0 0 0
1 0 1 0 0 0 0 0
1 0 1 0 0 0 0 0
1 1 0 0 0 0 0 0
1 0 1 0 0 0 0 0
1 0 0 1 0 0 0 0
1 0 0 0 1 0 0 0
= {255, 8, 52, 66 129} this is what is being sent

wow.

Loads of things in the code have possible issues, and lots of the code could be simplified

I’m not sure of the DisplayObjectClass stuff, as you don’t seem to have posted that code

You have not bounds checked this. i.e if you get more than 6 bytes you will get a buffer overflow and unexpected results

   char incomingString[6] = {0,0,0,0,0,0};
   while (Serial.available()>0){    //read incoming string    
      incomingString[byteIndex] = Serial.read(); 
      byteIndex++;
   }

Anywhere you are doing this

   for(int i=0;i<50;i++){obj.rmsgByteMap[i] = 0;}

you can use

memset(obj.rmsgByteMap,,0, 50 *SIZE_OF_ONE_ELEMENT);

(You just need to put in a #define for SIZE_OF_ONE_ELEMENT e.g.

#define SIZE_OF_ONE_ELEMENT sizeof(byte)

If its an array of bytes

for

 if (incomingString [0] == 'H' && incomingString [1] == 'e' && incomingString [2] == 'l' && incomingString [3] == 'l' && incomingString [4] == 'o' && incomingString [5] == '?')

just use strncmp()

e.g

strncmp(incomingString ,"Hello?",6);
           obj.rmsgByteMap[rIndex] = Serial.read();  //write gmsgByteMap
           rIndex++;

to

           obj.rmsgByteMap[rIndex++] = Serial.read();  //write gmsgByteMap

may save a few cycle or two

However I suspect your main issue is here

    do{
      obj.Shift = 0;
      for(int j=0;j<obj.numberOfCols+51;j++){  //scroll
      
        for(int i=0;i<obj.Speed;i++){  //speed
          obj.refresh();
        }  
        obj.Shift++;
      }
   } while(!Serial.available());

I suspect that this bit

  for(int j=0;j<obj.numberOfCols+51;j++){  //scroll
      
        for(int i=0;i<obj.Speed;i++){  //speed
          obj.refresh();
        }  
        obj.Shift++;
      }

takes a while to finish, and while its doing this, any serial input will buffer until the 64 byte buffer is full, and after that you will loose the first data to arrive

Please read the page on Serial Event http://arduino.cc/en/Tutorial/SerialEvent

I suspect your simplest option is to do this

Put this in the vars declaration at the top

volatile boolean serialDataHasArrived = false;

Add this function

void serialEvent() {
  serialDataHasArrived =true;
}
    do{
      obj.Shift = 0;
      for(int j=0;j<obj.numberOfCols+51;j++){  //scroll
      
        for(int i=0;i<obj.Speed;i++){  //speed
          obj.refresh();
        }  
        obj.Shift++;
      }
   } while(!serialDataHasArrived );

This should stop your loop as soon as more data starts to arrive.

I’m not sure if this is ideal in terms of what the do… while loop is doing, but it would probably get around your initial problem

As far as simplifying the code, I am new and lots of the things you mentioned I have never heard or dealt with before. I would rather do it the way I understand. At the end of the day simplifying it will not free up enough ram. I agree there are parts that need to be simplified, but I do not want to use C command language that I don’t know when I could use something that I do know and understand.

The “bit” that takes awhile to finish, is designed to be that way. It is the scrolling function of the display. If it didn’t take a bit to finish then it would be impossible for the human eye to read. If I interrupt that loop then everything mess up for the viewing experience.

You have not bounds checked this. i.e if you get more than 6 bytes you will get a buffer overflow and unexpected results
Code:
char incomingString[6] = {0,0,0,0,0,0};
while (Serial.available()>0){ //read incoming string
incomingString[byteIndex++] = Serial.read();

}

Should I do something like this:

 char incomingString[6] = {0,0,0,0,0,0};
   if (Serial.available()=6){    //read incoming string    
      incomingString[byteIndex] = Serial.read(); 
      byteIndex++;

I basically have three options:

  1. Use a atmega1284 which has 16kb of RAM.

  2. Use some external ram, eeprom, or sd card.

  3. Send the byte array that is only being displayed at that second. Then send the shifted byte array and so on. My matrix is 50 x 8 so I would recieve a 50 byte array, then receive another 50 byte array that is shifted for the scrolling effect.

Option number 3 would probably make my message scroll slower than I would like.

None the less I have to be able to recieve 64+ bytes over the buffer. Should I somehow clear the buffer, then send the bytes from the computer, and then recieve them fast from the arduino?

Or you could just do it correctly on the Uno. This code is total and completely wrong,

    while (Serial.available()>0){    //read incoming string    
      incomingString[byteIndex] = Serial.read(); 
      byteIndex++;
   }

Just because there is at least one char in the buffer does NOT mean that all the message has arrived. Serial comms is SLOW compared to the speed of the Uno.

So the only thing you can do with your current code is to scrap it. You need to start over again.

The correct way to deal the messages is not to wait until you have the whole message but to deal with each part of the message as it comes in.

Mark

I am a great believer in taking code out of loop() and putting it into functions. That way it becomes possible to get a quick overview of the flow of the logic.

For example you might have something like this

void loop() {
   getSerialData();
   updateDisplay();
}

void getSerialData() {

}

void updateDisplay() {

}

and each of the two functions would just have code for one activity.

The getSerialData() function would have nothing to do except save incoming bytes into a suitable buffer. It wouldn't try to interpret the bytes, except perhaps to detect the start and end of a message and record the fact that a complete message had been received.

The updateDisplay() function would work with the most recent complete message to do whatever is wanted with that. It wouldn't need to know whether a new message was coming or what might be in the new message until the complete message has been received. And the display function can operate at whatever speed it likes without interfering with the speed at which the data is received.

I don't like the serialEvent() function because it takes away your control over the incoming serial data. Polling with if (Serial.available) { leaves you in full control. There is no need to deal with every character the instant it arrives - the serial buffer allows a lot of flexibility.

...R

Or you could just do it correctly on the Uno.
This code is total and completely wrong,
Code:
while (Serial.available()>0){ //read incoming string
incomingString[byteIndex] = Serial.read();
byteIndex++;
}
Just because there is at least one char in the buffer does NOT mean that all the message has arrived. Serial comms is SLOW compared to the speed of the Uno.

So the only thing you can do with your current code is to scrap it. You need to start over again.

The correct way to deal the messages is not to wait until you have the whole message but to deal with each part of the message as it comes in.

Mark

It can’t be completely wrong…that part works!

There is no way that I am scraping my entire code. There are hundreds of lines that do not even deal with the problem I have mentioned.
Here’s an updated version:

  while (Serial.available()==6)    //wait for buffer to recieve all of incoming string
   for(int i=0;i<6;i++){
     incomingString[i] = Serial.read();
   }

In response to Robin2: I do have lots of functions outside of the loop, I just haven’t shown them to you because they are irrelevant to the issue at hand.

I think it is unclear what my problem is:

  1. Not enough ram on the atmega328 to store large byte arrays-solution-exteneded ram?-atmega1284?

  2. Not able to recieve more than 64 bytes.

You have failed to understand that loop keeps on being called over and over therefore it is pointless to do what you have done which is what on serial for somthing when wait on serial for somthing else take a look at Finite State Machines (in the playground).

As for that bit of code working thats just good luck and it won't work every time you go round loop. As I said serial is slow, very slow and you can't just assume that the whole message is there.

Mark

Is this better:

while (Serial.available()==6)    //wait for buffer to recieve all of incoming string
   for(int i=0;i<6;i++){
     incomingString[i] = Serial.read();
   }

I fail to understand what you are saying due to lack of punctuation/incoherent sentence. This sentence does not make sense:

You have failed to understand that loop keeps on being called over and over therefore it is pointless to do what you have done which is what on serial for somthing when wait on serial for somthing else take a look at Finite State Machines (in the playground).

kurtruk: In response to Robin2: I do have lots of functions outside of the loop, I just haven't shown them to you because they are irrelevant to the issue at hand.

In Reply #3 you posted the entire code and it doesn't have lots of functions outside loop.

In any case my comment was aimed at what is already inside loop() and which should be moved out of it.

In my opinion the purpose of loop() is to keep things moving, not to figure things out.

You could write code that works perfectly without taking any notice of my suggestions. But if the code is not easy to figure out you will find it hard to get useful advice from other people who are not familiar with it and have only 5 or 10 minutes to devote to your problem.

And, after you have not seen your own code for 6 months, will you understand it after a single read-through?

I don't expect anyone to do things my way - you should do things in the way that is most effective for you.

And to deal with your two questions in Reply #8

You need to be careful how you allocate memory on a small device like an Arduino - it is easy to exceed the available SRAM

And it should be utterly trivial to receive a few hundred bytes - assuming you have the space in which to store them.

...R

I have a class that I did not post. I suppose It would be wise to make another class with functions.

kurtruk:
I have a class that I did not post. I suppose It would be wise to make another class with functions.

Maybe not wise. I would strongly suggest backing away for a day and the rereading Robin2’s responses. You seem to believe SRAM is your issue, but it is not; rather it is code that cannot process fast enough to manage the data flow correctly. This is not acceptable unless you implement serial handshaking… Which should not be required.

Is this better:
Code:
while (Serial.available()==6) //wait for buffer to recieve all of incoming string
for(int i=0;i<6;i++){
incomingString = Serial.read();

  • }*
    [/quote]
    Suggestion: Take a long look at how GPS serial libraries buffer and parse tokens … You will learn lots.
    Ray

Hi Ray,

Just reading this thread as a bystander now, but very interested to read your comment about GPS libraries and token parsing.

I've written some code to communicate with a Serial -> Wifi module and I've coming up against the same sort of issues. Namely that the wifi module sends a lot of data to the Arduino, while the Arduino is actively creating and sending unrelated data to the module.

I know I need to re-write my code to use event driven serial handling and message parsing (in this case they are HTTP Requests) But, writing it from scratch is going to quite time consuming.

Can you suggest a good GPS library to look at and possible re-purpose the data handling?

Thanks

Roger

rogerClark: Hi Ray,

Just reading this thread as a bystander now, but very interested to read your comment about GPS libraries and token parsing. ... Can you suggest a good GPS library to look at and possible re-purpose the data handling?

Thanks Roger

"Good" is such a loose term... Some would ssy that any library that works correctly is "good 'nuff!"

I think maybe the Adafruit lib because it is so well documented and includes doubles buffering, too.

Ray

Added: https://github.com/adafruit/Adafruit-GPS-Library

Thanks