HELP -> 64 bytes buffer question

Hello everybody , i have a question about the buffer , i found information on the Internet

"How does the Arduino handle serial buffer overflow? Does it throw away the newest incoming data or the oldest? How many bytes can the buffer hold? "

“it stops attempting to insert received data into the queue when it is full” .

So if i send a 264 bytes to RX serial port , only the first 64 bytes stays on the buffer…

the other 200 bytes are not there…

how do you handle this limit so it can read all 264 bytes , when you can not
divide the incoming data ( only sends 264 bytes at once…to RX port ) ?

ptmir: how do you handle this limit so it can read all 264 bytes , when you can not divide the incoming data ( only sends 264 bytes at once..to RX port ) ?

Compared to the speed that the processor can process data, serial data arrives at a snail's pace. In between two of those 264 bytes arriving there is time for the Ardunio to recieve the byte, store it, analyze it, go make a sandwich, sit down and watch some TV, go out for a quick run, take a long hot bath, have a quick nap, wake up and brush the teeth, and finally read half of a novel before the next byte arrives.

So the trick is to read the bytes that are coming in at least as fast as they are arriving. Then, the buffer never fills up. But if the rest of your code blocks operations, and you take too long to get back and read the next byte, then the buffer will fill up and you'll lose data.

ptmir: Hello everybody , i have a question about the buffer , i found information on the Internet So if i send a 264 bytes to RX serial port , only the first 64 bytes stays on the buffer...

Max. 63 bytes stay in the buffer.

Though the size of the buffer is 64, it can keep 63 bytes only at the same time, due to its internal organization as a FIFO ringbuffer.

ptmir: how do you handle this limit so it can read all 264 bytes , when you can not divide the incoming data ( only sends 264 bytes at once..to RX port ) ?

"Serial" is one of the slowest things you can run on an Arduino. You will never receive "264 bytes at once". Even with 115200 baud you receive only 11520 bytes per second, which means that every 86 microseconds one byte arrives. So it goes like this: - byte #1 arrives - 86 microseconds ==> nothing happens - byte #2 arrives - 86 microseconds ==> nothing happens - byte #3 arrives - 86 microseconds ==> nothing happens - byte #4 arrives - 86 microseconds ==> nothing happens - byte #5 arrives - 86 microseconds ==> nothing happens

You can always use the "nothing happens" time to read the bytes from the Serial input buffer and save them in another buffer of the size you actually want.

With clever programming, your Serial input buffer will hardly contain more than 1 byte at any time, even when receiving all the time. So most likely, clever Arduino programs will never use more than 1 byte of the Serial input buffer, although up to max. 63 bytes could be in the Serial input buffer with some less clever programming.

Only dumb programs using dumb commands like "delay()" will lead to an overflow of the serial input puffer. So just avoid using dumb commands like "delay()" and all the "Serial.parseBullshit()" or other silly commands, and you get what you want: A clever program that handles all Serial input quickly.

well .. suppose i send the request and the incoming 264 bytes packet is coming ...

you say if i do a Serial.read(); i make place for another byte on buffer ?

so as the incoming 264 bytes are arriving i just have to keep doing , Serial.read(); and i will eventually read all of them ? is that so ?

ptmir: well .. suppose i send the request and the incoming 264 bytes packet is coming ...

you say if i do a Serial.read(); i make place for another byte on buffer ?

so as the incoming 264 bytes are arriving i just have to keep doing , Serial.read(); and i will eventually read all of them ? is that so ?

As long as you don't wait so long between calls to Serial.read that more than 63 of those bytes show up.

ptmir: you say if i do a Serial.read(); i make place for another byte on buffer ?

Exactly. With "Serial.read()" you read the received character/byte from the buffer, so it's no longer contained in the input buffer.

Then it's just a question of what you do with the packet data you're reading. One simple solution would be to read it into a buffer of your own - an array of 264 chars. RAM is in short supply on most arduinos though, so if possible, it would likely be preferable to parse the data as you receive it.

Thats mad :) , so i´m thinking doing this

as this is a Sysex message it begins with 0xF0 and ends with a 0xF7 ....

algorithm i will do :

i will catch the 0xF0 and store on an myArray[0] and Serial.read() to fill until myArray[63] now that i know what is on the :

myArray[61] , myArray[62] , myArray[63]

i will , Serial.flush(); ( clear all buffer )

and request again for 264 bytes

then i will catch doing Serial.read()´s the incoming myArray[61] , myArray[62] , myArray[63]

now i will keep Serial.read() to myArray[64] .... my array[127]

and so on .... until i get all the 264 bytes

what you think ?

No, you misunderstand. First off, that's not what Serial.flush() does. It has nothing to do with the incoming buffer. flush() tells the processor to stop what it is doing until it has finished SENDING data. It has nothing to do with receiving it.

WHen you call Serial.read, it removes the byte from the Serial buffer. You don't need any other code to clear the buffer out. The act of reading the bytes removes them from the Serial buffer.

I think you misunderstand the Serial buffer. So let's say you have an internal buffer called myArray that you want the message in.

So you request your message and the 264 byte transmission starts.

The first byte arrives, now the serial buffer has one byte in it. Read it (which removes it from the buffer) and put it in myArray[0]. Now the Serial buffer has 0 bytes in it. Now there will be some time waiting for the next byte. When it arrives, it goes in the Serial buffer, now the serial buffer has one byte in it. Read it and put it in myArray[1], now the Serial buffer has zero bytes in it. And now we wait for the next byte. Repeat ad infinitum. At no point will there be more than one byte sitting in that Serial buffer. You're not even close to the 63 byte limit.

The only reason you would need to worry about the 63 byte limit would be if say the 264 byte message is coming and your code reads one byte and then calls delay(5000) so that the processor is asleep for 5 seconds. Then during that 5 seconds of not reading the buffer could get full. As long as you don't do that, you don't have anything to worry about.

ok , know i see farther :)

i will try this

            requesSysex();

waitForSerial();
            
            //delayMicroseconds(258);//86 microseconds  for 115200 bauds ,  entao 258 microseconds para 31250 bauds

            
byte in = 0;
byte one = 0;
int index=0;
    
  if (Serial.available() > 0)
  {
    one = Serial.read();
    if (one == 0xF0)
    {     datax[0]= 0xF0;
    index++;
      delayMicroseconds(258);
      in = Serial.read();     
      while (in != 0xF7)
      {
      
        datax[index] = Serial.read();
        delayMicroseconds(258);
        index++;
      }      
      datax[263]=0xf7;
      doit=1;
      
    }
  }





void waitForSerial()
{
  while (Serial.available() == 0)
  {
  }
}





void requesSysex() {
  
  
          
            
            
            
            
            byte casioRe[]={0xF0, 0x44, 0x00, 0x00, 0x70, 0x10, 0x60 , 0x70, 0x31 , 0xF7} ;
            
            Serial.write(casioRe, sizeof(casioRe)); // request patch  from casio cz-1000
            
            
  
                   }

The examples in serial input basics are simple reliable ways to receive data without the buffer overflowing.

...R

THANKS every one , i steel working on this , hope i can resolve it , but i must THANK to all of YOU !

ptmir: requesSysex(); waitForSerial();

"Waiting" in an Arduino program is always bad.

If I'd to programm something like that the programming logic would read like:

 if (itsTimeToSendaRequest) requestSysex();
 if (serialInputIsReady())
 {
    // Do Something here
 }

i with this code :

           byte casioRe[]={0xF0, 0x44, 0x00, 0x00, 0x70, 0x10, 0x60 , 0x70, 0x31} ;
            
            Serial.write(casioRe, sizeof(casioRe)); // request  patch casio cz-1000
            Serial.flush();
            delay(100);
  

byte one = 0;
int index=0;
byte datax [264];           
int  doit=0;

    
  if (Serial.available() > 0)
  {
    one = Serial.read();
    if (one == 0xF0)
    {     datax[0]= 0xF0;
    index++;
      
          
      while (Serial.available()>0)
      {
        datax[index] = Serial.read();
       
        index++;
      }      
      datax[263]=0xf7;
      doit=1;    // flag to  save  file on SD card      
    }
  }

i´m getting this strange output :

so after 62 byte i can not get more....

You've ignored the advice above: Wikipedia tells me that MIDI runs at 31250 baud or ~3125 characters a second. Which means that delay(100) you're using is more than enough to overflow the serial buffer.

i did not have initially a delay there but without it i did not get any data .. so i implement a delay(100) i will try now a delayMicroseconds(258) instead , lets see :)

The real problem is that once you see the start byte, your code executes the while loop assuming that all of your data will be there. It won't - serial data transmission is slow as mentioned above. You need to structure your code so that it can read the data as and when it arrives.

When you do post your code again, better to post (or attach) the whole thing.

here the code not working …

//========================================================================================
//==  VOID     SYSEX receive from Synth !!
//========================================================================================


void receiveSysex() {
  

  
  Serial.end() ;          //desliga o serial output
  delay(200);
  Serial.begin(31250);    // para trabalhar no MIDI
  Serial.flush();         

  
            digitalWrite(8, LOW);
            delay(50);
            digitalWrite(8, HIGH);
  

//F0 Start of SysX
//44 00 00 Casio Manufacturer ID
//70 MIDI Channel 1 (70..7F = channel 1..16)
//must be the same channel als the CZ-5000 MIDI channel number
//10 Command "Dump Request"
//xx [00..1F] = PRESET Timbre numbers A1-D8
//[20..3F] = MEMORY Timbre numbers A1-D8
//[40..5F] = CARTRIDGE Timbres A1-D8
//[60] = Timbre in the Edit Buffer
//70 31 End of request (so not the usual F7 sysx terminator)

              //F0 44 00 00 70 10 60 70 31 F7 request patch from CURRENT SOUND area
              //byte casioRe[]={0xF0, 0x44, 0x00, 0x00, 0x70, 0x10, 0x60 , 0x70, 0x31 , 0xF7} ;
            byte casioRe[]={0xF0, 0x44, 0x00, 0x00, 0x70, 0x10, 0x60 , 0x70, 0x31} ;
            
          
            Serial.write(casioRe, sizeof(casioRe)); // request CURRENT SOUND casio cz-1000
            Serial.flush();
            delayMicroseconds(2200);
  

byte one = 0;
int index=0;
byte datax [264];           

 int  doit=0;

    
  if (Serial.available() > 0)
  {
    one = Serial.read();
    if (one == 0xF0)
    {     datax[0]= 0xF0;
    index++;
      
          
      while (Serial.available()>0)
      {
        datax[index] = Serial.read();
        
        index++;
      }      
      datax[263]=0xf7;
      doit=1;
      
    }
  }

            
            
            
            
            
           

 
    delay(2000);
  

    if (doit==1)
    {
    
              
             //------------------------------------------
             //GRAVAR em FICHEIRO
             //nome do file a gravar é o maximo + 1
             //----------------------------------------
             
             
char sysexName [3];
File sysexFile;
int  nmax;

nmax=maximo+1;
sprintf(sysexName,"%i",nmax);


//Serial.print(maximo,DEC); // DEBUGUING MAXIMO FILES .. OK FOR F16 FORMAT  SD CARD ...


 sysexFile.setTimeout(0);    // sets the maximum milliseconds to wait for serial data


  sysexFile = SD.open(sysexName, FILE_WRITE);    //This creates a file !!! cool :)  
  
  delay(500);
 
  
  // if the file is available, write to it:
  if (sysexFile) {
 
                for (int n=0; n< sizeof(datax) ; n++)
                {  sysexFile.write(datax [n]);   }

                sysexFile.close(); 
                maximo=maximo+1; // entao se foi gravado um novo file o maximo incrementa 1
               
                putINITXTinc() ;
            
                }




    
    
   delay(1000);  // para que evite o user de estar a pressionar este botao sem parar.... ,like crazy  lol
           
            digitalWrite(8, LOW);
            delay(50);
            digitalWrite(8, HIGH);
           
   
  }

            



}
//========================================================================================
//==  END  !!!   VOID     SYSEX receive from Synth !!
//========================================================================================

ptmir: so after 62 byte i can not get more....

The function call "delay(100);" means something like:

Prevent my program to execute for a very, very long time. 
The time shall be 100 milliseconds = 1.6 million controller cycles,
Do nothing until the Serial input buffer is flooded several times.
If ready with blocking everything, then continue.

Blocking all the program execution for millions of controller cycles until the input buffer is flooded several time is NOT a good programming logic.

yes but if i do not use this delay .

Serial.write(casioRe, sizeof(casioRe)); // request CURRENT SOUND casio cz-1000 Serial.flush(); delayMicroseconds(2350); // this one !!!

i do not get any data on my Serial.read();

and the program does not save any file ....