Long serial communication

Hi guys,
I need to send up to 70 bytes (and probably more) trough serial communication. I just learn that the serial buffer is limited to 64 bytes and that would explain why when I was sending only 62 bytes everything was working fine.

So how can I send more than 64 bytes trough serial ?

Here is my transmitter code:

for(int i = 0; i < 71; i++){
        Serial1.write (var[i]);    // send the number
    }

And here is my receiver code:

   if(Serial1.available() > 60){
         for(int i=0; i < 71; i++)  {
         var[i] = Serial1.read();
         Serial.println(var[i]);
}

Send a small number of bytes, then wait until serial is ready before sending more.

Something like this but also make sure i doesn't get out the array range.

   if(Serial1.available() ){
          var[i] = Serial1.read();
           i++;
  }
       
}

The code in the examples in Serial Input Basics will receive as much data as you have space for in the Arduino SRAM. You just need to change the line
const byte numChars = 32;
to something larger than the maximum number of bytes you will receive.

The size of the Arduino serial buffer is not a limiting factor because the code continuously empties the buffer.

...R

Both the incoming and outgoing serial buffers are limited to 64 characters. For outgoing data, though, that doesn't matter as the Arduino will simply block until there is room in the outgoing buffer for the data that it trying to send.

You can send hundreds of characters without worrying about the buffer size.

You can not do nothing until all the data has arrived, if you need to wait for more than 64 characters. You must read and save serial data as fast as it arrives. You use the saved data when the end of packet marker arrives.

That means, of course, that you must send an end of packet marker.

Why are you sending binary data? Are you, unreasonably, planning that no serial data will ever be lost?

   if ( Serial1.available() & ( i < ( arraySize - 1 ))
  {
          var[i] = Serial1.read();
           i++;
  }
  var[ i ] = 0;

tchinou1:
Hi guys,
I need to send up to 70 bytes (and probably more) trough serial communication. I just learn that the serial buffer is limited to 64 bytes and that would explain why when I was sending only 62 bytes everything was working fine.

So how can I send more than 64 bytes trough serial ?

Try sending 200 bytes, a repeating pattern will do, with the basic code you have.

I got a perfect working lib to send long arrays between 2 Arduinos - maybe this might help you...?

http://forum.arduino.cc/index.php?topic=346011.msg2386207#msg2386207
(look at the attachement !)

Watcher and GoForSmoke, I don't really understand how the "i" variable is rest if it's not in a for loop. My each variable of the array var[] are assign to a specific value, so I need that every time I receive a data the first byte to arrive is store is var[0].

Robin2, in the link that you send me, the serial.read() is store in a char array. I am planing on receive only int, so could I use a int array ?

Does the buffer is emptied every time the void loop run ? Is there a way to empty it manually ?

Sorry I'm kind of confused in all of this !

Thanks :slight_smile:

tchinou1:
Watcher and GoForSmoke, I don't really understand how the "i" variable is rest if it's not in a for loop.

Is rest?

There is no for loop required. Only the variable to use as index.
Without the for loop, there must be code to change the value in proper place, like the 4th line below.

   if ( Serial1.available() & ( i < ( arraySize - 1 ))
  {
          var[i] = Serial1.read();
           i++;
  }
  var[ i ] = 0;

What the snippet does not show is making the variable and setting it to zero before a new text string is received.

There's so much else that can be done, I don't have time or room to show and you are not ready.

This below especially can't work even if you add the missing }

   if(Serial1.available() > 60){
         for(int i=0; i < 71; i++)  {
         var[i] = Serial1.read();
         Serial.println(var[i]);
}

Reason is that serial data is 100's of times slower than Arduino, the code checks for 1 or more available characters in serial and then proceeds to collect 71 faster than another can arrive.
If you read serial with no data available, Serial.read() will return -1. You would get many -1 in var[].

If you are reading Robin2's thread then keep reading but also run any examples and take the time to learn what every bit is.

Just BTW, do you know where the Arduino Reference pages are? Most of these answers and many more are in them.

This page especially should answer many questions for you:
https://www.arduino.cc/en/Tutorial/Foundations

Reset** not rest ! sorry !

The code has to reset the index, and just where depends on the rest of the code.
I would do it after the string in var[] is filled and used, i=0 and then var*=0 to empty the string.*
*for-next does the count for you but as long as the loop runs, other tasks (pin watchers, time watchers, etc) get no attention. If the for-next loop is quick and doesn't run too many times it's fine but note that you still get the job done using loop() and your own indexes, and that lets other tasks get a chance to catch some action. *
What are you going to do with the serial data in var[] once you have it?

   if(Serial1.available() > 60){
         for(int i=0; i < 71; i++)  {
         var[i] = Serial1.read();
         Serial.println(var[i]);
}

That's just not the way to do it. What if you wanted to receive 200 bytes?

if(Serial1.available() > 60){
         for(int i=0; i < 71; i++)  {

Once you have 61 bytes, read all 70 of them. I bet the last 9 are 0xFF or -1, right?

Nick Gammon, it did some testing and changing this value didn't really changed anyway so I left it to what it was previously... :roll_eyes:

GoForSmoke, I tested your code, but I with the: & i < (arraySize -1) I receive only 18 variables. If I remove this parameter, I get pretty close to my goal. The first time I receive 70 value, but the second only 68 and the third 66. After that it seems to stay to 66 or 65
The index doesn't seems to reset in my code, in a test code it was working, but not in the main one. However, with the reset line, it's okay.

The variables received, are use for color and other parameter in a Vu meter,

PS: I just noticed that when I get rid of the fastled.show(), there is no more problem. I am using the fastled library.

Here is the code that is working the best for me right now:

 if ( Serial1.available() )
  {
          var[a] = Serial1.read();
          Serial.print(a);
          Serial.print(": ");
          Serial.println(var[a]);
           a++;
           if(a == 71) { a = 0; }
           
  }

 RL[0] = var[0];
 RL[1] = var[6];
 RL[2] = var[12];
 RL[3] = var[18];
 RL[4] = var[24];
 RL[5] = var[30];
 RL[6] = var[36];
 RL[7] = var[42];
 RL[8] = var[48];
 RL[9] = var[54];

 RR[0] = var[3];
 RR[1] = var[9];
 RR[2] = var[15];
 RR[3] = var[21];
 RR[4] = var[27];
 RR[5] = var[33];
 RR[6] = var[39];
 RR[7] = var[45];
 RR[8] = var[51];
 RR[9] = var[57];

 GL[0] = var[1];
 GL[1] = var[7];
 GL[2] = var[13];
 GL[3] = var[19];
 GL[4] = var[25];
 GL[5] = var[31];
 GL[6] = var[37];
 GL[7] = var[43];
 GL[8] = var[49];
 GL[9] = var[55];

 GR[0] = var[4];
 GR[1] = var[10];
 GR[2] = var[16];
 GR[3] = var[22];
 GR[4] = var[28];
 GR[5] = var[34];
 GR[6] = var[40];
 GR[7] = var[46];
 GR[8] = var[52];
 GR[9] = var[58];

 BL[0] = var[2];
 BL[1] = var[8];
 BL[2] = var[14];
 BL[3] = var[20];
 BL[4] = var[26];
 BL[5] = var[32];
 BL[6] = var[38];
 BL[7] = var[44];
 BL[8] = var[50];
 BL[9] = var[56];

 BR[0] = var[5];
 BR[1] = var[11];
 BR[2] = var[17];
 BR[3] = var[23];
 BR[4] = var[29];
 BR[5] = var[35];
 BR[6] = var[41];
 BR[7] = var[47];
 BR[8] = var[53];
 BR[9] = var[59];

ActiveMode =  var[60];
power = var[61];
rotation = var[62];
rotation_order = var[63];
rotation_time = var[64];
random_mode = var[65];
mode_time = var[66];
mood = var[67];
mood_type = var[68];
//sensitivity = var[69];

Thanks !

Reason is that serial data is 100's of times slower than Arduino, the code checks for 1 or more available characters in serial and then proceeds to collect 71 faster than another can arrive.

if(Serial1.available() > 60){
for(int i=0; i < 71; i++) {
var = Serial1.read();
_ Serial.println(var*);_
_
}*_
If the above sentence describes the attached code I can see the OP confusion.
Serial.available returns the number of characters currently collected in buffer.
It does not collect them nor waits for them.
In this code the count returned is evaluated and if it does not match the required condition (>60) the for loop is skipped.
My objection to this code is that there is an assumption that the buffer will be filed (>60) when accessed , otherwise it will have to be run again.
May as well put delay before executing the snippet- the serial buffer will still get filled or overrun either way. .
*Not very effective IMHO. I do prefer " waiting for Serial.available" using do / while and also process each data (character) as it arrives. *
*Of course if it is part of the loop() nothing to worry about. *

tchinou1:
GoForSmoke, I tested your code, but I with the: & i < (arraySize -1) I receive only 18 variables. If I remove this parameter, I get pretty close to my goal. The first time I receive 70 value, but the second only 68 and the third 66. After that it seems to stay to 66 or 65
The index doesn't seems to reset in my code, in a test code it was working, but not in the main one. However, with the reset line, it's okay.

You are creating problems by not posting the whole code after changing it.
I don't understand HOW you get the above much less WHY.

When I made the example out of your original snippet and then tried to show how the index might be maintained I used a variable name arraySize for what it is for ---

in your case the variable, whatever you call it, should be 71 elements to include a terminating zero because I made the Possibly Stupid Assumption (MY BAD) that the serial data is for a text string and text strings must always end with 0. So if you would collect 70 chars of text, you need a 71 char array.

Did I give you a link to the Arduino Fundamentals page?

The variables received, are use for color and other parameter in a Vu meter,

PS: I just noticed that when I get rid of the fastled.show(), there is no more problem. I am using the fastled library.

I can't begin to guess why without seeing the whole code that will tell me the full story.

Understand please that when looking for bugs, how the person who wrote them understands the code may well be why the bugs are there. The full code and wiring tell the true story including errors the coder has not (yet) detected.

So please! Full code! We can help you faster!

Here is the code that is working the best for me right now:

 if ( Serial1.available() )

{
          var[a] = Serial1.read();
          Serial.print(a);
          Serial.print(": ");
          Serial.println(var[a]);
          a++;
          if(a == 71) { a = 0; }
         
  }

RL[0] = var[0];
RL[1] = var[6];
RL[2] = var[12];
RL[3] = var[18];
RL[4] = var[24];
RL[5] = var[30];
RL[6] = var[36];
RL[7] = var[42];
RL[8] = var[48];
RL[9] = var[54];

RR[0] = var[3];
RR[1] = var[9];
RR[2] = var[15];
RR[3] = var[21];
RR[4] = var[27];
RR[5] = var[33];
RR[6] = var[39];
RR[7] = var[45];
RR[8] = var[51];
RR[9] = var[57];

GL[0] = var[1];
GL[1] = var[7];
GL[2] = var[13];
GL[3] = var[19];
GL[4] = var[25];
GL[5] = var[31];
GL[6] = var[37];
GL[7] = var[43];
GL[8] = var[49];
GL[9] = var[55];

GR[0] = var[4];
GR[1] = var[10];
GR[2] = var[16];
GR[3] = var[22];
GR[4] = var[28];
GR[5] = var[34];
GR[6] = var[40];
GR[7] = var[46];
GR[8] = var[52];
GR[9] = var[58];

BL[0] = var[2];
BL[1] = var[8];
BL[2] = var[14];
BL[3] = var[20];
BL[4] = var[26];
BL[5] = var[32];
BL[6] = var[38];
BL[7] = var[44];
BL[8] = var[50];
BL[9] = var[56];

BR[0] = var[5];
BR[1] = var[11];
BR[2] = var[17];
BR[3] = var[23];
BR[4] = var[29];
BR[5] = var[35];
BR[6] = var[41];
BR[7] = var[47];
BR[8] = var[53];
BR[9] = var[59];

ActiveMode =  var[60];
power = var[61];
rotation = var[62];
rotation_order = var[63];
rotation_time = var[64];
random_mode = var[65];
mode_time = var[66];
mood = var[67];
mood_type = var[68];
//sensitivity = var[69];




Thanks !

And that is all generated by code you wrote and can change, not just data straight from devices that you can't change?

Because here is one place where you could get errors; serial data has no guarantees. It's generally reliable but good code never treats it so. -----

** That's why we often send data as text. Corrupted text is easy to spot. I think you do this?
** Another thing to do is add identifiers that the receiving code looks for and acts upon. One or two letters is enough especially when the data is all numbers, doesn't have to be for every variable but the more you use, the more sure you can be that your serial data is good or not.
** Another thing to do, industry standard when binary data is used but good with text is to use CRC (cyclic redundancy check) as an error check. CRC can be simple or not so simply done, people have different ways to use the CRC principle. Easy way is to use a byte (unsigned char) that starts as zero and then add every byte in the message, always making a byte result to add the next until the end. That byte must match the CRC byte at the end of the message (or part checked) and if not matched is error.

I interpreted text from electric meters not so long ago and have done for gas pumps and cash registers and industrial machines. They use all the above tricks to make sure their code knows good data from bad.

All because serial transmission has no guarantees (thanks for that term... AWOL or Grumpy Mike?).

So perhaps you may want to change that 70 character message a little?

And if you like, I can show you how to read such a string without buffering any of it.

tchinou1:
Robin2, in the link that you send me, the serial.read() is store in a char array. I am planing on receive only int, so could I use a int array ?

I just noticed this now.

You should first receive the data as chars or bytes because that is what Serial sends. After you have the data you can convert pairs of bytes to ints.

You can change my char array to a byte array and make equivalent changes elsewhere. But don't change it to an int array.

HOWEVER if, by an int you just mean (for example) "12345" then you will be receiving characters and you can use my code unchanged.

...R

Here is the complete code:

#include "FastLED.h"
#include <EEPROM.h>

#define NUM_LEDS 10
#define DATA_PIN 4
CRGB left[NUM_LEDS];
CRGB right[NUM_LEDS];


int i = 0;
int var[75];
int RL[10];
int RR[10];
int GL[10];
int GR[10];
int BL[10];
int BR[10];

int R[10];
int G[10];
int B[10];

//new mode
const int sensitivityPin = A2;  // Analog input pin that the potentiometer is attached to
const int audioPin = A0;        // Analog input pin that the audio channel is attached to
const int ledCount = 10;        // The number of LEDs in the bar graph
const int numReadings = 20;     // Number of samples to keep track of (for smoothing)
const int buttonPin = 13;
int counter=0;
int audioValue;                 // Analog value read from audio channel
int maxAudioValue = 0;          // Maximum analog value read from audio channel
int sensitivityValue;           // Analog value read from potentiometer
int ledLevel;// Value to map to the output LEDs
int ledLeveltot;

int readings[numReadings];      // the readings from the analog input
int index = 0;                  // the index of the current reading
int total = 0;                  // the running total
int average = 0;                // the average
int power;
int ActiveMode;


//MSGEQ7
const int ledCount1 = 8;
const int ledCount2 = 8;
const int ledCount3 = 7;
const int ledCount4 = 7;
const int ledCount5 = 7;
const int ledCount6 = 7;
const int ledCount7 = 7;

int ledLevel1;
int ledLevel2;
int ledLevel3;
int ledLevel4;
int ledLevel5;
int ledLevel6;
int ledLevel7;

int analogPin = 0; // read from multiplexer using analog input 0
int fqc[7]; // to hold a2d values

int strobePin = 2; // MSGEQ7 STROBE 4
int resetPin = 3; // MSGEQ7 RESET 7
int filterValue = 0;

//dev var
int rotation;
int rotation_time;
int rotation_order;
unsigned long int nextrotation = 0;
int PreviousR;
int PreviousG;
int PreviousB;

int random_mode;
unsigned long int nextmode = 0;
int mode_time;

int mood = 1;
int mood_type = 1;
int v = 0;

int a = 0;

void setup()
{

  FastLED.addLeds<NEOPIXEL, 4>(left, NUM_LEDS);
  FastLED.addLeds<NEOPIXEL, 5>(right, NUM_LEDS);
  Serial.begin(9600);
  Serial1.begin(9600);



  RL[0] = EEPROM.read(0);
  //Same thing for the rest of the variables
}

void loop()
{

 if ( Serial1.available() )
  {
          var[a] = Serial1.read();
         // EEPROM.write(i, var[i]);
          Serial.print(a);
          Serial.print(": ");
          Serial.println(var[a]);
           a++;
           if(a == 71) { a = 0; }
           
  }

 RL[0] = var[0];
 RL[1] = var[6];
 RL[2] = var[12];
 RL[3] = var[18];
 RL[4] = var[24];
 RL[5] = var[30];
 RL[6] = var[36];
 RL[7] = var[42];
 RL[8] = var[48];
 RL[9] = var[54];

 RR[0] = var[3];
 RR[1] = var[9];
 RR[2] = var[15];
 RR[3] = var[21];
 RR[4] = var[27];
 RR[5] = var[33];
 RR[6] = var[39];
 RR[7] = var[45];
 RR[8] = var[51];
 RR[9] = var[57];

 GL[0] = var[1];
 GL[1] = var[7];
 GL[2] = var[13];
 GL[3] = var[19];
 GL[4] = var[25];
 GL[5] = var[31];
 GL[6] = var[37];
 GL[7] = var[43];
 GL[8] = var[49];
 GL[9] = var[55];

 GR[0] = var[4];
 GR[1] = var[10];
 GR[2] = var[16];
 GR[3] = var[22];
 GR[4] = var[28];
 GR[5] = var[34];
 GR[6] = var[40];
 GR[7] = var[46];
 GR[8] = var[52];
 GR[9] = var[58];

 BL[0] = var[2];
 BL[1] = var[8];
 BL[2] = var[14];
 BL[3] = var[20];
 BL[4] = var[26];
 BL[5] = var[32];
 BL[6] = var[38];
 BL[7] = var[44];
 BL[8] = var[50];
 BL[9] = var[56];

 BR[0] = var[5];
 BR[1] = var[11];
 BR[2] = var[17];
 BR[3] = var[23];
 BR[4] = var[29];
 BR[5] = var[35];
 BR[6] = var[41];
 BR[7] = var[47];
 BR[8] = var[53];
 BR[9] = var[59];

ActiveMode =  var[60];
power = var[61];
rotation = var[62];
rotation_order = var[63];
rotation_time = var[64];
random_mode = var[65];
mode_time = var[66];
mood = var[67];
mood_type = var[68];
//sensitivity = var[69];



if(power == 1)
{

//------ Mode 1 ------//
if(ActiveMode == 1)
{
  int ledPins[]  = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
  //int ledPins[]  = {9, 8, 7, 6, 5, 4, 3, 2, 1, 0};

    digitalWrite(resetPin, HIGH);
    digitalWrite(resetPin, LOW);

    for (int i = 0; i < 7; i++)
    {
      digitalWrite(strobePin, LOW);
      delayMicroseconds(30); // to allow the output to settle
      fqc[i] = analogRead(analogPin);
      digitalWrite(strobePin, HIGH);
    }

    ledLevel1 = map(fqc[0], 30, 1023, 0, 4);
    ledLevel2 = map(fqc[1], 30, 1023, 0, 4);
    ledLevel3 = map(fqc[2], 30, 1023, 0, 3);
    ledLevel4 = map(fqc[3], 30, 1023, 0, 3);
    ledLevel5 = map(fqc[4], 30, 1023, 0, 3);
    ledLevel6 = map(fqc[5], 30, 1023, 0, 3);
    ledLevel7 = map(fqc[6], 30, 1023, 0, 3);

    if (ledLevel1 > ledCount1)
    {
      ledLevel1 = ledCount1;
    }
    if (ledLevel2 > ledCount2)
    {
      ledLevel2 = ledCount2;
    }
    if (ledLevel3 > ledCount3)
    {
      ledLevel3 = ledCount3;
    }
    if (ledLevel4 > ledCount4)
    {
      ledLevel4 = ledCount4;
    }
    if (ledLevel5 > ledCount5)
    {
      ledLevel5 = ledCount5;
    }
    if (ledLevel6 > ledCount6)
    {
      ledLevel6 = ledCount6;
    }
    if (ledLevel7 > ledCount7)
    {
      ledLevel7 = ledCount7;
    }

    ledLevel = (ledLevel1 + ledLevel2 + ledLevel3 + ledLevel4 + ledLevel5 + ledLevel6 + ledLevel7);

  for (int thisLed = 0; thisLed < ledCount; thisLed++)
  {
    pinMode(ledPins[thisLed], OUTPUT);
    if (thisLed < ledLevel)
    {
      //  digitalWrite(ledPins[thisLed], HIGH);
 left[ledPins[thisLed]].setRGB(GL[ledPins[thisLed]],RL[ledPins[thisLed]],BL[ledPins[thisLed]]);
 right[ledPins[thisLed]].setRGB(GR[ledPins[thisLed]],RR[ledPins[thisLed]],BR[ledPins[thisLed]]);
      FastLED.show();
      delay (1);
    }
    else
    {
  left[ledPins[thisLed]].setRGB(0,0,0);
   right[ledPins[thisLed]].setRGB(0,0,0);
   FastLED.show();
      //ShiftPWM.SetRGB(ledPins[thisLed],0,0,0);
    }
  }
  delay(1);        // delay in between reads for stability
}




}

}

The code in Reply #18 (as well as being poorly formatted) seems to read one character then do a whole lot of stuff, then read another character.

It would not be surprising if the input buffer overflows before you get a chance to read all the data.

The examples in Serial Input Basics are designed to read all the data before trying to use any of it. That way there is no possibility of the buffer overflowing.

The code in loop() should probably be something like

void loop() {
   receiveData();
   updateLEDs();
}

...R