Go Down

Topic: What is the Arduino Uno Serial buffer size? (Read 20407 times) previous topic - next topic

Blaylock1988


A suitable method is described here:


I tried using your method with the processIncomingByte stuff, but unfortunately I still only received 67 bytes.

Here is my current code:
Code: [Select]
/* 
This program is used for an Arduino to request and retrieve
data from an 07 Buell XB9SX Lightning ECM DDFI-2,
EEPROM Version: BUEIB

Currently the Arduino successfully sends a request code and
receives only 67 bytes of the run-time response, and displays
on the LCD screen and via serial monitor.

Created by Michael Blaylock
*/

// include the library code:
#include <LiquidCrystal.h>
#include <SoftwareSerial.h>

//new serial pins for ECM, 0 and 1 caused interferance from PC
#define rxPin 2
#define txPin 3

// set up a new serial port
SoftwareSerial mySerial =  SoftwareSerial(rxPin, txPin);

// initialize the library with the numbers of the interface pins
LiquidCrystal lcd(7, 8, 9, 10, 11, 12);
byte inArray[9]; //request code for real-time data
byte outArray[107]; //real-time data series from ECM
byte long Value;
int First = 0;
int q;

void setup(){

  lcd.begin(16, 2);  // set up the LCD's number of columns and rows:


  mySerial.begin(9600);  // baud rate for the ECM is 9600
  Serial.begin(9600);
  q=0;
  inArray[0]=0x01; //SOH
  inArray[1]=0x00; //Emittend
  inArray[2]=0x42; //Recipient
  inArray[3]=0x02; //Data Size
  inArray[4]=0xFF; //EOH
  inArray[5]=0x02; //SOT
  inArray[6]=0x43; //Data 1 //0x56 = Get version, 0x43 = Get runttime data
  inArray[7]=0x03; //EOT
  inArray[8]=0xFD; //Checksum

  lcd.setCursor(0,0);
  //Stream request data series to ECM as HEX
  for (int i = 0; i<9; i+=1){
    mySerial.write(inArray[i]);
    lcd.print(inArray[i],HEX); //print sent HEX code
  }
}

void processIncomingByte (const byte c){
  q++;
  outArray[q] = c;
  Serial.println(outArray[q],HEX);
  if(q > 107){
    q = 0;
  }
}

void loop() {
  //read the incoming data
  if (mySerial.available())
  processIncomingByte (mySerial.read());

  //concatenate 2 byte RPM data
  //    unsigned Value = outArray[31] << 8 | outArray[30];
  //    lcd.setCursor(0,1);
  //    lcd.print("temp: ");
  //    lcd.print((Value*0.18-40));
  //    display(); 
}

Nick Gammon

Yes, well I was thinking more like this:

Code: [Select]
/* 
This program is used for an Arduino to request and retrieve
data from an 07 Buell XB9SX Lightning ECM DDFI-2,
EEPROM Version: BUEIB

Currently the Arduino successfully sends a request code and
receives only 67 bytes of the run-time response, and displays
on the LCD screen and via serial monitor.

Created by Michael Blaylock
*/

// include the library code:
#include <LiquidCrystal.h>
#include <SoftwareSerial.h>

//new serial pins for ECM, 0 and 1 caused interferance from PC
#define rxPin 2
#define txPin 3

// set up a new serial port
SoftwareSerial mySerial =  SoftwareSerial(rxPin, txPin);

// initialize the library with the numbers of the interface pins
LiquidCrystal lcd(7, 8, 9, 10, 11, 12);
const byte inArray[9] = {
  0x01, //SOH
  0x00, //Emittend
  0x42, //Recipient
  0x02, //Data Size
  0xFF, //EOH
  0x02, //SOT
  0x43, //Data 1 //0x56 = Get version, 0x43 = Get runttime data
  0x03, //EOT
  0xFD, //Checksum

}
; //request code for real-time data
byte outArray[107]; //real-time data series from ECM
int First = 0;
int pos;

void setup(){

  lcd.begin(16, 2);  // set up the LCD's number of columns and rows:

  mySerial.begin(9600);  // baud rate for the ECM is 9600
  Serial.begin(9600);
  pos=0;

  lcd.setCursor(0,0);
  //Stream request data series to ECM as HEX
  for (int i = 0; i<9; i+=1){
    mySerial.write(inArray[i]);
    lcd.print(inArray[i],HEX); //print sent HEX code
  }
}  // end of setup


void loop()
{
  // fill buffer
  if (Serial.available () > 0)
    outArray[pos++] = Serial.read ();

  if (pos >= 107)
  {
    // process it

    pos = 0;  // ready for next time 

  }  // end if full
}  // end of loop
Please post technical questions on the forum, not by personal message. Thanks!

More info:
http://www.gammon.com.au/electronics

Blaylock1988

Thank you very much for the help. I will give your version a shot, but I tried something very similar to it and I was getting overflows in the arduino buffer.

Where you commented "buffer" were you referring to the arduino buffer or the outArray?

Nick Gammon

outArray.

The code in loop is pulling data from the Arduino internal buffer (which has a fixed size) into your buffer (which is the size you want) as fast as it can. Then when you have the number of bytes you want, you process them.

Are you certain you are getting 107 bytes? Or is it "up to" 107 bytes?
Please post technical questions on the forum, not by personal message. Thanks!

More info:
http://www.gammon.com.au/electronics

Blaylock1988

The run-time data that the ECM sends out is a 107 byte series of data.

The first table on the EcmSpy website shows the data that is coming in:
http://www.ecmspy.com/cgi-bin/runtime.cgi

The only time that the ECM will be sending out anything smaller is if there is an error code (which is around 10 bytes) but this only happens when I send an incorrect request or if I crash the bike.

Blaylock1988

#20
Mar 22, 2012, 01:59 am Last Edit: Mar 22, 2012, 02:01 am by Blaylock1988 Reason: 1
Well, I used your code, and it partially worked. I now get a full 107 byte buffer, however, every byte past byte 67 is 0, including the checksum, so I know I am still not reading the data correctly

here is my current code:

Code: [Select]
/* 
This program is used for an Arduino to request and retrieve
data from an 07 Buell XB9SX Lightning ECM DDFI-2,
EEPROM Version: BUEIB

Currently the Arduino successfully sends a request code and
receives only 67 bytes of the run-time response, and displays
on the LCD screen and via serial monitor.

Created by Michael Blaylock
*/

// include the library code:
#include <LiquidCrystal.h>
#include <SoftwareSerial.h>

//new serial pins for ECM, 0 and 1 caused interferance from PC
#define rxPin 2
#define txPin 3

// set up a new serial port
SoftwareSerial mySerial =  SoftwareSerial(rxPin, txPin);

// initialize the library with the numbers of the interface pins
LiquidCrystal lcd(7, 8, 9, 10, 11, 12);
byte inArray[9]; //request code for real-time data
byte outArray[107]; //real-time data series from ECM
byte long Value;
int First = 0;
int q;
int j;

void setup(){

  lcd.begin(16, 2);  // set up the LCD's number of columns and rows:


  mySerial.begin(9600);  // baud rate for the ECM is 9600
  Serial.begin(9600);
  q=0;
  inArray[0]=0x01; //SOH
  inArray[1]=0x00; //Emittend
  inArray[2]=0x42; //Recipient
  inArray[3]=0x02; //Data Size
  inArray[4]=0xFF; //EOH
  inArray[5]=0x02; //SOT
  inArray[6]=0x43; //Data 1 //0x56 = Get version, 0x43 = Get runttime data
  inArray[7]=0x03; //EOT
  inArray[8]=0xFD; //Checksum

  lcd.setCursor(0,0);
  //Stream request data series to ECM as HEX
  for (int i = 0; i<9; i+=1){
    mySerial.write(inArray[i]);
    lcd.print(inArray[i],HEX); //print sent HEX code
  }
}

void processIncomingByte (const byte c){
  q++;
  outArray[q] = c;
  Serial.println(outArray[q],HEX);
  if(q > 107){
    q = 0;
  }
}

//display the array in serial monitor one time
void display(){
  if(First < 1){
    if(j<107){
      Serial.println(outArray[j],HEX);
      j++;
    }
    else{
      First++;
      j=0;
    }
  }
}

//void loop() {
//  //read the incoming data
//  if (mySerial.available())
//    processIncomingByte (mySerial.read());
//
//  //concatenate 2 byte RPM data
//  //    unsigned Value = outArray[31] << 8 | outArray[30];
//  //    lcd.setCursor(0,1);
//  //    lcd.print("temp: ");
//  //    lcd.print((Value*0.18-40));
//  //    display(); 
//}

void loop() {
  // fill buffer
  if (mySerial.available () > 0)
    outArray[q++] = mySerial.read ();
  if(mySerial.overflow())
    Serial.print("overflow");
  if (q >= 107){
    // process it
    q = 0;  // ready for next time
  }  // end if full
  display();
}  // end of loop


and this is the response I got from the ECM:
Code: [Select]
1
42
0
overflow64
overflowFF
2
6
7B
E
0
0
0
0
0
0
0
0
BB
20
BB
20
B5
35
B5
35
33
0
F
A6
4
65
2
93
2
0
0
6B
2
1F
6
E8
3
0
0
E8
3
E8
3
E8
3
E8
3
30
4
30
4
0
80
0
0
0
0
0
0
0
FC
10
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0


I noticed that when I tried to read more than 107 bytes (up to 500) the code began to repeat after about 116 bytes (but no header), then after that repeat, it was just a bunch of random Hex mess.
Any ideas?

Nick Gammon

Code: [Select]
void loop() {
  // fill buffer
  if (mySerial.available () > 0)
    outArray[q++] = mySerial.read ();
  if(mySerial.overflow())
    Serial.print("overflow");
  if (q >= 107){
    // process it
    q = 0;  // ready for next time
  }  // end if full
  display();
}  // end of loop


But there is no point in displaying it until all 107 bytes have arrived.
Please post technical questions on the forum, not by personal message. Thanks!

More info:
http://www.gammon.com.au/electronics

PaulS

Quote
But there is no point in displaying it until all 107 bytes have arrived.

Not only is there no point in doing it, there is harm in doing it. It takes time to do the output, and, with the incoming stream of data that you have, time is something you don't have a lot of.

Variable names like q are easy to type, but they don't mean diddly. They are fine for loop indices, but not the way you are using them. Do yourself a big favor and use meaningful names.

Blaylock1988

Oh, I see what you are saying. I will move display() to this:

Code: [Select]
if (q >= 107) {
  // process it
  q = 0;  // ready for next time
  display();
}// end if full


and will rename the iteration count names and see how it goes.

Blaylock1988

Good news, I finally got it to read all 107 bytes of data. The ECM sends back 1 response per request so I need to continuously request and read the data from the ECM. I tried doing this by moving the request code into the the loop, but it seemed to be restarting the loop too quickly and it won't read all of the data. I tried it a different way and it was taking like 20 seconds between each time it read the data, and I have no idea why. Do you have any suggestions that I can continuously request and read the data, while performing math and other calculations at the same time quickly? I only need it to refresh the data a couple times a second.

This is the latest code that reads all 107 bytes of data one time and prints it:
Code: [Select]
/* 
This program is used for an Arduino to request and retrieve
data from an 07 Buell XB9SX Lightning ECM DDFI-2,
EEPROM Version: BUEIB

Currently the Arduino successfully sends a request code and
receives all 107 bytes of Real-time data, and displays
on the LCD screen and via serial monitor.

Created by Michael Blaylock
*/

// include the library code:
#include <LiquidCrystal.h>
#include <SoftwareSerial.h>

//new serial pins for ECM, 0 and 1 caused interferance from PC
#define rxPin 2
#define txPin 3

// set up a new serial port
SoftwareSerial mySerial =  SoftwareSerial(rxPin, txPin);

// initialize the library with the numbers of the interface pins
LiquidCrystal lcd(7, 8, 9, 10, 11, 12);
byte inArray[9]; //request code for real-time data
byte outArray[107]; //real-time data series from ECM
byte long Value;
int First = 0;
int q;
int j;

void setup(){

  lcd.begin(16, 2);  // set up the LCD's number of columns and rows:
  mySerial.begin(9600);  // baud rate for the ECM is 9600
  Serial.begin(9600);
  q=0;
  j=0;
  inArray[0]=0x01; //SOH
  inArray[1]=0x00; //Emittend
  inArray[2]=0x42; //Recipient
  inArray[3]=0x02; //Data Size
  inArray[4]=0xFF; //EOH
  inArray[5]=0x02; //SOT
  inArray[6]=0x43; //Data 1 //0x56 = Get version, 0x43 = Get runttime data
  inArray[7]=0x03; //EOT
  inArray[8]=0xFD; //Checksum

  lcd.setCursor(0,0);
  //Stream request data series to ECM as HEX
  for (int i = 0; i<9; i+=1){
    mySerial.write(inArray[i]);
    lcd.print(inArray[i],HEX); //print sent HEX code
  }
  //mySerial.write(0x01 0x00 0x43 0x02 0xFF 0x02 0x43 0x03 0xFD); //not sure on the correct syntax for this
}


//display the array in serial monitor one time
void display(){
  if(First < 1){
    for (int j = 0; j<107; j+=1){
      Serial.println(outArray[j],HEX);
    }
  }
  else(First++);
}


void loop() {
  // fill buffer
  if (mySerial.available () > 0)
    outArray[q++] = mySerial.read ();
  if (q > 106){
    // process it
    q = 0;  // ready for next time
    display();
  }  // end if full
}  // end of loop


I didn't get a chance to rename the indexes yet but I am working on it.

PaulS

So, inArray contains data the you send out, and outArray contains data that came in. Got it.

Quote
The ECM sends back 1 response per request so I need to continuously request and read the data from the ECM. I tried doing this by moving the request code into the the loop, but it seemed to be restarting the loop too quickly and it won't read all of the data.

Seems, then, like you should be sending the request for more data only when the end of the currently-being-received packet is detected.

Quote
I tried it a different way and it was taking like 20 seconds between each time it read the data, and I have no idea why.

Without showing us what you tried, we should know?

One assumption you are making is not a valid assumption. That assumption is that every byte sent will make it to the receiver, and that every byte the receiver sends will make it back. That assumption will need to be addressed.

There must be something in the packet sent that marks the start of a packet, and there should be something that marks the end of a packet. You should be checking for that start marker before storing the data. The end marker may arrive before the 107th byte, at which point you should stop reading, and possibly discard that packet.

There should also be some way to tell the ECM to send data in a continuous fashion. It seems difficult to believe that it needs to be polled for every bit of data.

Blaylock1988


So, inArray contains data the you send out, and outArray contains data that came in. Got it.

Yeah the frame of reference was the ECM, I might swap them because it isn't intuitive.

Quote
There must be something in the packet sent that marks the start of a packet, and there should be something that marks the end of a packet. You should be checking for that start marker before storing the data. The end marker may arrive before the 107th byte, at which point you should stop reading, and possibly discard that packet.

The beginning of the response has a header of data that is always the exact same 7 bytes, and the data always has a checksum. I suppose I would make sure that bytes 0-6 and byte 106 match, and then if it does save it to a new array.

Quote
There should also be some way to tell the ECM to send data in a continuous fashion. It seems difficult to believe that it needs to be polled for every bit of data.

I certainly hope there is, but from the wording from the EcmSpy website that has a PC program for communicating with the ECM:
Quote
"As mentioned in the chapter above, the ECM can be triggered to send a record of runtime data. These data consist of the current state (at the time of the query) of many input and output values, for example temperature sensores, injector pulsewidth and lots of other usefull information."

...it doesn't really sound like it

This project has proven to be pretty tough. It's hard to work on it because I have to have the bike with the key on (engine off) when working on the program, and the battery dies after a couple hours of work (even while its on a charger), so I have limited time each weekend to get a lot done.

PaulS

Quote
I suppose I would make sure that bytes 0-6 and byte 106 match, and then if it does save it to a new array.

No, don't try to copy all that data. If the 1st 6 bytes or the checksum don't match, just ignore the data in the array.

Quote
...it doesn't really sound like it

OK. It does sound like it expects to be asked for data. That should be OK, though, because that means it won't be flooding you with data faster than you can read it.

Put the code that initiates the transaction, now in setup(), into a function. Call it whenever you are done reading/storing/using a packet, and want another one.

Quote
This project has proven to be pretty tough. It's hard to work on it because I have to have the bike with the key on (engine off) when working on the program, and the battery dies after a couple hours of work (even while its on a charger), so I have limited time each weekend to get a lot done.

Bummer. But, it sounds like you are making good progress.

Blaylock1988

Quote
No, don't try to copy all that data. If the 1st 6 bytes or the checksum don't match, just ignore the data in the array.


The only bad thing is I need to process every byte in order to get the correct checksum, and I already have all the data in memory because the serial buffer is only 64 bytes.

Nick Gammon

You don't to keep 107 bytes in memory. A state machine, as I describe here, will pull the data out "on the fly"...

http://www.gammon.com.au/serial

As each byte arrives you can also add it to the running checksum. Then at the end, you check the checksum, and if it agrees, use the variables the state machine arrived at during its operation.

I did an example on that page for someone who had rather simpler requirements, but yours is basically more of the same concept.
Please post technical questions on the forum, not by personal message. Thanks!

More info:
http://www.gammon.com.au/electronics

Go Up