Questions about Serial for continuous data project

system · August 23, 2014, 3:46pm

Good day to everyone, may i remind you that I am pretty new to using serial commands in my experience of arduino and I may have a learning curve through out this thread.

First things first lets list down all the equipment i will be using in this project:

Arduino Mega
Power Analyzer MKII (See PDF attached.)

Obviously I will be using the Power Analyzer to send a continuous stream of data (VRMS,IRMS,PF, etc) in ASCII to my Arduino Mega. I will then display all of this data later on using processing and arduino/firmata.

I started my code like so:
(I'm using software serial at the moment , I know mega has 3 serials this is just what I started with)
Buffer is 64bits

#include <SoftwareSerial.h> 

SoftwareSerial mySerial(10,11);

void setup() {

  Serial.begin(9600); 
  mySerial.begin(9600); 
  mySerial.write(0x02); // Start  transmission
  mySerial.print("M3"); // to send a setting of M3 = [b]Comma Seperated Value[/b]
  mySerial.write(0x03); // End transmission
  // we initiated a csv setting   
}


void loop() {
  // put your main code here, to run repeatedly:
  delay(50);

  char value = mySerial.read();
  Serial.print(value); // using Serial.Print to check data read from mySerial
  delay(50);                                                                                 
  
   Serial.flush();
   delay(50);
}

THE PROBLEM:
I can read the data from the Power Analyzer just fine in CSV format it starts as follows:

STAT,VERSION,VRMS,IRMS,PREAL,VA,QAVERAGE,QINSTANT,PF,TEMPERA TURE,PHARMONIC,PFUNDAMENTAL,PQFUNDAMENTAL,WATTHR,INT

It is quite accurate but in the Serial Monitor, this data is only presented once and doesn't repeat again. My theory is that the buffer size of 64 bits is not enough and is filled up with the first reading and the rest of the data receive is trashed.

I need the community's help to make me understand the concept of buffer sizes.

How can gather my continuous data if there is a buffer limit ?
Can I delete the buffer each time I am done displaying the data so it can measure the values again?
How can i refresh the buffer ?
Do you have any suggestions for my project on how can i approach my Data?

Note: The Power analyzer used has a "on demand" feature that lets me just get the data i needed and not in a steam of CSV.

I thank you in advance for reading this long post and hopefully helping me out in my project.

Power Analyzer Manual rev 1r0.pdf (3.63 MB)

wildbill · August 23, 2014, 4:09pm

You set it to data on demand with that M3 command. Try M1 or M2 instead. Also, lose the delays and the flush.

Edit: I see you already know about on demand. You'll need to request data using D commands once it's in that mode, which explains why you never get any more.

system · August 23, 2014, 4:19pm

This is an excellent question and I am sure there are many other people who would be interested in the answer, since the product you are trying to interface to is only one of many that use the same or similar communication protocol. If you can get to the bottom of this it could open up a door to a myriad of other products that could be interfaced with the arduino.
Unfortunately my specialty is not software . It is hardware. Since I only have a hammer , everything looks like a nail to me. Without the
knowledge of an experienced programmer my first instinct would be to increase the EEPROM memory:

The ATmega2560 has 256 KB of flash memory for storing code
(of which 8 KB is used for the bootloader),
8 KB of SRAM and 4 KB of EEPROM (which can be read and written with the EEPROM library).

[ EEPROM Arduino Library, Non-volatile data storage](http:// EEPROM Arduino Library, Non-volatile data storage)

USE RTC (REAL TIME CLOCK)

1- read data into an array (save TIMESTAMP FOR EACH DATAGROUP#) (ie: Group-1:TS,,Group-2:TS ,etc)
2- store to eeprom in array format
3- read data using array index
4-store data to sd in whatever format you choose.

Can I delete the buffer each time I am done displaying the data so it can measure the values again?

el_supremo · August 23, 2014, 4:34pm

You aren't reading the serial data properly.
Try this to display the CSV lines.

#define C_STX 0x02
#define C_ETX 0x03
void loop()
{
  while(Serial.available()) {
    char c = Serial.read();
    // ignore the STX
    if(c == C_STX)continue;
    // print a new line for ETX
    if(c == C_ETX) {
      Serial.println("");
      continue;
    }
    // print anything else
    Serial.print(c);
  }
}

[edit] The reason the buffer filled up on you was because you keep delaying for 50ms. During each delay of 50ms the device can send about 50 characters. You have 3 delay(50) so it's no wonder that the buffer fills up.

Pete

system · August 23, 2014, 4:38pm

Here is a snippet of code I just cobbled togethter.
Use it as one suggested way to retrieve and temporary store the data. Or store it to EEPROM if you need it after power down. Same method anyway.
strTextEntry is in my case String of fixed length , what you called a buffer, and I use index to add data into it.
Basically you can use fixed length array of variable of your liking and index thru it monitoring for exit data or index count.
Or both or more conditions.

Or use pointer, pretty much same way.

You do not have to flush it, just reset the index to store next batch of data.
Have fun.
Vaclav

[code]

 do
  {
    while (!Serial.available());     // waits for input
    IncomingText = Serial.read();
   strTextEntry.setCharAt(index++,IncomingText);
.....
  // I exit while loop using return x; 

  }
  while(1);

[/code]

system · August 23, 2014, 5:02pm

Could you please post a link for the vendor of the power analyzer ?

zoomkat · August 23, 2014, 5:33pm

It appears your data is comma delimited. if so, you might use code similar to the below.

//zoomkat 3-5-12 simple delimited ',' string  
//from serial port input (via serial monitor)
//and print result out serial port

String readString;

void setup() {
  Serial.begin(9600);
  Serial.println("serial delimit test 1.0"); // so I can keep track of what is loaded
}

void loop() {

  //expect a string like wer,qwe rty,123 456,hyre kjhg,
  //or like hello world,who are you?,bye!,
  
  if (Serial.available())  {
    char c = Serial.read();  //gets one byte from serial buffer
    if (c == ',') {
      //do stuff
      Serial.println(readString); //prints string to serial port out
      readString=""; //clears variable for new input      
     }  
    else {     
      readString += c; //makes the string readString
    }
  }
}

system · August 24, 2014, 2:58am

wildbill:
You set it to data on demand with that M3 command. Try M1 or M2 instead. Also, lose the delays and the flush.

Edit: I see you already know about on demand. You'll need to request data using D commands once it's in that mode, which explains why you never get any more.

Yes I've tried that but as i mentioned the buffer keeps trashing extra input. I've read other replies i will try their suggested codes.

el_supremo:
You aren't reading the serial data properly.
Try this to display the CSV lines.
#define C_STX 0x02

#define C_ETX 0x03
void loop()
{
while(Serial.available()) {
char c = Serial.read();
// ignore the STX
if(c == C_STX)continue;
// print a new line for ETX
if(c == C_ETX) {
Serial.println("");
continue;
}
// print anything else
Serial.print(c);
}
}




[edit] The reason the buffer filled up on you was because you keep delaying for 50ms. During each delay of 50ms the device can send about 50 characters. You have 3 delay(50) so it's no wonder that the buffer fills up.

Pete

Thanks for this idea, I will try the code and update everyone later on.

system · August 24, 2014, 3:00am

raschemmel:
Could you please post a link for the vendor of the power analyzer ?

http://www.e-gizmo.com/

system · August 24, 2014, 3:01am

Vaclav:
Here is a snippet of code I just cobbled togethter.
Use it as one suggested way to retrieve and temporary store the data. Or store it to EEPROM if you need it after power down. Same method anyway.
strTextEntry is in my case String of fixed length , what you called a buffer, and I use index to add data into it.
Basically you can use fixed length array of variable of your liking and index thru it monitoring for exit data or index count.
Or both or more conditions.

Or use pointer, pretty much same way.

You do not have to flush it, just reset the index to store next batch of data.
Have fun.
Vaclav
[code]

do
{
while (!Serial.available()); // waits for input
IncomingText = Serial.read();
strTextEntry.setCharAt(index++,IncomingText);
.....
// I exit while loop using return x;

}
while(1);


[/code]

i will try this one also, Thanks for the replies!

wildbill · August 24, 2014, 3:48pm

Yes I've tried that but as i mentioned the buffer keeps trashing extra input.

The issue is your use of delay. There are 150ms of delay in loop, so as written, it takes ~150ms to read one character. In human readable form (M1) it looks like there are somewhere in the region of 250 chars transmitted, so it will take ~37 seconds to read a single transmission from the power analyzer. It transmits once a second, so while you're reading the first few bytes of that first transmission, another 36 sets of data have been sent to the arduino and thrown away because the read buffer was full.

You need to get rid of the delays and use Myserial.available() to check whether there is something to read and print.

On demand could work for you even with your delays because you can ask for a single data item at a time and then the buffer won't fill up, but I see no reason to do it this way unless you only want a subset of the data and even then, there's no need to do it so slowly.

system · August 27, 2014, 2:19pm

UPDATES:

I've tried el_supremo's script and it worked like a charm. All my buffer loss is now gone, Waiting for an available transmission also helped in gathering data I can now gather my CSV without trashing any serial reads.

like so:
http://postimg.org/image/f3s92vmqz/

The code now is:

void setup() {
  // initiate lang ng data
  Serial.begin(9600); //baud rate natin
  
  Serial1.begin(9600); //baud rate ng Serial1 
  Serial1.write(0x02); // Start natin ang transmission
  Serial1.print("M2"); // to send a setting of M2 = CSV
  Serial1.write(0x03); // End natin ang transmission
  
  Serial2.begin(9600); //baud rate ng Serial2
  Serial2.write(0x02); // Start natin ang transmission
  Serial2.print("M2"); // to send a setting of M2 = CSV
  Serial2.write(0x03); // End natin ang transmission
  
  Serial3.begin(9600); //baud rate ng Serial3 
  Serial3.write(0x02); // Start natin ang transmission
  Serial3.print("M2"); // to send a setting of M2 = CSV
  Serial3.write(0x03); // End natin ang transmission
  
  // we initiated a csv setting   
}
#define C_STX 0x02
#define C_ETX 0x03
//print
//write
//read

void loop() {
  // put your main code here, to run repeatedly:
    while(Serial1.available()) 
    { char c1 = Serial1.read();
    
    // ignore the STX
    if(c1 == C_STX)
    { Serial.print("A=");
      continue;}
    // print a new line for ETX
    if(c1 == C_ETX) {
      Serial1.println("");
       Serial.println();
      continue;}
    // print anything else
    Serial.print(c1);
    
    }
    
    ///////////B
    
    
    while(Serial2.available()) 
    {char c2 = Serial2.read();
    // ignore the STX
    if(c2 == C_STX)
    {Serial.print("B=");
      continue;}
    // print a new line for ETX
    if(c2 == C_ETX) {
      Serial2.println("");
      Serial.println();
      continue;}
    // print anything else
    Serial.print(c2);
    }
    
    
   /* /////////////C
    
    
    while(Serial3.available()) 
    {char c3 = Serial3.read();
    // ignore the STX
    if(c3 == C_STX)
    {Serial.print("C=");
      continue;}
    // print a new line for ETX
    if(c3 == C_ETX) {
      Serial3.println("");
      Serial.println();
      continue;}
    // print anything else
    Serial.print(c3);
    }
    */
}

Now after successfully gathering all the CSV, I now simultaneously operate 3 of my POWER ANALYZERS together in a 3 phase source having 1 power analyzer per line.

I first tried reading CSV from two power analyzers simultaneously:
http://postimg.org/image/5hyomkvl7/
Success as you can see A and B are indicated before the data is displayed.

NOW THE PROBLEM OCCURS AGAIN.

Now I've tried operating all three of the power analyzer and display their data in the serial monitor.

Serial Monitor Readings:
http://postimg.org/image/5x9yflhij/
AS you can see, We are having problems with regards when C is integrated to the serial monitor. There is data loss.

Now I need the community's help to tell me how do I approach this and troubleshoot the problem. I will accept any tips and tricks that you can share to me on how to resolve this.

BTW, Thank you for everyone's contribution to this topic so far. I am truly grateful.

system · August 27, 2014, 2:52pm

Serial data arrives slowly. Data may arrive at any time, on any serial port. Since you are mixing reading and printing, you will get confusion. What you need to do is read AND STORE the data, without printing it, from each port until the end of packet marker arrives. When that happens, print the stored data.

system · August 28, 2014, 7:52am

PaulS:
Serial data arrives slowly. Data may arrive at any time, on any serial port. Since you are mixing reading and printing, you will get confusion. What you need to do is read AND STORE the data, without printing it, from each port until the end of packet marker arrives. When that happens, print the stored data.

I see, So can I just put the values that I get from lets say; Serial1.Read and pass it to a string variable?

system · August 28, 2014, 9:33am

So can I just put the values that I get from lets say; Serial1.Read and pass it to a string variable?

Sure. Just don't mistake that for a String variable. Don't go there.

zoomkat · August 29, 2014, 2:45am

eraldino:

PaulS:
Serial data arrives slowly. Data may arrive at any time, on any serial port. Since you are mixing reading and printing, you will get confusion. What you need to do is read AND STORE the data, without printing it, from each port until the end of packet marker arrives. When that happens, print the stored data.

I see, So can I just put the values that I get from lets say; Serial1.Read and pass it to a string variable?

yes

http://forum.arduino.cc/index.php?topic=262658.msg1852993#msg1852993

system · August 30, 2014, 10:58pm

Okay, No problem my idea was to to use strcat after a serial read so all the data are sent in to a character array; c[100], then all data is sent there then i just output that concatenated character variable to the serial.

system · September 1, 2014, 5:53am

UPDATE:

I have tried to store the data into a string and output them again on the serial monitor. I get the same error as this:

http://postimg.org/image/5x9yflhij/

my code is:

void setup() {
  // initiate lang ng data
  Serial.begin(9600); //baud rate natin
  
  Serial1.begin(9600); //baud rate ng Serial1 
  Serial1.write(0x02); // Start natin ang transmission
  Serial1.print("M2"); // to send a setting of M2 = CSV
  Serial1.write(0x03); // End natin ang transmission
  
  Serial2.begin(9600); //baud rate ng Serial2
  Serial2.write(0x02); // Start natin ang transmission
  Serial2.print("M2"); // to send a setting of M2 = CSV
  Serial2.write(0x03); // End natin ang transmission
  
  Serial3.begin(9600); //baud rate ng Serial3 
  Serial3.write(0x02); // Start natin ang transmission
  Serial3.print("M2"); // to send a setting of M2 = CSV
  Serial3.write(0x03); // End natin ang transmission
  
  // we initiated a csv setting   
}
#define C_STX 0x02
#define C_ETX 0x03
//print
//write
//read

void loop() {
  String content;
  String content2;
  String content3;
  // put your main code here, to run repeatedly:
   while(Serial1.available()) 
    { char c1 = Serial1.read();
    // ignore the STX
    if(c1 == C_STX)
    {Serial.print("A=");
     continue;}
    // print a new line for ETX
    if(c1 == C_ETX) {
      Serial.println();
      continue;}
    // print anything else
    content.concat(c1);
    }
    Serial.print(content);
    
    
    
      while(Serial2.available()) 
    { char c2 = Serial2.read();
    // ignore the STX
    if(c2 == C_STX)
    {Serial.print("B=");
     continue;}
    // print a new line for ETX
    if(c2 == C_ETX) {
      Serial.println();
      continue;}
    // print anything else
    content2.concat(c2);
    }
    Serial.print(content2);
  
    while(Serial3.available()) 
    { char c3 = Serial3.read();
    // ignore the STX
    if(c3 == C_STX)
    {Serial.print("C=");
     continue;}
    // print a new line for ETX
    if(c3 == C_ETX) {
      Serial.println();
      continue;}
    // print anything else
    content3.concat(c3);
    }
    Serial.print(content3);
}

I don't know what went wrong now to generate the same error. I am storing the serial data into a string but seems like B and C are overlapping and I dont know why.

system · September 1, 2014, 6:37am

bump

wildbill · September 1, 2014, 12:03pm

You're still relying on your data coming in one continuous stream and you're servicing the power meters one after the other which is giving opportunities for overflow. Also, if for some reason you don't get all the data for one of them, what you have is thrown away because your content variables are local. In addition, you're printing three sets of data on a single serial port at the same speed- if you're unlucky this has overflow possibilities too.

So,

Make the content variables global (or static)
Change your while loops to if statements
Print and clear content only when you get ETX
Increase the baud rate on Serial

Also of course, String objects are bad news on an Arduino. Once you get this working, change it to use C strings instead. Search the forums for many threads debating why Strings are problematic.

Topic		Replies	Views
Character limit? Programming	34	9262	May 5, 2021
Getting reliable Serial.read()'s without a stop character Programming	23	8743	May 5, 2021
CSV file read stops after 9 lines? Programming	24	8281	May 5, 2021
Arduino-Arduino Serial Communication Errors Programming	21	9851	May 5, 2021
Serial freeze Interfacing w/ Software on the Computer	24	13368	May 6, 2021

Questions about Serial for continuous data project

Related topics