Serial data capture unreliable

Hi Guys,

Im using the Uno R3 board to read a whole bunch of sensors and then output the information into a CSV file using the Terminal.exe software found here. Terminal

I have had a search around on the forum and nothing I can find seems to be similar to this:
The problem I have been getting is occasionally, one of the lines that is output will be missing a value, or randomly have an extra delimiter (comma in this case) thrown in, which as you can understand upsets my columns.

Now I am trying something a little more complicated where I include an excel equation as the last line to print, this is garbelled about 10% of the time, which I dont understand why.

Is this a problem with my baud rate? I have tried from 1200 to 19200 and the results seem pretty similar. Should I be using .print or .write? I think .write is quicker? Or do you think there is some problem with the capture software I am using? It doesnt seem like it could be that as its just doing a basic task of reading serial data. OR any recommendations on better ways to log to a CSV file with the end user having no coding background?

My problematic code is below, along with a screenshot

//Include lambda decay equations inside the CSV file and select the correct index...................................................
  Serial.write("Lambda Decay value:");
  Serial.write(",");
  Serial.println("R^2 Value:");
  delay(100);  
  Serial.write("\"");
  Serial.write("=INDEX(LINEST(LN(C");
  Serial.print(index1);
  Serial.write(":C");
  Serial.print(index2);
  Serial.write(")");
  Serial.write(",A");
  Serial.print(index1);
  Serial.write(":A");
  Serial.print(index2);
  Serial.write("/1000000),");
  Serial.write("1)\"");
  delay(100); 
  Serial.write(",");
  Serial.write("\"");
  Serial.write("=RSQ(LN(C");
  Serial.print(index1);
  Serial.write(":C");
  Serial.print(index2);
  Serial.write(")");
  Serial.write(",A");
  Serial.print(index1);
  Serial.write(":A");
  Serial.print(index2);
  Serial.write(")");  
  Serial.println("\"");
  //...

Any help would be appreciated, it could be as simple as poor poor programming,

Thanks

outputerror.png1366×768 68.2 KB

You might try PLX-DAQ which is a free macro for Excel. Arduino feeds a mixture of Data and Excel commands. The code looks a lot simpler than what you are doing, and all Excel equations are done in Excel.

Thanks for the suggestion Nick_Pyner, I have had a look at PLX-DAQ in the past and while it is pretty great I think for my application I need to use Terminal. The reason for this is every piece of data sent from the arduino needs to be logged and saved into a CSV file by a user whom may not be tech savvy. The equation I am outputting isnt actually calculated by the arduino just printed. The CSV file(excel) is actually the program that crunches the equation, I am just using the arduino to do the typing for me.

The crux of my problem seems to be some form of corruption of the serial data and not the data itself.

Thanks

Just for the sake of clarity I have attached an image of the correct output

outputCorrect.png1366×768 114 KB

JStainton:
I have had a look at PLX-DAQ in the past and while it is pretty great I think for my application I need to use Terminal. The reason for this is every piece of data sent from the arduino needs to be logged and saved into a CSV file by a user whom may not be tech savvy.

I don't think that is a reason. Excel with PLX-DAQ saves a CSV just like any other terminal programme. What is offers is that it can timestamp and crunch numbers on the fly, thereby both absolving Arduino from doing maths, and reducing the amount of data that needs to be sent.. Having said that, it wouldn't be any better at handling bad data and I suspect you are right - it is poor programming.

I don't know why you need all that mixture of indexes, serial.write and serial.print. I send out data to CSV all the time, I have never had to do anything like that. It is serial.print all the time

  Serial.print(InTemp);
  Serial.print(" ,  ");
  Serial.print(OutTemp);
  Serial.print(" ,  ");
  Serial.print(diff);
  Serial.print(" ,  ");
  Serial.println(DrainTemp);

It works, and and it works the same irrespective of the terminal programme at the other end. I don't see why the terminal you use should be any different. One CSV file is, by definition, much the same as another.

Is this a problem with my baud rate? I have tried from 1200 to 19200 and the results seem pretty similar.

Why not 115200? Get out of the stone age.

Should I be using .print or .write?

print() for ASCII data. write() for binary data. I don't see any binary data.

My problematic code is below

No. Some of your code is there, but not all of it. I'd be concerned that you don't have enough memory for all those string literals.

Nick_Pyner:
I don't know why you need all that mixture of indexes, serial.write and serial.print. I send out data to CSV all the time, I have never had to do anything like that. It is serial.print all the time

Thanks for the reply Nick_Pyner, I originally wrote all the code using only Serial.print() and I always planned to use print() I only changed it based on a comment made somewhere else on the forums that write() is more robust in some cases.

Nick_Pyner:
I don't think that is a reason. Excel with PLX-DAQ saves a CSV just like any other terminal programme. What is offers is that it can timestamp and crunch numbers on the fly, thereby both absolving Arduino from doing maths, and reducing the amount of data that needs to be sent.. Having said that, it wouldn't be any better at handling bad data and I suspect you are right - it is poor programming.

I admit I haven't used PLX-DAQ, just looked at online tutorials that demonstrate some of its functionality I based my choice off of parameters I was given in my brief and Terminal seemed to be a closer fit (I could be wrong) as it also has macro buttons that load up each time.

PaulS:
Why not 115200? Get out of the stone age.

Why not a billion? Or a trillion? Because I assumed with data corruption increasing the baud rate can cause more transmission errors. Unless you think the baud rate being too slow is what is causing my problems, your suggestion is similar to doubling your speed on the motorway because you can feel vibrations coming through the steering wheel.

PaulS:
No. Some of your code is there, but not all of it. I'd be concerned that you don't have enough memory for all those string literals.

Yes, that is definitely SOME of my code. It would be amazing if I could write a program using just those lines. Sorry I dont feel comfortable uploading too much more as the code(+comments) would give away too much about what we are using the ardiuno for, which is commercially sensitive.
When you say you are concerned about running out of memory for all the string literals, what do you mean? I thought after each print() is run, the memory it used it instantly freed up.
Any suggestions to improve this method of printing? I could always use string concatenate and then print one string that contains all my relevant data.

Thanks

When you say you are concerned about running out of memory for all the string literals, what do you mean? I thought after each print() is run, the memory it used it instantly freed up.

Better rethink your commercial aspirations.
The strings are copied to RAM, where they remain throughout the life of the sketch.

AWOL:

When you say you are concerned about running out of memory for all the string literals, what do you mean? I thought after each print() is run, the memory it used it instantly freed up.

Better rethink your commercial aspirations.
The strings are copied to RAM, where they remain throughout the life of the sketch.

Ah! I did not realise they are copied to RAM, I thought they were just executed. Can you see a way of reducing the amount of RAM usage that could be causing this problem. Say, would concatenating everything into one string and then printing that, reduce the RAM overhead?

Why do I need to rethink my commercial aspirations? My coding? Because (beginner as they may be) I dont think my coding skills are poor enough to ruin the company.

Can you see a way of reducing the amount of RAM usage that could be causing this problem

Yes, of course.
Leave the constant strings in flash.
Make better use of the F() macro.

JStainton:
I originally wrote all the code using only Serial.print() and I always planned to use print() I only changed it based on a comment made somewhere else on the forums that write() is more robust in some cases.

I think you were right the first time, and this stuff about robustness is nonsense. Print and write serve different purposes.

Since we are talking Excel and CSV, I'm sure the data is numeric, so all you need do is read the numbers and print them - plus the commas. There is no need to use strings, they are literally a waste of time, can lead to disaster, and are best avoided.

I admit I haven't used PLX-DAQ, just looked at online tutorials that demonstrate some of its functionality I based my choice off of parameters I was given in my brief and Terminal seemed to be a closer fit (I could be wrong) as it also has macro buttons that load up each time.

I only raised PLX because I thought the data feed was simpler. I bet this was wrong, the data is probably much the same for both, and your method of feeding is the problem. Stick with Terminal.

I bet he baud rate has nothing to do with the problem, all you need is for the Arduino and Terminal to match. I never use anything but 9600 - principally because there has never been any evidence of a need to change. Solve the feed first, and then worry about the speed - in the unlikely event that you need to.

You have probably shown all the code you really need to, even though I can't even tell what your variables are - maybe index1 is one of them.

AWOL:

Can you see a way of reducing the amount of RAM usage that could be causing this problem

Yes, of course.
Leave the constant strings in flash.
Make better use of the F() macro.

Thanks for the suggestion, it doesnt really fix much. Here is my code for two separate lines being printed, still about 10% of the time the output on the Terminal and in the CSV file is all wrong, sometimes with all the correct characters but just in a different order. Or new lines have been entered, delimeters are all over the show.
Then it behaves itself for a few tests, with no problems.

    Serial.print(F("Sample #,Differential Pressure [Pa],Total Pore Pressure Decay,Pref ADC Value,Reference Vessel Pressure [Pa],dp ADC Value,Acetal Temp[C],Micro Temp[C]"));

and another problematic line is

Serial.print("\"");
  Serial.print(F("=INDEX(LINEST(LN(C"));
  Serial.print(index1);
  Serial.print(":C");
  Serial.print(index2);
  Serial.print(")");
  Serial.print(",A");
  Serial.print(index1);
  Serial.print(":A");
  Serial.print(index2);
  Serial.print(F("/1000000),"));
  Serial.print("1)\"");
  delay(100); 
  Serial.print(",");
  Serial.print("\"");
  Serial.print("=RSQ(LN(C");
  Serial.print(index1);
  Serial.print(":C");
  Serial.print(index2);
  Serial.print(")");
  Serial.print(",A");
  Serial.print(index1);
  Serial.write(":A");
  Serial.print(index2);
  Serial.write(")");  
  Serial.println("\"");

Note I am now using the F() macro to hold some of my strings in Flash to try free up some SRAM. (Hopefully I am using this correctly?)
FYI index1 and index2 are just for the excel formula to make sure it selects the correct cells, as the cells change slightly between test runs.

This just seems like a strange problem that it works well most of the time, and then sometimes the serial data being logged is all a bit odd

JStainton:
about 10% of the time the output on the Terminal and in the CSV file is all wrong,

I am now using the F() macro to hold some of my strings in Flash to try free up some SRAM. (Hopefully I am using this correctly?)

Probably not, as suggested by the erratic nature of your output. More to the point, do you have a good reason for using strings?

Nick_Pyner:
Probably not, as suggested by the erratic nature of your output. More to the point, do you have a good reason for using strings?

What is the alternative to using strings? I do need to have headings to my columns for readability (and to meet QA requirements),
and the last output is all strings as I need to input that equation into my CSV file so it can crunch my numbers for me.

I have never had a problem printing out strings before, but if I have, it usually fully works, or nothing works. Not this "inconsistent" output that it is currently doing.

JStainton:
What is the alternative to using strings?

Is the data you are collecting strings or numbers? Does the receiver need strings or numbers to do what you need it to do? If the latter to both, why do you need to send it strings, instead of sending it number like everybody else does?

I do need to have headings to my columns for readability (and to meet QA requirements),

Really? As I recall it, the end product is in Excel and the heading could be in there as a template. As it happens, I don't see a problem with sending a fixed heading. I do it every time the file starts

and the last output is all strings as I need to input that equation into my CSV file so it can crunch my numbers for me.

Does the equation vary? If so, I guess it's not an equation, it's data. If not, you either do the calc in Arduino or do it in Excel. Excel doesn't need to be told what to do by Arduino, it just has a column that responds to the data fed to other columns. That is what Excel is all about. The only reason for having Arduino do the calcs is that the result is needed locally.

I have never had a problem printing out strings before, but if I have, it usually fully works, or nothing works. Not this "inconsistent" output that it is currently doing.

Indeed, but messy data is the only explanation I can see, and the unnecessary use of strings is the root cause of it. The only reason I can see for ever using datastrings is where the data receiver demands it. Excel doesn't, surely.

It might not help you, but there is a similar three ringed circus here.

http://forum.arduino.cc/index.php?topic=212255.msg1558900#msg1558900

Since you are using a Uno, you may also be having memory problems unrelated to all this - compile OK, but runs rough.

Hey guys,
Still haven't managed to rectify the problem, the project is being handed over today so I really just haven't had time to look at the code too much. I had our head software designer look at it and while he doesn't use the Arduino environment, he couldn't find what was causing the problems.

Nick_Pyner:

JStainton:
What is the alternative to using strings?

Is the data you are collecting strings or numbers? Does the receiver need strings or numbers to do what you need it to do? If the latter to both, why do you need to send it strings, instead of sending it number like everybody else does?

In response to Nick's last message, I was initially planning on using a template and only inputting numerical(floats and ints) data into the csv file, this seemed to be slightly less user friendly for the people trying to use it, so I tried to adapt quickly. I have also noticed while cruising some of the raw data files that actually some of the numerical data is incorrect too, i.e data that should read: 1,2,3,4,5 sometimes reads 1,4,5,5,5,2,3,2 so all the correct sensor readings are there but in strange orders and often repeated or on different lines all together. This could still be an issue with running out of memory/using a lot of string literals, or it could be indicating a different problem.

Does the equation vary? If so, I guess it's not an equation, it's data. If not, you either do the calc in Arduino or do it in Excel. Excel doesn't need to be told what to do by Arduino, it just has a column that responds to the data fed to other columns. That is what Excel is all about. The only reason for having Arduino do the calcs is that the result is needed locally.
I am definitely keen on tidying this problem up myself though, but it will have to be at a later date.

The equation does not vary, but the parameters of the equation do vary, ie which cells the equation uses does change everytime. And the cells to use is decided by the Arduino. You are correct, the Arduino does not need the answer to the equation locally, therefore it leaves the equation for excel. I was initially asked by my supervisor to have the Arduino crunch the numbers and spit out just the end product, but it would have to finish all the sampling and then do the equation, and there is not enough memory for that.

Shoutout to Nick_Pyner for providing a lot of useful discussion and also not being patronising. I have spent a lot of time on these forums over the past 5 years and I think they are great, but I still dont understand why some people feel that they are the only people on here that are intelligent, and everyone else is a drain on their time. Arduinos are key for young people learning about coding and electronics, something we need to encourage and nurture, instead of belittle and humiliate.
Posting on these forums shouldn't be daunting, it should be inviting and friendly and most of all educational.

Can you show us your code?

This is the code that prints out the numerical data. This is all the code I can really show you. NOTE the for loop doesnt close as there are some other things that are executed inside the for loop.
This is all I can really show you, so I can understand it is difficult to diagnose problems without seeing the whole thing.

for(int y=0; y<num; y++)
    {

      sensors.requestTemperatures();
      int refsensorVal = 0;
      int dpsensorVal = 0;
      for(int i=0; i<numAveSamples; i++)
      { 
        refsensorVal += analogRead(refSensor); //Takes 100uS per cycle
        dpsensorVal += analogRead(dPSensor); //Takes another 100uS per cycle  
      }

      refsensorVal /= numAveSamples;
      dpsensorVal /= numAveSamples;
      if(y==1){                          //Set our stable Pref value.
        firstPressValue = refsensorVal*absconversion;
      }
      if(dpsensorVal>=1023)
      {
        digitalWrite(ledBlue, LOW);
        break;
      }
      digitalWrite(ledBlue, HIGH);
      Serial.print(y, DEC);
      Serial.print(",");
      Serial.print((dpsensorVal-caliDP)*dpconversion);
      Serial.print(",");
      Serial.print(firstPressValue-((dpsensorVal-caliDP)*dpconversion));
      Serial.print(",");
      Serial.print(refsensorVal);
      Serial.print(",");
      Serial.print(refsensorVal*absconversion);
      Serial.print(",");
      Serial.print(dpsensorVal);
      Serial.print(",");
      printTemperature(redTempSensor);
      Serial.print(",");
      printTemperature(greenTempSensor);
      Serial.println("");
      digitalWrite(ledBlue, LOW);

This is all the code I can really show you.

Why?
Where's the rest of it?

AWOL:

When you say you are concerned about running out of memory for all the string literals, what do you mean? I thought after each print() is run, the memory it used it instantly freed up.

Better rethink your commercial aspirations.
The strings are copied to RAM, where they remain throughout the life of the sketch.

I would have thought by you making this smartarse comment that you had already read that I am not willing to show you ALL my code because it is commercially private.

OK.
Good luck.