Serial data capture unreliable

JStainton:
the project is being handed over today

data that should read: 1,2,3,4,5 sometimes reads 1,4,5,5,5,2,3,2 so all the correct sensor readings are there but in strange orders and often repeated or on different lines all together. This could still be an issue with running out of memory

I think this could be right. I can't remember the details and I don't fully understand all this, but I spent quite a bit of time banging my head against the wall while trying to sort the garbled output from my Uno, and strings of any sort were not involved at all. You should check the memory allocation as indicated when you compile. If you have gone past the 32k the IDE will tell you. If you are over 30k it doesn't, but that only means the programme fits and is no guarantee that it will run properly. In my case, the solution was to stop immediately, drop the Uno, and get a Mega.

It's only noon over there, so if you can buy, beg. borrow, or steal a Mega in the lunch break, you might find you can snatch victory from the jaws of defeat.

It's only noon over there, so if you can buy, beg. borrow, or steal a Mega in the lunch break, you might find you can snatch victory from the jaws of defeat.

Thanks Nick, we/I are actually running into some huge problems with part of the system that is NOT the Arduino, therefore, the code/program is not a priority right now, but getting a mega in the next few weeks for an upgrade is a great option and we will probably do that.
The sketch isnt actually too big though, it comiles to about 12k, of the 32k allowable so it shouldnt be a problem with the Static memory, and could be a volatile MEM problem at runtime though. I will poke and prod the code and try a Mega on it and keep you guys updated. Finding a fix is always satisfying.

Here is some output, I have included a used memory value, from the freeMemory() function and it shows a lot of free memory but also the free memory INCREASES after a few strings are printed. NOTE: its output not code. The astrix are replacements for what is actually text, and the middle samples are just deleted to save space.

FreeMEM=1518
Please enter desired pressure[kPa] between 0.00 --> 50.00
Desired pressure=15.00kPa
Samples:
Sample #,Differential Pressure [Pa],********,Pref ADC Value,Reference Vessel Pressure [Pa],dp ADC Value,********,*******,FreeMEM=
0,-84.55,84.55,293,14254.50,471,23.000,28.250,1538
...
...
...
...
146,-167.88,14422.39,293,14254.50,334,23.062,28.562,1538
147,-167.88,14422.39,293,14254.50,334,23.125,28.562,1538
148,-167.88,14422.39,294,14303.15,334,23.125,28.625,1538
149,-167.88,14422.39,294,14303.15,334,23.062,28.625,1538
Recording complete
Lambda Decay value:,R^2 Value:
"=INDEX(LINEST(LN(C59:C157),A59:A157/1000000),1)","=RSQ(LN(C59:C157),A59:A157)"
FreeMEM=1538

Based on this do you still think there is a memory issue? There still may well be, I'm not trying to disprove your theory, I was surprised myself when I saw there was a large amount of free MEM.

JStainton:
it compiles to about 12k, of the 32k allowable so it shouldnt be a problem with the Static memory,

Indeed - and I consequently withdraw my suggestion! That's not the sort of figure I had in mind so, if the problem looks sorted by simply bringing in the Mega, it probably isn't, and is merely being put off till a more embarrassing time.

I submit you have only one goal at the moment, and that is to see sensible numbers on a screen. To this end you might try abandoning all thought of strings of any sort, do all the calcs in Arduino, and send rows of comma separated floats to the serial monitor. There is a good chance it will work first time - on the Uno!

One datalogging exercise is much the same as another, so the 24k of string-free code below might be useful. There is a swag of irrelevant stuff but the format should be similar. Yes, in this case the calc is required locally.

#include <OneWire.h>
#include <DallasTemperature.h>
#include <SPI.h>                  

#include <PCD8544.h>             // Nokia 5110
#include <SD.h>                  // SD card
#include <string.h>              // from Date As Filename 
#include "RTClib.h"              // from Date As Filename
#include "Wire.h"                // MUST HAVE lib for LCD disp, SD card, and serial

#define DS1307_ADDRESS 0x68

RTC_DS1307 RTC;
static PCD8544 lcd;

File myFile;
char filename[] = "00000000.CSV";

// Custom symbols
static const byte DEGREES_CHAR = 1;
static const byte degrees_glyph[] = { 0x00, 0x07, 0x05, 0x07, 0x00 };
static const byte SLASH_CHAR = 2;
static const byte slash_glyph[] = {0x00,0x20,0x10,0x08};

// Yellow group Lismore
byte InThermo[8] =  {
  0x28, 0x39, 0xFD, 0x50, 0x04, 0x00, 0x00, 0X69};
byte OutThermo[8] = {
  0x28, 0x09, 0xA9, 0xC0, 0x03, 0x00, 0x00, 0x95};
byte DrainThermo[8] = {
  0x28, 0x62, 0xA5, 0x2D, 0x04, 0x00, 0x00, 0x21}; 

#define ONE_WIRE_BUS 3
OneWire oneWire(ONE_WIRE_BUS);
DallasTemperature sensors(&oneWire);

int  second, minute, hour, weekDay, monthDay, month, year;
int k=0;

const int chipSelect = 4;

float InTemp, OutTemp, DrainTemp, diff;    

// Define the strings for our datastream IDs
char sensorId0[] = "InThermo";
char sensorId1[] = "OutThermo";
char sensorId2[] = "DrainThermo";
char calcId1[] = "diff";

void setup() {
   lcd.begin(84, 48);
     // Register the custom symbols...
  lcd.createChar(DEGREES_CHAR, degrees_glyph);
  lcd.createChar(SLASH_CHAR, slash_glyph);
  
  Wire.begin();
  Serial.begin(9600);

  delay(300);//Wait for newly restarted system to stabilize
  
  lcd.setCursor (0,0);
  lcd.print("Initializing");
  delay(2000);
  lcd.setCursor (0,1);

  pinMode(53, OUTPUT);

 if (!SD.begin(chipSelect)) 
  {
    lcd.print("failed!");
    delay (2000);
    return;
  }
  lcd.println("init. OK!");
  delay(2000);
      getFileName();
      lcd.println(filename);
        delay(2000);
  lcd.clear();

// First serial output "LABEL" is an Excel command, printed on reset
  Serial.println("LABEL,Time,InTemp,OutTemp,diff,DrainTemp");

  sensors.setResolution(InThermo, 12);
  sensors.setResolution(OutThermo, 12);
  sensors.setResolution(DrainThermo, 12);
  
    running();
}

void loop() {

  GetClock();
  if (hour == 0 && minute == 0 && second <2)
  {
    getFileName();
  }

  //get the values from the DS8B20's 
  sensors.requestTemperatures();

  InTemp = (sensorValue(InThermo));
  OutTemp = (sensorValue(OutThermo));  
  DrainTemp = (sensorValue(DrainThermo)); 

  diff = OutTemp - InTemp;
  
 Serial.print("DATA,TIME,");  // Excel commands 
  Serial.print(InTemp);
  Serial.print(" ,  ");
  Serial.print(OutTemp);
  Serial.print(" ,  ");
  Serial.print(diff);
  Serial.print(" ,  ");
  Serial.println(DrainTemp);
 
  lcd.setCursor(49,0);
  lcd.print(InTemp);
  lcd.setCursor(49,1);
  lcd.print (OutTemp);
  lcd.setCursor(49,2);
  lcd.print(DrainTemp);
  lcd.setCursor(49,3);
  lcd.print(diff);  
  

  k=k+1;  

  if (k>9 )
  {  
  myFile = SD.open(filename, FILE_WRITE);//<<<<<<<<<<<<< OPEN
  myFile.print(hour);
  myFile.print(":");
  myFile.print(minute);
  myFile.print(":");
  myFile.print(second);
  myFile.print(",");

  myFile.print(InTemp);
  myFile.print(",");
  myFile.print(OutTemp);
  myFile.print(",");
  myFile.print(DrainTemp);
  myFile.print(",");
  myFile.println();
       myFile.close();//>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>CLOSE
       
      k=0;
  }
  delay(850);
}  // loop ends here

//sensorValue function
float sensorValue (byte deviceAddress[])
{
  float tempC = sensors.getTempC (deviceAddress);
  return tempC;
}

byte bcdToDec(byte val)  {
  // Convert binary coded decimal to normal decimal numbers
  return ( (val/16*10) + (val%16) );
}

void GetClock(){
  // Reset the register pointer
  Wire.beginTransmission(DS1307_ADDRESS);
  byte zero = 0x00;
  Wire.write(zero);
  Wire.endTransmission();
  Wire.requestFrom(DS1307_ADDRESS, 7);

  second = bcdToDec(Wire.read());
  minute = bcdToDec(Wire.read());
  hour = bcdToDec(Wire.read() & 0b111111); //24 hour time
  weekDay = bcdToDec(Wire.read()); //0-6 -> sunday - Saturday
  monthDay = bcdToDec(Wire.read());
  month = bcdToDec(Wire.read());
  year = bcdToDec(Wire.read());
}

void getFileName(){

  DateTime now = RTC.now();

  filename[0] = (now.year()/1000)%10 + '0'; //To get 1st digit from year()
  filename[1] = (now.year()/100)%10 + '0'; //To get 2nd digit from year()
  filename[2] = (now.year()/10)%10 + '0'; //To get 3rd digit from year()
  filename[3] = now.year()%10 + '0'; //To get 4th digit from year()
  filename[4] = now.month()/10 + '0'; //To get 1st digit from month()
  filename[5] = now.month()%10 + '0'; //To get 2nd digit from month()
  filename[6] = now.day()/10 + '0'; //To get 1st digit from day()
  filename[7] = now.day()%10 + '0'; //To get 2nd digit from day()
}

void running(){
  lcd.setCursor(0,0);
  lcd.print("In");
  lcd.setCursor(31,0);
  lcd.print("\001C ");
  lcd.setCursor(0,1);
  lcd.print("Out");
  lcd.setCursor(31,1);
  lcd.print("\001C ");
  lcd.setCursor(0,2);
  lcd.print("Drain");
  lcd.setCursor(31,2);
  lcd.print("\001C ");
  lcd.setCursor(0,3);
  lcd.print("diff");
  lcd.setCursor(31,3);
  lcd.print("\001C "); 
}

I have just seen your last post. I'm afraid it is beyond me and I can't comment, but I still think what I have said above is valid!

I have included a used memory value, from the freeMemory() function and it shows a lot of free memory but also the free memory INCREASES after a few strings are printed.

The underlying assumption is that the first value returned by freeMemory() is actually valid. I'm willing to bet that, in this case, it is not. I suspect that you have completely overrun memory before the application even starts.

You might consider downloading the 1.5+ version of the IDE that shows how much of SRAM is being used AT COMPILE TIME. The value is of NO use at run time, but if the compiler tells you that you are using n of m bytes, and n is greater than m, you know that it isn't going to ever get better.

Its a real pain... I have written an enhanced serial monitor - ESM as I call it.

I hoped that once my Arduino was running and the PC was running the comms would be 100%...

Every once in a while (over hours of soak testing with dummy data) a rogue transmission/receive would occur. I know this as I check the receive data to be complete and correct... and it is'nt.

So I wrote checking logic into the ESM and if the data received doessn't agree I throw an exception... which can be ignored or dealt with as you think fit.

What do I think is going on? Not 100% sure but (or even 95%, 90%...) but possibly the Arduino sent the right stuff but the PC got its knickers twisted while/when something I didn't control kicked in. like a virus check?

If you are running of memory then you can do something about that in your Arduino code (up to a point). That's a differnt sort of problem which I hit and hurt me for a long time especially when its marginal.

By the way my tests also showed that some clone Unos were unreliable at very low baud rates but in practice unlikley to be of concern as who needs them.

Still upsets me but at least I know it can happen (despite what I was told) and so deal with it in my code... if its really critical you need to develop a handshaking protocol and if the data isn't received correctly ask for it again... bullet proofing code is tricky.

Also checked this with someone at Cambs university workking on comms stuff and said... "you have to plan for it going wrong and write to code appropriately , or plan for a rough time in front of the boss when it goes wrong", 'Nuff said!

I don't think the situation is as bad as you make out. I just use the standard procedures and don't claim any great expertise, but whenever I have seen rogue data, it has always been the result of one thing - rogue code. And you don't usually have to wait too long for that to show up.

Agreed about rogue code but rogue behviour once in a blue moon... you'll have decide whether it is important or not, which will depend on what you are doing.

Don't assume that for everybody and all applications its safe to ignore - put on your thinking hat first.

I do believe it's important but I have never seen any evidence of it. I guess the importance also depends on whether it is just data, or data that automatically triggers a critical action.

Hi Guys!
Been away from the office for weeks now, only just sitting back in front of my desk.
Update on the project, plugging the USB into the brand new computer purchased just to run the arduino. The problem is worse. Dramatically so.
Plugging it into a brand new computer with slightly different specs, ok but still some of the original problems.
In speaking to one of the Embedded lecturers at Canterbury University, his suggestion was, problems with Windows and its virtual serial buffer, and to run it on linux. Well that is completely out of the question. He also suggested some form of flow control.
But I dont really understand why it would need flow control, its basic information, the communication doesnt seem to have any bit level corruption and finally this is an arduino. They do communicate with windows computers everyday, if flow control was really needed this problem would have probably already been solved and/or well documented.

Sorry also meant to say, running with BETA IDE I can see I have used 41% of program mem which I already knew, and 35% of the dynamic memory if that helps anyone
Cheers for your help

JStainton:
In speaking to one of the Embedded lecturers at Canterbury University, his suggestion was, problems with Windows and its virtual serial buffer, and to run it on linux. Well that is completely out of the question. He also suggested some form of flow control.
But I dont really understand why it would need flow control, its basic information, the communication doesnt seem to have any bit level corruption and finally this is an arduino.

Yes, this is an Arduino, and yes, it looks like just basic information - rather like what a lot of other people are doing

The first statement by the embedded lecturer (presumably extremely well-embedded) amply demonstrates he has no idea of what he is talking about, and therefore you can confidently ignore the second.

At this stage, I don't suppose you are going to get anywhere without a proper discussion of the code but I'm sure that is where the problem lies. I have already shown you mine. If you can't show yours that doesn't work, you might be able to elaborate on the differences from mine, that does. Most particularly on the differences in objective - if any.

I believe this will suffice for source code requested:
http://forum.arduino.cc/index.php?topic=560231