CHANGE/manipulate data format from an analytical scale (several lines output)

Hi all,

I am coming from this (successful) thread.

HERE

the problem is now HOW to format the data properly. I have a huge gap between the readings in the CSV file!

Any help will be really appreciated :slight_smile:

// Code for receiving the data from my analytical scale through RS232 (V5)

// Libraries
#include <LiquidCrystal.h>
#include <SPI.h>
#include <SD.h>

// Interface and Objects definitions
LiquidCrystal lcd(8, 9, 4, 5, 6, 7); // Creates a LCD LiquidCrystal object
const int chipSelect = 10; // Assigns the pin for the SPI-SD chip selection
boolean newData = false;
int receivedData; // Receives an integer in ASCII to transform in CHAR to be visualised/printed

void setup()
{
  Serial.begin(9600, SERIAL_7N1); // Includes the scale parameters
  boolean newData = false;
  pinMode(chipSelect, OUTPUT); // Ensures that the SPI-SD selection pin is an output
  lcd.begin(16, 2); // Initialises the interface to the LCD screen
  lcd.clear(); // // Clears the LCD screen and positions the cursor in the upper-left corner
  if (!SD.begin(chipSelect)) // Checks if the SD card is working
  {
    lcd.println("SD failed!");
    delay(2000);
    return;
  }
}

void loop()
{
  recvData();
  saveData();
}

void recvData()
{
  if (Serial.available() > 0)
{
  receivedData = Serial.read();
  newData = true;
}
}

void saveData()
{
  if (newData == true)
  {
    File readings = SD.open("readings.csv", FILE_WRITE); // Opens the file on the SD card
    if (readings) // If the file opened OK, write to it on SD card
    {
      readings.print((char)receivedData); // Saves the data on the SD card
      // readings.print(',');
      readings.close(); // Closes the file to save the data on the SD card
    }
    else // If the file didn't open, print an error
    {
      lcd.println("FILE NOT OPEN");
      delay(2000);
      lcd.clear();
      newData = false;
    }
  }
}

Thanks a lot.

I have a huge gap between the readings in the CSV file!

what do you mean? writing to the SD is too slow compared to the pace at which data comes in and you loose data? opening & closing the file descriptor for every character might be somewhat suboptimal...

which SD library do you use? (seems the standard one - you should envision the SDFat library and handle opening & closing your file differently. have a look at the example LowLatencyLogger.ino

I am trying to be more accurate in my question/scope:

At the moment I am able to obtain the same output format of the Putty screen. I am suggesting THIS is the way my scale is sending the information, right. Plain and simple:

And the next reading is after lot of raws:

What I would like to achieve is this, date, time and reading in different column and then carriage return and no gaps:

This is in an ideal world. But we are here to try to learn, don't we? I will try to improve my code in the next 20mins, but any suggestion will be really helpful. I am very "new" on this subject.

Mario

How are things wired?
How do you get data to your "Putty screen"?

What is with those "upload and share your photos" links?

Things are wired Arduino UNO + TLL converter (with RX/TX swapped).

I gather data on Putty through a NULL cable connected with a USB RS232 adaptor on my laptop, I was able to program the scale to sent the information every 10sec.

Please refer to the link in the first post to the original thread where I was asking for feasibility of this project. Thanks a lot.

Please note originally I had also the readings showing on the LCD screen but AT THIS STAGE I removed from the code to solve 1 step per time...

Coding Badly sorry that link is due by the image uploader I am using to post the screenshot here...

Thanks a lot guys for starting this conversation. I am really appreciating.

I know that might be "annoying" but all the information re this project are in the other thread (first post of this conversation), it is not too long...

Mario

J-M-L:
what do you mean? writing to the SD is too slow compared to the pace at which data comes in and you loose data? opening & closing the file descriptor for every character might be somewhat suboptimal...

which SD library do you use? (seems the standard one - you should envision the SDFat library and handle opening & closing your file differently. have a look at the example LowLatencyLogger.ino

No I do not loose data, they are simply too far away as you can see from the number of raw in the screenshots.

I would suggest to try to make it works and then get into the details otherwise for me it will be difficult to learn from zero lot of complicated notions..if you agree :slight_smile:

I will now try to separate the data in columns in the next code and think what is causing that massive gap...

well in post #14 there you were showing this type of input


what has changed?

Here I am:

// Code for receiving the data from my analytical scale through RS232 (V5)

// Libraries
#include <LiquidCrystal.h>
#include <SPI.h>
#include <SD.h>

// Interface and Objects definitions
LiquidCrystal lcd(8, 9, 4, 5, 6, 7); // Creates a LCD LiquidCrystal object
const int chipSelect = 10; // Assigns the pin for the SPI-SD chip selection
boolean newData = false;
int receivedData; // Receives an integer in ASCII to transform in CHAR to be visualised/printed

void setup()
{
  Serial.begin(9600, SERIAL_7N1); // Includes the scale parameters
  boolean newData = false;
  pinMode(chipSelect, OUTPUT); // Ensures that the SPI-SD selection pin is an output
  lcd.begin(16, 2); // Initialises the interface to the LCD screen
  lcd.clear(); // // Clears the LCD screen and positions the cursor in the upper-left corner
  if (!SD.begin(chipSelect)) // Checks if the SD card is working
  {
    lcd.println("SD failed!");
    delay(2000);
    return;
  }
}

void loop()
{
  recvData();
  saveData();
}

void recvData()
{
  if (Serial.available() > 0)
{
  receivedData = Serial.read();
  newData = true;
}
}

void saveData()
{
  if (newData == true)
  {
    File readings = SD.open("readings.csv", FILE_WRITE); // Opens the file on the SD card
    if (readings) // If the file opened OK, write to it on SD card
    {
      readings.print((char)receivedData); // Saves the data on the SD card
      readings.print(',');
      readings.close(); // Closes the file to save the data on the SD card
    }
    else // If the file didn't open, print an error
    {
      lcd.println("FILE NOT OPEN");
      delay(2000);
      lcd.clear();
    }
    newData = false;  
  }
}

I put a comma to separate the cell. The data seems to have a carriage return from the scale itself after the date and time. I do not know how to manipulate that.

The gap was due by the newData condition that was always TRUE after the first loop. My distraction sorry, now I put it outside the IF condition as you might notice....

I think ATM whith MY knowledge this is the best I could get:

We are getting in there but now I would really need help guys, please.

M.

I think in the other thread there was mention of Serial Input Basics

you could see how to deal with incoming bytes, build a buffer and then parse the line to extract what you want

Some functions will come handy from the standard C libraries stdlib.h and string.h

J-M-L:
well in post #14 there you were showing this type of input


what has changed?

Yes, I thought that was the end of "feasibility" section. I mean, I was able to obtain data from the scale, I thought the formatting was related to the programming section of the forum...

In particular at that stage, There was a "println" command there, my mistake that I solved with the following code.

M.

Give this a try (just typed it in based on your code, so not even sure it compiles - but this gives you an idea of how to build the buffer for one full line. I'm assuming your scale sends an '\n' as a end of message character)

// Code for receiving the data from my analytical scale through RS232 (V5)

// Libraries
#include <LiquidCrystal.h>
#include <SPI.h>
#include <SD.h>

// Interface and Objects definitions
LiquidCrystal lcd(8, 9, 4, 5, 6, 7); // Creates a LCD LiquidCrystal object
const int chipSelect = 10; // Assigns the pin for the SPI-SD chip selection

int receivedData; // Receives an integer in ASCII to transform in CHAR to be visualised/printed
const byte bufferSize = 100; // enough for your max length of a message
char inputBuffer[bufferSize + 1];
byte bufferPosition;

void setup()
{
  Serial.begin(9600, SERIAL_7N1); // Includes the scale parameters

  pinMode(chipSelect, OUTPUT); // Ensures that the SPI-SD selection pin is an output
  lcd.begin(16, 2); // Initialises the interface to the LCD screen
  lcd.clear(); // // Clears the LCD screen and positions the cursor in the upper-left corner
  if (!SD.begin(chipSelect)) // Checks if the SD card is working
  {
    lcd.println("SD failed!");
    delay(2000);
    return;
  }

  bufferPosition = 0;
  inputBuffer[0] = '\0'; // initialize with empty string

}

void recvData()
{
  if (Serial.available() > 0) {
    receivedData = Serial.read();
    if (receivedData != -1) { // valid input, add to our buffer
      if ((receivedData != '\r') && (receivedData != '\n')) { // is it a meaningful input?
        inputBuffer[bufferPosition++] = (char) receivedData;
        inputBuffer[bufferPosition] = '\0'; // terminates that string
        if (bufferPosition >= bufferSize) bufferPosition = bufferSize - 1; // prevent overflow
      } else if (receivedData == '\n') { // we have a full line, save it and reset buffer
        saveData();
        bufferPosition = 0;
        inputBuffer[0] = '\0'; // initialize with empty string
      }
    }
  }
}

void saveData()
{
   // **********************************************************************
   // here the inputBuffer has a full line, you could parse it for content 
   // I'm just saving the full line to the SD card
   // **********************************************************************
  File readings = SD.open("readings.csv", FILE_WRITE); // Opens the file on the SD card
  if (readings) // If the file opened OK, write to it on SD card
  {
    readings.println(inputBuffer); // Saves the data on the SD card with new line
    readings.close(); // Closes the file to save the data on the SD card
  } else { // If the file didn't open, print an error
    lcd.println("FILE NOT OPEN");
    delay(2000);
    lcd.clear();
  }
}


void loop()
{
  recvData();
}

J-M-L:
Give this a try

Thanks a lot for this. I will have another proper read on that topic "serial input basics" and try to "study" your code as well. After that, I will come back here.

Anyway, the output of your code is:

I believe it is a lot more elegant and efficient, so I just need to parse the different digit/letters the way I like, right?

My next step will be to study all this and come back here if you and the other guys are available for help. I need to have it done soon but at the same time I do not want others doing the job for me otherwise, I will not learn anything.

Mario

I believe it is a lot more elegant and efficient, so I just need to parse the different digit/letters the way I like, right?

yes and functions such as those from stdlib.h (for example atoi()) and string.h will help (for example split and parse your buffer using [url=http://www.cplusplus.com/reference/cstring/strtok/?kw=strtok]strtok()[/url])

J-M-L:
yes and functions such as those from stdlib.h (for example atoi()) and string.h will help (for example split and parse your buffer using [url=http://www.cplusplus.com/reference/cstring/strtok/?kw=strtok]strtok()[/url])

I will have a look, just need a bit of time now and I will come back here :slight_smile: Thanks a lot J-M-L!

LordKelvin:
I will have a look, just need a bit of time now and I will come back here :slight_smile: Thanks a lot J-M-L!

with pleasure - always happy to contribute and help out members putting energy into trying and learning.

to help understand my code, it's not rocket science:

I listen to the serial port for an incoming character and if it's a legit characters different than an end of line I just add it into the buffer, making sure I don't overflow. Once i receive the end of line ('\n') character then I save the line to the SD card and reset the buffer, ready for the next line.

looking at the output it seems that the format you get is

[color=blue]Time[/color] [color=red]Date[/color] (end of line)
[color=purple]floating point value[/color] [color=green]G OK[/color]  (end of line)
(empty line)

should not be too difficult to parse.

J-M-L:
with pleasure - always happy to contribute and help out members putting energy into trying and learning.

should not be too difficult to parse.

Please do not kill me :slight_smile: :slight_smile: :slight_smile:

I prepared a code that probably is a mess, but it is my first attempt on this subject.

Please refer to the post number 3 where I explain what output I would like to achieve.

I am trying to get the whole buffer, create a temporary one for parsing, parsing parts of the buffer in several variables (date_month, time_hour, etc.) and then save these variables in different cells in the CSV file.

I put "together" what I understood of your code and of the examples 5 and 2 from that topic on "serial input basics".

// Code for receiving the data from my analytical scale through RS232 with parsing

// Libraries
#include <LiquidCrystal.h>
#include <SPI.h>
#include <SD.h>

// Interface and Objects definitions
LiquidCrystal lcd(8, 9, 4, 5, 6, 7); // Creates a LCD LiquidCrystal object
const int chipSelect = 10; // Assigns the pin for the SPI-SD chip selection

const byte bufferSize = 100; // enough for your max length of a message
static byte bufferPosition = 0;
char inputBuffer[bufferSize]; // my array of the data from the scale
char tempBuffer[bufferSize]; // temporary array for use when parsing

// variables to hold the parsed data
char date_month[bufferSize] = {0};
char date_day[bufferSize] = {0};
char date_year[bufferSize] = {0};
char time_hour[bufferSize] = {0};
char time_min[bufferSize] = {0};
char time_sec[bufferSize] = {0};
float reading = 0.0;

boolean newData = false;

//============

void setup()
{
  Serial.begin(9600, SERIAL_7N1); // Includes the scale parameters

  pinMode(chipSelect, OUTPUT); // Ensures that the SPI-SD selection pin is an output
  lcd.begin(16, 2); // Initialises the interface to the LCD screen
  lcd.clear(); // // Clears the LCD screen and positions the cursor in the upper-left corner
  if (!SD.begin(chipSelect)) // Checks if the SD card is working
  {
    lcd.println("SD failed!");
    delay(2000);
    return;
  }

  bufferPosition = 0;
  inputBuffer[0] = '\0'; // initialize with empty string

}

//============

void loop() {
    recvWithEndMarker();
    if (newData == true) {
        strcpy(tempBuffer, inputBuffer);
            // this temporary copy is necessary to protect the original data
            //   because strtok() used in parseData() replaces the commas with \0
        parseData();
        saveData();
        newData = false;
    }
}

//============

void recvWithEndMarker() {
    char endMarker = '\n';
    char receivedData;
   
    while (Serial.available() > 0 && newData == false) {
        receivedData = Serial.read();

        if (receivedData != endMarker) {
            inputBuffer[bufferPosition] = receivedData;
            bufferPosition++;
            if (bufferPosition >= bufferSize) {
                bufferPosition = bufferSize - 1;
            }
        }
        else {
            inputBuffer[bufferPosition] = '\0'; // terminate the string
            bufferPosition = 0;
            newData = true;
        }
    }
}


//============


void parseData() {      // split the data into its parts

    char * strtokIndx; // this is used by strtok() as an index

    strtokIndx = strtok(tempBuffer,","); // get the first bit
    strcpy(date_month, strtokIndx); // copy it to month variable

    strtokIndx = strtok(tempBuffer,","); // carry on...
    strcpy(date_month, strtokIndx); // carry on copy this byte in the same variable

    strtokIndx = strtok(tempBuffer,",");
    strcpy(date_day, strtokIndx); // copy it to month variable
    
    strtokIndx = strtok(tempBuffer,",");
    strcpy(date_day, strtokIndx); // carry on copy this byte in the same variable, etc.

    strtokIndx = strtok(tempBuffer,",");
    strcpy(date_year, strtokIndx);

    strtokIndx = strtok(tempBuffer,",");
    strcpy(date_year, strtokIndx);
    
 
    strtokIndx = strtok(NULL, ","); // I do the same with the time...
    strcpy(time_hour, strtokIndx);

    strtokIndx = strtok(NULL, ",");
    strcpy(time_hour, strtokIndx);

    strtokIndx = strtok(NULL, ",");
    strcpy(time_min, strtokIndx);

    strtokIndx = strtok(NULL, ",");
    strcpy(time_min, strtokIndx);

    strtokIndx = strtok(NULL, ",");
    strcpy(time_sec, strtokIndx);

    strtokIndx = strtok(NULL, ",");
    strcpy(time_sec, strtokIndx);
    

    strtokIndx = strtok(NULL, ",");
    reading = atof(strtokIndx); // convert this part to a float for the reading cell

}

//============

void saveData()
{
   // **********************************************************************
   // here the inputBuffer has a full line, you could parse it for content
   // I'm just saving the full line to the SD card
   // **********************************************************************
  File readings = SD.open("readings.csv", FILE_WRITE); // Opens the file on the SD card
  if (readings) // If the file opened OK, write to it on SD card
  {
    readings.print(date_month);
    readings.print(',');
    readings.print(date_day);
    readings.print(',');
    readings.print(date_year);
    readings.print(',');
    readings.print(time_hour);
    readings.print(',');    
    readings.print(time_min);
    readings.print(',');
    readings.print(time_sec);
    readings.print(',');
    readings.println(reading);
     
    readings.close(); // Closes the file to save the data on the SD card
  } else { // If the file didn't open, print an error
    lcd.println("FILE NOT OPEN");
    delay(2000);
    lcd.clear();
  }
}

Hope it is not too horrible...may you please troubleshoot or tell me where to improve it please?

It is complying anyway.

I will try now to see the output but I am concerned because at the end of compile process it says low on memory (84%)...

Thanks a lot.

M.

I'll give it a go when I get a bit of time.

for the memory, I sized the buffer at 100 chars and you are now creating two apparently, that's a lot of memory, you probably don't need that. --> look at the max length of your input line

Yes please, thanks a lot :slight_smile:

I put 50 instead of 100, now it is OK!

wow - i had not seen those as well..

// variables to hold the parsed data
char date_month[bufferSize] = {0};
char date_day[bufferSize] = {0};
char date_year[bufferSize] = {0};
char time_hour[bufferSize] = {0};
char time_min[bufferSize] = {0};
char time_sec[bufferSize] = {0};

come on, don't "piss away" :slight_smile: memory! a month will be 2 digits, add a '\0' at the end and that's 3... no need to save 50 bytes for that, right? and you could probably even read that as a byte for month number...

I think you will have an issue with your code as the info you get is over 2 lines. so you can't parse ever line the same way. you should build a buffer until you get the word OK or the empty line.