Pages: [1]   Go Down
Author Topic: Help on speeding up datalogging into an SD card  (Read 2851 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Newbie
*
Karma: 0
Posts: 12
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi,
I'm working on a project in which I am trying to log data values that have been obtained from sensors into an SD card.
I am using an SD card shield v4.0 by Seeed studio connected to an Arduino Mega. Currently, I'm using the EM-406A gps module for the project and I have been initially writing the data values obtained from the gps simply by opening the file within the loop and logging the data as it goes.

However, I have read that logging data in this manner would operate would slow (and it definitely did).  So I read that logging "blocks" of data can improve the speed, but I'm not really sure how to do that. As of now, I'm storing the gps values (latitude, longitude, speed, altitude) as arrays and as the array reaches a certain size, I open the file within the SD card and write the values in using a for loop. I set the array size limit as a number as 128 and once the array reaches 128 values, it writes each data into the SD card file. But for some reason, this only happens once. It would reach 128 values and write it into the SD card, but the process seems to stop after that time.

Code:
int i = 0;
float lat[128], lon[128];
float alt[128];
float cours[128];
float vel[128];
unsigned long times[128];
// This is the main loop of the code. All it does is check for data on
// the RX pin of the ardiuno, makes sure the data is valid NMEA sentences,
// then jumps to the getgps() function.
void loop()
{
  time = millis();
  while(uart_gps.available())     // While there is data on the RX pin...
  {
      int c = uart_gps.read();    // load the data into a variable...
      if(gps.encode(c))      // if there is a new valid sentence...
      {
        getgps(gps);         // then grab the data.
      }
  }
}

// The getgps function will get and print the values we want.
void getgps(TinyGPS &gps)
{
  // To get all of the data into varialbes that you can use in your code,
  // all you need to do is define variables and query the object for the
  // data. To see the complete list of functions see keywords.txt file in
  // the TinyGPS and NewSoftSerial libs.
 
  // Define the variables that will be used
  float latitude, longitude;
  // Then call this function
  gps.f_get_position(&latitude, &longitude);
  // You can now print variables latitude and longitude
  Serial.print("Lat/Long: ");
  Serial.print(latitude,5);
  Serial.print(", ");
  Serial.println(longitude,5);
 
  // Same goes for date and time
  int year;
  byte month, day, hour, minute, second, hundredths;
  gps.crack_datetime(&year,&month,&day,&hour,&minute,&second,&hundredths);
  // Print data and time
  Serial.print("Date: "); Serial.print(month, DEC); Serial.print("/");
  Serial.print(day, DEC); Serial.print("/"); Serial.print(year);
  Serial.print("  Time: "); Serial.print(hour, DEC); Serial.print(":");
  Serial.print(minute, DEC); Serial.print(":"); Serial.print(second, DEC);
  Serial.print("."); Serial.println(hundredths, DEC);
  //Since month, day, hour, minute, second, and hundr
 
  // Here you can print the altitude and course values directly since
  // there is only one value for the function
  Serial.print("Altitude (meters): "); Serial.println(gps.f_altitude()); 
  // Same goes for course
  Serial.print("Course (degrees): "); Serial.println(gps.f_course());
  // And same goes for speed
  Serial.print("Speed(kmph): "); Serial.println(gps.f_speed_kmph());

  lat[i] = latitude,5;
  lon[i] = longitude,5;
  alt[i] = gps.f_altitude();
  cours[i] = gps.f_course();
  vel[i] = gps.f_speed_kmph();
  times[i] = time;
  if (i == 128)
  {
    Serial.println();
    Serial.println(millis(),DEC);
   
    for (i = 0; i < 128; i++)
    {
      File dataFile = SD.open("gps.csv", FILE_WRITE);
      if (dataFile)
      {
        dataFile.print(times[i]);
        dataFile.print(",");
        dataFile.print(lat[i],5);
        dataFile.print(",");
        dataFile.print(lon[i],5);
        dataFile.print(",");
        dataFile.print(alt[i]);
        dataFile.print(",");
        dataFile.print(cours[i]);
        dataFile.print(",");
        dataFile.println(vel[i]);
        dataFile.close();
      }
      else
      {
        Serial.println("Couldn't log GPS data");
      }
    }
    Serial.println(millis(),DEC);
    Serial.println();
    i = 0;
  }
  else
  {
    i++;
  }

  // Here you can print statistics on the sentences.
  unsigned long chars;
  unsigned short sentences, failed_checksum;
  gps.stats(&chars, &sentences, &failed_checksum);
  //Serial.print("Failed Checksums: ");Serial.print(failed_checksum);
  //Serial.println(); Serial.println();
}

I don't know too much about Arduino since I just started using it about a month or two ago. But from reading through the forums it seems that writing data as "blocks" into the SD card is the fastest way. Could anyone explain more into how to write the data as blocks? Is that not just making it an array and then logging the array into the SD card?

Any help is appreciated!
Logged

Global Moderator
Boston area, metrowest
Online Online
Brattain Member
*****
Karma: 524
Posts: 26486
Author of "Arduino for Teens". Available for Design & Build services. Now with Unlimited Eagle board sizes!
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Checkout fat16lib's sdfat.h library. He's got datalogging running really fast.
Logged

Designing & building electrical circuits for over 25 years. Check out the ATMega1284P based Bobuino and other '328P & '1284P creations & offerings at  www.crossroadsfencing.com/BobuinoRev17.
Arduino for Teens available at Amazon.com.

Offline Offline
Newbie
*
Karma: 0
Posts: 12
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Thanks,

I have checked out fat16lib's sdfat library and also his fastlogger. However, I'm having a little trouble understanding the code since  it doesn't seem to be writing data into the SD card the same way the SD library does. I'm familiar with writing the data using commands such as:

File dataFile = SD.open("asdf.csv", FILE_WRITE)

and then just printing the data into it.

How do I implement the SdFat library to initialize writing into the file created for logging?
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 12
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Sorry, never mind, I managed to figure it out through one of the examples in the library. The SD data logging appears to writing data faster.

But I still can't manage to figure out why the whole sketch stops running after it logs the first time. I wrote the code to continue logging data as an array until it reached a size of 50 (just for testing, probably will be changed later) and then write it into the SD card. After the first set of 50 data values are written into the SD card, it seems as if it ends the process and nothing happens after. Is there something that I'm missing?
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 601
Posts: 48543
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
But I still can't manage to figure out why the whole sketch stops running after it logs the first time. I wrote the code to continue logging data as an array until it reached a size of 50 (just for testing, probably will be changed later) and then write it into the SD card. After the first set of 50 data values are written into the SD card, it seems as if it ends the process and nothing happens after. Is there something that I'm missing?
Yes, there is. Look back through this thread. Where is your code? What library is that using/ What library are you now using?
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 12
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Sorry, attached is the sketch for the GPS that I am working on. I initially used Sd.h library for writing into the SD card but now I changed it to the SdFat library. Using either library still has the same problem though, it would store the data into an array up to 50 values, write into the card and then stop.

* GPSworking.ino (6.77 KB - downloaded 13 times.)
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 601
Posts: 48543
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Code:
  String header = "Time, Latitude, Longitude, Altitude, Course, Speed [kmph]";
  dataFile.println(header);
What a waste of resources.
Code:
  dataFile.println("Time, Latitude, Longitude, Altitude, Course, Speed [kmph]");
Same result. No waste.

Code:
  dataFile.println(", , , ,");
6 values implied by the header. 5 values implied by this line. This is written before the header. Why?

Code:
    for (i = 0; i < 50; i++)
    {
      if (dataFile.open("gps.csv", O_RDWR | O_CREAT | O_AT_END))
      {
Why are you opening and closing the file 50 times?

Do you still get serial output after the program quits logging data?
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 12
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Yeah I've simply been following a simple tutorial on basic SD card writing and that's how it's been demonstrated but I was planning on changing it by taking out the "header" variable and printing the header directly.

Oh I didn't realize I was reopening and closing the file within the for loop, I'll make sure I take that out of the loop. How much does that actually slow down the write? After the program quits logging data, I don't get a serial output anymore. Could the arduino be running out of memory because the data values are being stored within an array? I realized some of my other sketches ran out of memory as it stored more data values as an array, which resulted in the serial output stopping. Because of this, I don't feel like this is the efficient way to log data. Logging data as it is obtained from the sensors appears to slow down the serial output but storing them as an array leads to the Arduino running out of memory. I've looked through other posts of people mentioning methods of sending chunks of bytes but I still don't have a clear understanding on how to do that.
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 601
Posts: 48543
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Writing to an SD card is already buffered. I don't see the value in storing data in arrays, and then writing the data to the buffer (and on to the SD card when the buffer gets full). Write when you have data to write.

It is possible that you are running out of memory, though that seems unlikely on a Mega. 6 arrays with 50 4-byte elements is 1200 bytes. The Mega has 8000 bytes of SRAM.

Use this: http://playground.arduino.cc/Code/AvailableMemory
to know for certain.
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 12
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Thanks, I'll make sure to check that out!

If I do write in the data when it is available it slows down the serial output by a lot. From what I've been reading, it seems like slowing down is expected but the values obtained from the sensors are significantly slower. Obtaining the values from the sensor slows down to a point where I can't really analyze the data because there would be so much more missing data values in between each logged one.

I wrote the code as shown below but I don't know how else to write it without significantly slowing down the serial output
Code:
if (dataFile4.open("gps.csv", O_RDWR | O_CREAT | O_AT_END))
      {
        dataFile4.print(timer);
        dataFile4.print(",");
        dataFile4.print(latitude,5);
        dataFile4.print(",");
        dataFile4.print(longitude,5);
        dataFile4.print(",");
        dataFile4.print(gps.f_altitude());
        dataFile4.print(",");
        dataFile4.print(gps.f_course());
        dataFile4.print(",");
        dataFile4.println(gps.f_speed_kmph());
      }

What would be a more efficient way to log data as it becomes available?
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 601
Posts: 48543
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
From what I've been reading, it seems like slowing down is expected but the values obtained from the sensors are significantly slower.
You are only logging GPS data. So, I have no clue what sensors you are talking about. Not do I understand the relationship between logging GPS data and writing to the serial port.

You are only logging 5 values from the GPS plus something called timer that probably isn't a timer.

The GPS (typically) only sends a sentence once a second. You should be able to read that sentence, parse it, log it, and still have time for a cup of coffee before the next sentence starts to come in. A small cup, anyway.

Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 601
Posts: 48543
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I looked at your code again. I see that you are using SoftwareSerial to read the GPS data. When you have 4 hardware serial ports, why? You clearly are letting the other three hardware serial ports do nothing, while SoftwareSerial is pressed into service. For a machine that only has one hardware serial port, that is a reasonable thing to do. For a machine with three idle hardware serial ports, it isn't.

You are streaming a LOT of data to the serial port. Is that ALL really necessary?
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 12
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The other sensors that I mentioned are a gyroscope, pressure transducer, altimeter and a sonar that I have in a different sketch but I wanted to focus on speeding up the GPS' datalogging before getting into all the sensors data logging done. If I split the amount of data going through different serial ports, would I need some sort of external serial monitor to display what's outputting to the other ports? If so, would there be an alternative way to view the outputs?
Logged

Pages: [1]   Go Up
Jump to: