Pages: [1]   Go Down
Author Topic: SD card as a buffer (using sdfatlib beta)  (Read 1975 times)
0 Members and 1 Guest are viewing this topic.
São Paulo/SP/Brazil
Offline Offline
Sr. Member
****
Karma: 2
Posts: 305
Brazilian Arduino Team
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi,

My weather station is working but I'm having issues when my internet is down. So I added an SD card to it.

I start by trying to send and if not possible I save the data to a file. When internet is back I flush SD data. It's working fine but when file reaches some size(>20 hours of 1 minute data), it last more than one minute to send one minute of data. My data is 26 bytes each minute and I'm already saving in binary instead ascii data to save space.

The "algorithm" I'm using is to send first line, erase first line(I'm opening a new file, streaming main file to temporary, remove and rename back). I don't know a better way to do this so it's a question: "Is it possible to improve this using sdfatlib beta?"

I'm trying to make additional files using some files like 0001.txt, 0002.txt with 1 hour of data each. But I'm finding a hard way dealing with ls() method. Can you post some example on dealing with this like ls, open first file, close, another file and so on?

I'm using millis() to time things. On first 5 seconds I get, try to send(timeout after 5 seconds) and if not possible, save to SD. The another 55 seconds I try to "flush" SD data. If it detects internet is down it try to send once each minute. Internet is down equals to not receive CHECKSUM from a webservice I made.

Thanks,

Wagner Sartori Junior
Logged

Seattle, WA USA
Online Online
Brattain Member
*****
Karma: 653
Posts: 50924
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
My data is 26 bytes each minute
So, the Arduino should be able to save 10 minutes worth of data, while uploading the contents of the SD card (that was saved while the internet was not available).

Surely that's enough time to copy the whole file to the server. There would then be no reason to copy/remove/copy/rename just to remove one line from the file.

Quote
I'm trying to make additional files using some files like 0001.txt, 0002.txt with 1 hour of data each. But I'm finding a hard way dealing with ls() method. Can you post some example on dealing with this like ls, open first file, close, another file and so on?
Post a link to the library you are using. Post the code that you wrote to test the ls() method.

You could try to open hourly file. If it succeeds, that file already exists. Close it, and try the next number. When you get a failure, that file doesn't exist, so, create it and write to it.
Logged

São Paulo/SP/Brazil
Offline Offline
Sr. Member
****
Karma: 2
Posts: 305
Brazilian Arduino Team
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

I'm trying to be safe. It's a good idea(I don't think about it before) try to send the whole file and make a simple checksum on file content to be sure the whole file is sent correctly.

I'm using http://code.google.com/p/sdfatlib/ but the beta available on http://code.google.com/p/beta-lib/downloads/list. The working code(using only one file) is available on https://bitbucket.org/trunet/weatherstation. This code is not using binary and writing everything on ASCII mode to easy debug.

The ls code is a simple root.ls(), I want to iterate on this method but I think it's not possible without hacking the library(not sure about this).
Logged

Seattle, WA USA
Online Online
Brattain Member
*****
Karma: 653
Posts: 50924
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
The ls code is a simple root.ls(), I want to iterate on this method but I think it's not possible without hacking the library(not sure about this).
I'm not sure what you mean by "iterate on this method". The method outputs a list of files in the current directory. How can you iterate on that?
Logged

São Paulo/SP/Brazil
Offline Offline
Sr. Member
****
Karma: 2
Posts: 305
Brazilian Arduino Team
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

I know I can't iterate. I just want a way for doing that.

Maybe getting the list and split on an array but I have no idea on how to do that.
Logged

0
Offline Offline
Edison Member
*
Karma: 67
Posts: 1673
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset


I don't think ls() is what you want. 

Maybe openNext() would be better.  It will allow you to iterate through all files in the root directory.

Here is a example from the newest version of SdFat, the 20110902 version in http://code.google.com/p/sdfatlib/.

It opens all files in the root directory and prints their names.  You could check the name and then read the file if the name matches some pattern.
Code:
/*
 * Open all files in the root dir and print their filename
 */
#include <SdFat.h>

// SD chip select pin
const uint8_t chipSelect = SS_PIN;

// file system object
SdFat sd;

SdFile file;

// define a serial output stream
ArduinoOutStream cout(Serial);
//------------------------------------------------------------------------------
void setup() {
  char name[13];

  Serial.begin(9600);

  // initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  if (!sd.init(SPI_HALF_SPEED, chipSelect)) sd.initErrorHalt();

  // open next file in root.  The volume working directory, vwd, is root
  while (file.openNext(sd.vwd(), O_READ)) {
    file.getFilename(name);
    cout << name << endl;
    file.close();
  }
  cout << "Done" << endl;
}
//------------------------------------------------------------------------------
void loop() {}
Logged

São Paulo/SP/Brazil
Offline Offline
Sr. Member
****
Karma: 2
Posts: 305
Brazilian Arduino Team
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Ow,

Thank you very much. This is what I was looking for! smiley-wink

Best regards,
« Last Edit: September 08, 2011, 04:29:04 pm by Wagner Sartori Junior » Logged

São Paulo/SP/Brazil
Offline Offline
Sr. Member
****
Karma: 2
Posts: 305
Brazilian Arduino Team
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Do you know why this code is looping forever if my file only has 256K? I'm supposing that it's not detecting EOF. I'm saving files in binary mode(saving chunks of bytes) though I can't use readline().

Code:
int checksum = 0;
byte c;
while ((c = sentFile.read()) >= 0) {
  checksum ^= c;
  Serial.write(c);
}
Logged

São Paulo/SP/Brazil
Offline Offline
Sr. Member
****
Karma: 2
Posts: 305
Brazilian Arduino Team
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

I uploaded the whole code to http://pastebin.com/ZXvZMv7t. Take a look on the sendFile() function.

I'm receiving more than 256Kb on the other side indicating the looping is happening forever.
Logged

São Paulo/SP/Brazil
Offline Offline
Sr. Member
****
Karma: 2
Posts: 305
Brazilian Arduino Team
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

As I really don't know why the hell my while does not work(it never finds the EOF and loops forever), the code below is working flawless.

Code:
long i = 0;
byte c, checksum;
while ((c = sentFile.read()) >= 0) {
      if (i > sentFile.fileSize())
        break;
      i++;
      checksum += c;
      Serial.write(c);
} // while ((c = sentFile.read()) >= 0) {
checksum = 0xff - checksum;
Serial.write(checksum);
Logged

Seattle, WA USA
Online Online
Brattain Member
*****
Karma: 653
Posts: 50924
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
As I really don't know why the hell my while does not work
For some reason, I read this as "my wife does not", and I was thinking you have serious communications issues. Which is funny since the whole thread is about communication failures.
Logged

Greenville, IL
Offline Offline
Edison Member
*
Karma: 15
Posts: 1330
Warning Novice on board! 0 to 1 chance of errors!
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Could this be it?
Code:
if (i > sentFile.fileSize())

Change to:
Code:
if (i >= sentFile.fileSize())
Logged


São Paulo/SP/Brazil
Offline Offline
Sr. Member
****
Karma: 2
Posts: 305
Brazilian Arduino Team
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

For some reason, I read this as "my wife does not", and I was thinking you have serious communications issues. Which is funny since the whole thread is about communication failures.
hahaha that was really funny! smiley-grin

Could this be it?
Code:
if (i > sentFile.fileSize())
Change to:
Code:
if (i >= sentFile.fileSize())
this code is working already... I want it to work without my DIY byte counter. the while loop has to work directly as the example files on library:
Code:
while ((c = sentFile.read()) >= 0) {
// do something
}
Logged

Pages: [1]   Go Up
Jump to: