SD card datalogger

Projects Networking, Protocols, and Devices
1

I have build a weather station using an arduino and an ethernetshield as webserver. With my processing sketch (client) I can log in the arduino and get it's life sensor data printed on my processing window when I send any keystroke.

I have also let somebody 500km away from me succesfully read out my sensor data using my home IP address.

I am also logging the data every hour on an SD-card. Via processing, I can send any keystroke to the arduino. The keystrokes gets printed on the serial monitor.

if I hit the 'a' key the arduino also turns on a led and prints "YOLO" on the serial monitor, and if I hit the 'u' key, the led goes off and "SWAG" (was tired of hello world ;)? gets printed on the serial monitor.

If I hit the 'd' key, I order the arduino to dump it's datalog over the LAN to my processing window, this works fine. code:

void dumpData()
{
    File dataFile = SD.open("logboek.txt", FILE_READ);

    while (dataFile.available()) {
      client.write(dataFile.read());
    }

    dataFile.close();
    client.write("\r\n");
  }

I post a screenshot with the serial monitor and the proccesing window so you can see how it functions.

For now, the end goal is to request the data of the last 24 hours on command and transform this data into 6 coloured lines in a processing graph. Including time stamps, a ds1307 RTC on the arduino is tracking the time.

The datalog grows every hour but I am only interested in the last 24 hours.

I was wondering what the best way would be to separate the requested bytes from the rest.
I figured that it would be easier to tackle this problem in the arduino than in processing.

When the datalog gets dumped, every last character get's ran through the arduino code.
I figured that it would be wise to let the arduino process the datalog file, before sending anything to processing, and look for a matching time/date stamp to compare it with the current time of the RTC minus 1 day.

So when I give the command for a datadump at 23:40 14-08-15. The arduino runs through the datalog until it finds (data gets logged at the hour so minutes must be 00) 23:00 13-08-15

I am now thinking to use a switch-case statement to look for the right time/date stamp. As of now I only devised this:

byte check = 0;

while (dataFile.available()) {
 char c = dataFile.read();

  // this is code to look for 23:00 13-08-15 this would be stored as 231308 on the SD card (minutes are always 0, thus irrelevant and I dont care about the year for now)
  // so we are looking for the character sequence  
  //               231308
 
  switch (check) {  //        (2) 
    case 0: if(c == (char)hour_tenth) check++; // so if c matches the 2 of 23:00 hours go to the next case
            break;
 //                           (3)
    case 1: if(c == (char)hour_unit) check++; // if c matches the 3 of 23:00 hours go to the next case
            else check = 0; break;            // if c doesnt match the 3, we have the wrong time, back to case 0
//                           (1)
    case 2: if(c == (char)day_tenth) check++;
            else check = 0; break;
//                           (3)
    case 3: if(c == (char)day_unit) check++;
            else check = 0; break; 
//                            (0)
    case 4: if(c == (char)month_tenth) check++;
            else check = 0; break;
//                           (8)
    case 5: if(c == (char)day_unit) check++;
            else check = 0; break;  

//
    case 6: send_remaining_data_to_processing_function(); // and this will send the remaining data of the SD overthe LAN to processing window
            check = 0; break;                          // reset check to 0 after the last data has been send                                    
  }
}

This was about the only thing I could come up with.

Is this a viable approach? (if so, is it smart to do?), or are there other/better solutions or functions I dont know about?

aaa (2).png
aaa (2).png1440×900 130 KB

2

For now, the end goal is to request the data of the last 24 hours on command

I don't know anything about Processing but it all looks very convoluted. You might consider using the date as a filename and changing the name at midnight. You then send Arduino MMDD and get the 24 hours you need. Something like in here:

http://homepages.ihug.com.au/~npyner/Arduino/BT_prac_application.ino

3

what I mean, if I request the data at 13:06 15-08-15, I want the data of the last 24 hours prior to 13:00 15-08-15. So I get the data from 13:00 14-08-15 to 13:00 15-08-15.

But if I do it at 15:35 I want the data from 15:00 14-08-15 to 15:00 15-08-15.

I have it working now. That switch case statement was a bit hard. At first I couldnt make the comparison at the if statement. There was a lot of confusion with serial.write, serial.print, integers and characters. but +48 solved the entire problem.

void dumpData()
{
  datum--;                     // this makes sure we are looking for time stamp of 1 day ago
  datum_tiental = (char)datum / 10;     // splits date into the tenths
  datum_eenheid = (char)datum % 10;  // and units
  datum++;

  Serial.print(uur_tiental); Serial.print(uur_eenheid); Serial.print(datum_tiental);
  Serial.print(datum_eenheid); Serial.print(maand_tiental); Serial.println(maand_eenheid); // debugging

  byte check = 0;
  byte c;   // could also be char C doesnt matter
  File dataFile = SD.open("logboek.txt", FILE_READ);
  while (dataFile.available()) {
    c = dataFile.read();

    switch (check) {
      case 0: if (c == uur_tiental + 48) check++; 
        break;

      case 1: if (c == uur_eenheid + 48) check++; 
        else check = 0; break;       

      case 2: if (c == datum_tiental + 48) check++;  // I tried (char)datum_tiental at first, but that did not work
        else check = 0; break;         // I couldnt get a comparison, but after I added all the + 48, it worked

      case 3: if (c == datum_eenheid + 48) check++;
        else check = 0; break;

      case 4: if (c == maand_tiental + 48) check++;
        else check = 0; break;

      case 5: if (c == maand_eenheid + 48) check++;
        else check = 0; break;


      case 6:
        client.print((String)uur_tiental);      // dont forget to also send the time stamp where we were looking for. This time stamp belong to the first sensor data sequence
        client.print((String)uur_eenheid);
        client.print((String)datum_tiental);
        client.print((String)datum_eenheid);
        client.print((String)maand_tiental);
        client.print((String)maand_eenheid);

        while (dataFile.available()) {
          client.write(dataFile.read());       // this will now send only the data/time of the past 24 hours of NOW
        }
        check = 0;
        break;
    }
  }
  dataFile.close();
  client.write("\r\n");
}

so now when processing request for data, the arduino will scan the entire datalog for the specific time-stamp which is always the current time - 24 hours.

There is one tiny little flaw in it. I cannot ever request data on every 1st day of the month. If it is 15:05 01-09-15 I would let the arduino look for current time minus 1 day aka 15:00 00-08-15. And this day/time does not exist. And I really dont want to program an entire calander myself.
I can use the RTC to memorize the date of the previous day so I know if it was a 30th, 31st or a 28th. Than I can still look for the proper time stamp. I will add this feature when I have all my sensors working.

4

bask185:
I want the data of the last 24 hours prior to 13:00 15-08-15.

OK, I've got it - a moving target!

5

I thought it would be cool if I could see the 6 graphs with my sensor data gets refreshed every hour.

I can always log data every 2 minutes so I can refresh my graphs every 2 minutes.

With a 64GB card I can log like over 2.6 billion times. I can than log data for the next 281.000 years :smiley:
I love technology.