SD card read/write with Arduino

Hi Charlie,

Thanks for clarifying that for me, I think I get it now.

I think I can probably work with the 512 byte write each time... My program is monitoring a temperature reading and the voltages of a battery and a solar panel, in an enclosure housing a camera to take a few photos each day, producing 10 bytes of data per minute. I could probably drop this down to 8 bytes if writing 10 bytes/min causes problems with the 512 byte sector size, but I'd prefer not to.

Here's my code (over 2 posts):

#include <Wire.h>
#include <math.h>
#include <DateTime.h>
#include <avr/pgmspace.h>
#include <wprogram.h>
#include <microfat2.h>
#include <mmc.h>

#define ThermistorPIN 0            // input read pin for LM335 is Analog Pin 0
float temperature = 0;                 // variable which will be calculated in process
long val=0;                       // variable to store the value coming from the thermistor
float temp;

unsigned long sector = 0;
unsigned long byteSize;

unsigned long WriteAddress = 0;    // initial address for writing to the SD card.
byte value;                        // will be used...
int incomingByte = 0;                 // ...for incoming serial data
byte tempBytes[10];                // for sending data to the SD card
byte sector_buffer[512];           // for sending AND retrieving data to/from the SD card

int hour;
int minute;
int second;
int month;
int day_of_week;
int day;
int year;

char* dow[7] = {"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"};

float Thermistor(int RawADC) {  //  Inputs ADC Value from Thermistor and outputs Temperature in Celsius
 long Resistance;
 float Temp;                                      // Dual-Purpose variable to save space.
 Resistance=((10240000/RawADC) - 10000);          // Assuming a 10k Thermistor.  Calculation is actually: Resistance = (1024/ADC)
 Temp = log(Resistance);                          // Saving the Log(resistance) so not to calculate it 4 times later. "Temp" means "Temporary" on this line.
 Temp = 1 / (0.001129148 + (0.000234125 * Temp) + (0.0000000876741 * Temp * Temp * Temp));   // Now it means both "Temporary" and "Temperature"
 Temp = Temp - 273.15;  // Convert Kelvin to Celsius                                            Now it only means "Temperature"
 return Temp;           // Return the Temperature
}

void print_prompt_P(const char *data) {         // print prompts and messages from program flash to save SRAM
   while (pgm_read_byte(data) != 0x00)
     Serial.print(pgm_read_byte(data++));
}

void TakePhoto() {
  digitalWrite(3,HIGH); // first, switch on the fans to blow away dust or bugs...
  delay(10000);
  digitalWrite(3,LOW);
  delay(2000);
  digitalWrite(9,HIGH); // then send power to the camera
  delay(500);           // time for current to get to camera - not sure if this is needed...
  digitalWrite(6,HIGH); // trigger the solenoid for 100 msec
  delay(100);
  digitalWrite(6,LOW);
  delay(15000);         // wait 15 seconds for the camera to take a photo then shutdown
  digitalWrite(9,LOW);  // stop sending power to the camera.
}

void DataDump () {
  int DaysToDump = WriteAddress;
  Serial.println("");
  print_prompt_P(PSTR("          Data dump initiated: dumping "));
  Serial.print(DaysToDump/10);
  print_prompt_P(PSTR(" days of data.\n"));
    print_prompt_P(PSTR("Date,Time,Temp,VBatt,VPanel\n"));
      for(int p=0;p<DaysToDump;p=p+10) {
        sector_buffer[p]=digitalRead(13);
        Serial.print(sector_buffer[p]);

        Serial.print(analogRead(0));  // this bit dumps the date in 'dd/mm/yy' format...
          print_prompt_P(PSTR("/"));
//          ++f;
          Serial.print(analogRead(0));
          print_prompt_P(PSTR("/"));
          delay(50);
          Serial.print(analogRead(0));
          print_prompt_P(PSTR(","));
          Serial.print(analogRead(0));  // this bit prints the time, as ',HH:MM,'...
          print_prompt_P(PSTR(":"));
//          ++f;
          Serial.print(analogRead(0));
          print_prompt_P(PSTR(","));
//          ++f;
          delay(50);
          Serial.print(analogRead(0));  // this bit prints the temperature...
          print_prompt_P(PSTR(","));
//          ++f;
          delay(50);
          Serial.print(analogRead(0));  // this bit prints the Battery Voltage in 'VV.vv,' format...
          print_prompt_P(PSTR("."));
//          ++f;
          Serial.print(analogRead(0));
          print_prompt_P(PSTR(","));
//          ++f;
          delay(50);
          Serial.print(analogRead(0));  // this bit prints the Solar Panel Voltage in 'VV.vv,' format...
          print_prompt_P(PSTR("."));
//          ++f;
          Serial.print(analogRead(0));
//          ++f;
          delay(50);
        Serial.println();
      }
  WriteAddress = 0;
}

void error(const char* s)
{
  print_prompt_P(PSTR("Error: "));
  print_prompt_P(s);
  print_prompt_P(PSTR("<press reset>\n"));
  for( /* ever */ ; ; ) 
  {
    digitalWrite(13, (millis() / 250) & 1);
  }
}

void SDWrite() {
  // Pass in the sector-sized buffer we'll be using ourselves later.
  // uFat doesn't own it, it just needs to use it temporarily.
  // We also pass in the address of a function that is used to read
  // sectors from our device.
  //  
  if (!microfat2::initialize(sector_buffer, &mmc::readSectors))
  {
  error(PSTR("uFat init failed.\n"));
  }
  prog_char string_0[] PROGMEM = "DATA    BIN";

  if (microfat2::locateFileStart(PSTR("DATA    BIN"), sector, byteSize)) // check if the target file exists
  {
    if (byteSize >= 512)                                                 // check it's bigger than 512 bytes
    {
      if (RES_OK == mmc::readSectors(sector_buffer, sector, 1))          // checks a single-SECTOR read works OK
      {
          print_prompt_P(PSTR("Data read test SUCCESSFUL.\n"));
          delay(500);
        if (RES_OK == mmc::writeSectors(sector_buffer, sector, 1))       // checks a single-SECTOR write works OK
        {
          print_prompt_P(PSTR("Data write test SUCCESSFUL.\n"));
          delay(500);
        }
        else
        {
          print_prompt_P(PSTR("Failed to write updated data."));         // if the write fails, end here...?
        }
      }
      else
      {
        print_prompt_P(PSTR("Failed to read data.bin."));
      }
    }
    else
    {
      error(PSTR("Found data.bin, but it's too small."));
    }
  }
  else
  {
    print_prompt_P(PSTR("data.bin not present on card."));
  }
  Serial.println("Initialisation COMPLETE!!");
}

void setup() {

 pinMode(3,OUTPUT);       // FANS control - YELLOW wire
 digitalWrite(3,LOW);
 pinMode(6,OUTPUT);       // SOLENOID power control - WHITE wire
 digitalWrite(6,LOW);
 pinMode(9,OUTPUT);       // CAMERA power control - BLUE wire
 digitalWrite(9,LOW);
// pinMode(13,OUTPUT);
 Serial.begin(115200);
 Wire.begin();

// The rest of this setup() procedure is checking that the SD card is present & working.

  while(mmc::initialize() != RES_OK)
  {
    print_prompt_P(PSTR("SD init failed. retrying...\n"));
    delay(100);
  }
  SDWrite;
}

void loop() {

 // Below required to reset the register address to 0.
 Wire.beginTransmission(104); // transmit to device #104, the ds 1307
 Wire.send(0x00);
 Wire.endTransmission();      // stop transmitting

 Wire.requestFrom(104, 7);    // request 7 bytes from slave ds1307, we'll assume it'll send them all even though it doesn't have to
 second = Wire.receive(); 
 minute = Wire.receive(); 
 hour = Wire.receive(); 
 day_of_week=Wire.receive(); 
 day = Wire.receive(); 
 month = Wire.receive(); 
 year = Wire.receive(); 

 // Convert all the BCD values that might have "tens" to decimal.
 second = second/16 * 10 + second % 16;
 minute = minute/16 * 10 + minute % 16;
 hour = hour/16 * 10 + hour % 16;
 day = day/16 * 10 + day % 16;
 month = month/16 * 10 + month % 16;
 year = 2000 + year/16 * 10 + year % 16;
 
// Change times below to desired photo times. Copy & paste to add more photos per day.
// NOTE: for some reason, 8's & 9's cause an error, so don't use them on their own below;
// 18 & 19 work fine, but 08 & 09 do not.

 if (hour == 6) { if (minute == 00) { if (second == 00) { TakePhoto();}}}
 if (hour == 12) { if (minute == 00) { if (second == 00) { TakePhoto();}}}
 if (hour == 14) { if (minute == 00) { if (second == 00) { TakePhoto();}}}
 if (hour == 18) { if (minute == 00) { if (second == 00) { TakePhoto();}}}

 Serial.print(hour);
 print_prompt_P(PSTR(":"));
 if (minute < 10) { print_prompt_P(PSTR("0")); }
 Serial.print(minute);
 print_prompt_P(PSTR(":"));
 if (second < 10) { print_prompt_P(PSTR("0")); }
 Serial.print(second);
 print_prompt_P(PSTR(" on "));
 Serial.print(dow[day_of_week]);
 print_prompt_P(PSTR(", "));
 Serial.print(day);
 print_prompt_P(PSTR("/"));
 Serial.print(month);
 print_prompt_P(PSTR("/"));
 Serial.print(year);
 Serial.println();
  byte span = 20;  int aRead = 0;
  for (byte i = 0; i < span; i++) {        //loop to get average of 20 readings
    aRead = aRead + analogRead(ThermistorPIN);
    }
  aRead = aRead / span;
  temperature = Thermistor(aRead);

...continued next post...

...rest of the code:

 if (second == 0) {  // this code writes data to the sector_buffer every minute
   Serial.println("Writing data...");
   sector_buffer[WriteAddress] = day;
//   Serial.print(sector_buffer[WriteAddress]);
//   Serial.print(", day = ");
//   Serial.println(day);
   ++WriteAddress;
   sector_buffer[WriteAddress] = month;
//   Serial.print(sector_buffer[WriteAddress]);
//   Serial.print(", month = ");
//   Serial.println(month);
   ++WriteAddress;
   sector_buffer[WriteAddress] = (year-2000);
   ++WriteAddress;
   sector_buffer[WriteAddress] = hour;
   ++WriteAddress;
   sector_buffer[WriteAddress] = minute;
   ++WriteAddress;
   sector_buffer[WriteAddress] = byte(temperature);
   ++WriteAddress;
      span = 50;  unsigned int VBatt = 0;
      for (byte b = 0; b < span; b++) {        //loop to get average of 50 readings
        VBatt = VBatt + analogRead(1);
        }
      VBatt = VBatt / span;
   sector_buffer[WriteAddress] = byte(VBatt/79);
   ++WriteAddress;
   sector_buffer[WriteAddress] = byte(VBatt%79);
   ++WriteAddress;
// Note:
// The battery voltage on the divider ranges from 0 to 13.5 volts.
// The voltage on the arduino pin ranges from 0 to 5 volts using 2200- & 3900-ohm resistors.
// So an analogRead value of 1023 = 13 volts on the divider
// Therefore one bit is equal to 1024 divided by 13V = 78.769, rounded to 79.
      span = 50;  unsigned int VPanel = 0;
      for (byte P = 0; P < span; P++) {        //loop to get average of 50 readings
        VPanel = VPanel + analogRead(2);
        }
      VPanel = VPanel / span;
// *** NEED TO FIND OUT max V output of solar panel, divide 1024 by this, then replace 79 with the result. ***
   sector_buffer[WriteAddress] = byte(VPanel/79);
   ++WriteAddress;
   sector_buffer[WriteAddress] = byte(VPanel%79);
   ++WriteAddress;
   if (WriteAddress > 512) {
     SDWrite;
     ++sector;
     WriteAddress = 0;
   }
  }

  // check for any incoming serial input, then Dump Data only if 'd' is pressed
  if (Serial.available() > 0) {
    incomingByte = Serial.read();      
    if (incomingByte == 100) {
      DataDump();
    }
  }
 delay(1000);
}

There are a few redundunt bits & pieces, & lots of comments for my own benefit, but it shouldn't be real hard for the rest of you who are as new to this as I am to follow! :slight_smile:

Sorry, I've got to head out of the office now so this is a bit of a rushed response... If more info is required I'll check in again later today/tonight.

Thanks again,

JB.

-* EDIT *- After having a bit more time to digest your response I've made a couple of modifications to the code; it probably still has some gaping errors but it's semi-functional & allows for testing...

Howdy

Thanks for posting the code - it helps everyone in the end. I realise it can be a bit like exposing your very soul but we're all in this together.

I have a few questions. Do you only want to collect the last 50 or so readings or would you prefer to keep a rolling record?

When you dump the data do you just want to see the currently buffered entries or all of the data on the card or would you prefer to be able to say 'last 10 entries' or similar?

From reading the code I can't quite make out what your requirements are. Do you think you could write a few paragraphs about this? Try and be condescendingly pedantic :slight_smile:

One thing you could consider is trading code volume for end-result readability. Specifying that you require the data file to be human-readable might add an up-front code cost but could give you huge benefits...

C

Hi Charlie,

I'ts probably more accurate to say that 'my' code is exposing about half a dozen forum poster's souls - there's not too much in there that I've written myself but rather cobbled together from this wonderfully supportive community! :slight_smile:

The original rundown of my project was as follows:

  • A remote, solar-powered, automated camera system to take photos of key wetlands at specified times each day
  • Using a Canon Powershot A720 consumer camera, with modified firmware (see CHDK Wiki | Fandom for more info - I'm sure there are people here who would be interested in what the CHDK community has achieved)
  • Thermistor to monitor enclosure temperature, and operate fans when required
  • DS1307 mini-board (DS1307 Real Time Clock Mini Board) for RTC
  • Arduino to control fans, camera power & solenoid to turn camera on at specific times
  • Camera should happily run unattended for around 3 months
    So, basically the sketch as originally conceived was designed to:- monitor the time from the RTC, and the temperature from the thermistor.
  • check the temperature against the previous second's readings; if it is higher than the MaxTemp, make this reading the new MaxTemp; if it is lower than the MinTemp, make it the new MinTemp.
  • at midnight, record the date & daily MaxTemp to EEPROM & reset MaxTemp to 0; around midnight, record the daily MinTemp & reset to 40.
  • if the temperature is over 35 degrees C, start the fans; if the temp has dropped below 33 degrees 1 minute later, switch the fans off, otherwise leave them running & keep checking each minute.
  • check to see if the time matches any of the pre-determined times to take a photo.
  • If it's time for a photo: (1) record whether the fans are on or off, and switch them off if they are on; (2) send power to the camera; (3); trigger the solenoid for 100mS, which pushes the camera's power button; (4) leave power on to the camera for 15 seconds so the camera can take a photo then power itself off; (5) return fans to their previous status.
    However, on implementing a prototype system, we have found that the enclosure & heat-shield we've employed makes the temperature control functionality redundant - the interior temperature is the same as the external temp, so the fans don't actually provide any cooling effect. I'll now use them to just blow air over the camera aperature a few times a day, in order to remove dust and/or insects.

We thought that it would be helpful to record daily maximum & minimum temperatures, so I set up a sketch which would record the date (dd/mm/yy) & max & min temps in 5-byte chunks once each day in EEPROM. The data dump procedure was initially set up to read the EEPROM & display it in a table-like output like this:
Data dump initiated:

Date MaxTemp MinTemp
1/1/2009 20 12
2/1/2009 21 9
3/1/2009 29 12
4/1/2009 30 11
5/1/2009 25 13
6/1/2009 29 16
7/1/2009 23 14
8/1/2009 21 10
9/1/2009 23 10
10/1/2009 24 12
11/1/2009 22 13
12/1/2009 28 17
13/1/2009 43 18

which I could cut & paste to Excel or a simple text file. However, with uFat I'm thinking that a data dump procedure may be a bit redundant, if I can simply pull out the SD card, plug it into my laptop & copy the file. One thing that I do need to keep in mind in this project is that it won't always necessarily be ME who services these cameras, so I need to keep their operation as user-friendly as possible...

Anyway, the idea with writing data to the SD card is that I may as well use the huge amount of capacity available & write data to it every minute. At this stage, I'm looking at recording data in the following manner:

Date, Time, Temp, VV.vv (Battery voltage), VV.vv (Solar Panel voltage)

where Date = DD/MM/YY (3 bytes)
Time = HH:MM (2 bytes)
Temp = degrees C (1 byte)
VV.vv = voltages to 2 decimal places - (2 bytes each)

So, I'm left with 10 bytes of actual data going onto the SD card. However, in keeping with the spirit of simplicity mentioned above, I'm thinking that ideally the data written to the SD card would be in comma seperated value (.csv) format; that way we can easily import it into a spreadsheet for viewing & manipulation. This would add another 10 bytes or so for /'s, commas, decimal points etc.

My concern here is that 10 or 20 bytes obviously don't divide too well into 512; what happens when the buffer is full? Do I lose some data there? I'm not up to speed on exactly how sectors work, so forgive me if this is a stupid question... Could I make sector_buffer a 520-byte array (for example), write 512 bytes to SD card sector 0, then put bytes 513-520 back into the first 8 bytes of the array, to be written to sector 1 when the array is full again? Will a computer see this as one continuous file?

I think I'll leave it there for now - hopefully this answers most of your questions, and since this post is getting a bit(!) lengthy now I'll let you digest this lot & ask any more questions as required.

Cheers

JB

I have written a simple library based on uFat which allows you to write to the SD file as many bytes as you want at a time. It buffers the data until you have a full sector and then writes that to the card, and then starts buffering for the next sector.

You can download it from http://code.jhodges.co.uk/SDWriter/SDWriter.zip (unzip to arduino/hardware/libraries/)

I have only just finished this, and it definitely is not polished, and probably lacks some error checking. There are also some Serial.print's in the library itself that I used for testing which you will probably want to remove if you are using serial yourself.

I have posted this in its infancy as it looks like it could be of use to the posters on this thread, JB especially :wink:

There is a commented example of usage in there.

Hope this is of help. :slight_smile:

Downloaded & tried, thanks bobemoe!

Here is what is in my data.bin file after running the sketch:

C:\arduino-0012\output.jpg
-< edit >- bugger - how do I put a photo in here?!? There should be a bmp of strange shapes & symbols instead of numbers here...

Do you know why I'm getting this wierd stuff instead of numbers?

Thanks for your help anyway!!

JB

Hi JB,

You wont be able to open it in a text editor... its saved as binary - the actual bytes rather than an ASCII representation of those numbers.

Each byte (value 0-255) in a TEXT file represents one character. So a file containing the string "123", consists of 3 bytes of value: 49,50 and 51, which represent the characters 1,2 and 3 (see http://www.asciitable.com/)

If we write the single byte value 123 to a file we get a file which consists of 1 byte of data, value 123. But if we view this as TEXT we will see only 1 character, a { in this case, which is represented by the value 123. Which is why you are seeing funny characters, as a lot of the ASCII set is symbols and other weird stuff!

I will add a writeString() function to the library, but I think you would probably have trouble converting your byte's and int's to a string, I did look into this briefly (something to do with stdio.h, sprintf() and bigger code!) but quickly decided I will stick to writing values not strings to the file.

I wrote a simple PHP script that converts the raw data into a CSV. What operating system are you using? I can share this with you, and help you with it if your on Mac or Linux, but wouldn't be able to help so much if on Windows.

I understand you want relatively user-friendly access to your data. This conversion process could be packaged into a single command or double clickable file, data.bin goes in, data.csv comes out.

It seems you are at a bit of a fork in the road, and I am not sure the best route for you. What does everyone else think? I thought string manipulation looked a little tricky in Arduino/C. I've only had my Arduino a few weeks, also I am more used to PHP/Java/BASH. Is this conversion a sensible trade-off for avoiding strings in this case?

If you were writing lots of data, and fast, you would most certainly want to write byte values rather than ASCII strings, as this will keep file sizes much smaller and reduce the amount of time spent writing to the card (exactly what I need).

In your case trying to figure out this string thing (its actually an array of chars) might be an idea.

Let me know which route you choose.

I will add the writeString() function. Can anyone link us to any good tutorials on strings and sprintf?

Thanks for explaining the weird characters, your post is very informative!!

Unfortunately (for me) I'm using Windows... don't get much choice with the Government!! ;D

I guess that's another option to look at; does anyone know of an existing Windows program which could convert the raw data to CSV or just plain text? I had hoped to have been able to just pull the SD card out, plug it into the laptop & view the output in 'real' numbers, but I now understand that I can't do that currently. I can't get over how generous the Arduino community is in terms of writing & sharing code with people like me, and I would be deeply grateful to have a look at a writeString() function if you were to play around with one!

Since my data is only taken once a minute, and the capacity of SD cards provides essentially unlimited storage in my case, speed and file size are not really an issue. Your comment:

In your case trying to figure out this string thing (its actually an array of chars) might be an idea.

...has got me thinking though, so I'll start playing around with that & see where it takes me...

Thanks again!

I could help you by writing a small Windows script or program for it.. sooner or later i will have to do that anyway.

I think i got the basic idea of how those values are written but bobemoe could you though give me your php code so i could simple rewrite it for windows?

I think the way to go is to use the code already present in the libraries. Check out .\hardware\cores\arduino\Print.cpp

This base class provides the 'printing' functionality. Since you're already wearing the cost of the code whenever you use the Serial library it can be used virtually free. All you need to do is provide a subclass which provides a write byte function which puts a byte to the MMC buffer and writes the sector when necessary.

What you get is the ability to write strings using the exact same functions as you'd use for Serial! How cool is that.

As a super bonus, you could interchangeably use the MMC output with the Serial output for testing..

Here's how. (Please forgive typos, I'm away from arduino right now so I can't test)

class MMCWriter : public Print
{
  public:
    void begin( .. )
    {
       // initialise the mmc byte-by-byte writer here
    }

    void flush(void)
    {
       // flush the buffer to the card here
    }

    virtual void write(uint8_t theByte)
    {
       // call the function that puts a byte in the buffer here 
    }
};

MMCWriter MMCio;

void setup(void)
{
  MMCio.begin();
  MMCio.println("Hello! This string will go to the buffer!");
  MMCio.println(123, DEC);
  MMCio.flush();
}

That's it. I'll leave the serial/mmc swaperoo trick for another post (or you could work it out if you've got 5 mins).

There's a huge amount of power in the arduino core - take some time to have an explore and who knows what you might find :slight_smile:

C

Here's a few general notes that I've been meaning to make for a while.

One thing to remember is that uFat simply overwrites data that is pre-allocated on the card. This needs to be a contiguous file else wierdness will ensue. One way to ensure this is to format the card before writing your proxy file to it. Deleting files from the card can leave holes in the FAT which your operating system will use next time a file is created on the card leading to a fragmented file.

Unfortunately resizing a file isn't a simple case of adjusting a value in the directory structure. It entails re-linking the FAT table chains, which means a lot more code in uFat, which is counter to its design criteria. I think it's a reasonable trade-off. Actually, thinking about it, if people were happy with yet another trade off then we might be able to come up with a small extension that can work with one or two files on a card and set their lengths in a one-off manner... But I digress.

The length of the file indicated in the directory is the raw size of the original proxy file, not the number of bytes you've written. Its size remains constant no matter how you write to the space it occupies.

If you write binary data to the file then you'll no doubt have a client application that understands it.

If you write ascii data then things are a little (but only a tiny bit) harder. When the file is written to and later removed to the computer and viewed in a text editor, there may well be a warning that the file is corrupt (at best) to a refusal to load it at all (at worst). This is to do with non-ascii ranged bytes in the file, as explained in a previous post. Warnings can be ignored on the whole. It's best practice to empty the buffer before writing data to it. You could fill it with character 0x20 (ascii for 'space') but that would mean a bloated file when editing. You could fill with 0x0d (carriage returns), which might be better. I'd go with 0 myself, unless it was making editing hard.

Thoughts encouraged.
C

Nachtwind, and those who want it, my PHP for converting to CSV:

<?php
$packet=array(3,2,2); //the number of bytes to read for each value of the 'packet' of data to be converted to a csv line.

$f=fopen('data.bin','r');
while(!feof($f)){ //repeat until end of file
      $csv_line='';
      foreach($packet AS $bytes){ // loop through the packet and generate the CSV line
            if($csv_line) $csv_line.=","; //append a comma if needed
            $csv_line.=getVal($f,$bytes); //append the next value
      }
      echo $csv_line."\n"; //echo the line
}

//reads $bytes bytes from file $f and convert them to a value
//does this by reading each bite as a character, and converting it to its ASCII value
function getVal($f,$bytes){
      $val=0;
      for($i=0;$i<$bytes;$i++){ // loop for however many bytes we require
            $b=ord(fgetc($f)); // read a character and convert it to it's ASCII byte value with ord()
            $val=($val<<8)+$b; // shift the current value left eight bits (8 bits == 1 byte) and then add the byte we just read to the value
      }
      return $val;
}
?>

You must adjust $packet to reflect the number of bytes used for each value. (This is currently configured to read the data generated from my example in SDWriter)

The magic of joining the bytes into larger values happens on this line: $val=($val<<8)+$b
this shifting of bits is how we join two bytes back into an int
this could also be achieved by multiplying by 256: $val=($val*256)+$b;

val byte2 byte1 example values and how they would be stored in multiople bytes
255 255 as we probably know, 255 is the most one byte to contain
256 1 0 to store 256 or more we need another byte to store multiples of 256. (1*256) + 0 =256
261 1 5 (1*256) + 5 =261
65535 255 255 (255*256) + 255 = 65535 //max value of an int (2 bytes)

you could now add a third byte to store multiples of 65536
this can be done for as many bytes as you need to contain your range of numbers, with only a few more bytes you can store over billions!

sirmorris, I find the functionality of uFat is excellent and the trade-offs are not a problem, especially as it seems we are all really only wanting to log data, I can't really see the need for more functionality. If I wanted to do much more I think I would probably want a full FAT filesystem. But only the future can tell :wink:

Also your method of printing to the buffer looks really cool, I haven't got time to study it greatly now, but it looks like it's exactly what JB needs and wouldn't require my suggested writeString function. In fact I guess I could extend my SDWriter in the same way to add the print/println functionality to it :smiley:

Thanks!

It looks like SDWriter would just pop right on top there!

Great stuff. I'm thrilled that uFat is finding some use.

C

I had to change about two lines and now my SDWriter supports .print() and .println() in the same way Serial does, thanks to sirmorris' genius idea :slight_smile:

The new code is available in the same place at: http://code.jhodges.co.uk/SDWriter/SDWriter.zip

There is a new example in there called Print_Data that creates a CSV style file.

The old example is still there as writing byte values is still supported.

If I may I'd like to offer a couple of comments on the writer.

Having the write function which is used by the print() calls go through a general-purpose write function is less efficient than making things work the other way viz:

void SDWriter::write(uint8_t theByte)
{
  sector_buffer[buffer_pos]= theByte;
  ++buffer_pos;
  if(buffer_pos==512)
  {
    flush();
  }
}

void SDWriter::write(unsigned long val,int bytes)
{
  for(int b=bytes-1;b>=0;--b)
  {
    write((val>>(8*b)) & 0xff);
  }
}

This way the print calls all go through the minimal version.

Good point. I guess it is the little things that count when you only have a little processor and memory :wink:

The library is updated with that change.

Thanks again.

You guys are AMAZING!! :smiley:

I've only had time this morning to quickly compile & try bobemoe's Print_Data example, but I can see that it's EXACTLY what I was after. I should be able to incorporate this code to 'format' the data stream in the most user-friendly way possible, which will make life so much easier for my workmates (and me!).

I'll be away from my computer today so I won't be putting much time into this, but in the next couple of days I'll let you know exactly how things have gone.

Charlie & bobemoe - you're work here will be extremely valuable to many others like myself I'm sure; we can't thank you guys enough.

Oh, one last thing - Charlie, did you have any thoughts on why my RTC data is being 'corrupted'?

Thanks again everyone!!

JB

I've looked at the code and there are a couple of things that I'd like to go over with you - as they're application specific maybe you should PM me and we'll take it offline via email. We can post final findings but the interim comms might be a little noisy for this topic.

Is there any hope on getting the read from SD to work?

Hi

Reading from SD is a solved problem. The discussion happening here was to do with a specific problem sketch.

Charlie