reading text from an sd card

Show us your code.

mrboni:
I'm playing with the sd card read/write tutorial. The example sketch works fine.

When I create a text file in notepad, copy it to an sd card and open it using the arduino, the formatting is different. Ie -

Original text - this is a test 1 2 34
Text printed in serial monitor - tisisa es 12 4

I've tried asni, utf8 etc

any ideas?

mrboni:
Original text - this is a test 1 2 34
Serial text - t is is a es 1 2 4

As previously mentioned code would help; however, the serial text is a subset of the original which would indicate data being dropped, either in the serial transmission to the computer or possibly in your code reading the SD card.

Ha, I just rewrote the code and it works. Not sure what it was as the code is mostly just the example sketch.

One question, this code is printing to serial using Serial.write, not Serial.print. As I understand it this is because the data is in binary form. Given this, how do I assign the text (ascii) contents of this file to a string?

Here is the code -

#include <SD.h>

File myFile;

void setup()
{
  Serial.begin(9600);
  Serial.print("Initializing SD card...");
  // On the Ethernet Shield, CS is pin 4. It's set as an output by default.
  // Note that even if it's not used as the CS pin, the hardware SS pin 
  // (10 on most Arduino boards, 53 on the Mega) must be left as an output 
  // or the SD library functions will not work. 
   pinMode(53, OUTPUT);
   
  if (!SD.begin(4)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  
  
  // re-open the file for reading:
  myFile = SD.open("test.txt");
  if (myFile) {
    Serial.println("test.txt:");
    
    // read from the file until there's nothing else in it:
    while (myFile.available()) {
    	Serial.write(myFile.read());
    }
    // close the file:
    myFile.close();
  } else {
  	// if the file didn't open, print an error:
    Serial.println("error opening test.txt");
  }
}

void loop()
{
	// nothing happens after setup
}

The corresponding method to Serial.read() is Serial.write().

Additionally, write() can also send null terminated strings, if you provide a char* parameter.
Serial.print() and println() are rather convenience methods to format various data types into readable text.

To read longer texts from a file into variables you have to take care where to store it ( and how long ), as Arduino RAM is very limited.

The sample code does not store it at all, but simply uses every read character just as output parameter to Serial.write().
This way, it can handle files of any length.

For any other use, the solution depends on what you need.

I want to read in the text "46,110,173,237;"

and store it as a String "46,110,173,237;"

See my question: "Problem with DmxSimple.h". My code does exactly what you want and then even parses the lines of text. Of course it seems to be un-usable when including DmxSimple.h but that is another topic :slight_smile:

Regards

Simon

I'm trying to make the same thing :slight_smile:

Well a popular thing to do I guess :-). Thanks to pylon my code now works. I have placed a lot of the debug messages in comments aqnd now the codes works fine even with the include DmxSimple.h You can clean it up as this was my "test and debug" version but it does what it should. You'll still have top add the proper calls to DmxSimple but at least the lines are parsed correctly and are available in an array.

Here is the "new" version of my code:

//SD CSV File Read

#include <SD.h>
#include <String.h>
//#include <DMXSerial.h>
#include <DmxSimple.h>

//Set by default for the SD Card Library
//MOSI = Pin 11
//MISO = Pin 12
//SCLK = PIN 13
//We always need to set the CS Pin
int CS_Pin = 10;
//int PowerPin = 8;

// Declare DMXValues Array and DMXValueIndex Counter. Keep Unsigned Integers below 65535
unsigned int DMXValues[15] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
int DMXValueIndex = 0;
int IterationCounter = 0;
int AvailableCounter = 0;

// Declare CSVStringBuffer Input Character Buffer, SimpleToken "container" and IntermediateBuffer Temporary Location for Read char from File
char CSVStringBuffer[80];
char *SimpleToken = NULL;
char IntermediateBuffer = NULL;

void setup()
{
  Serial.begin(9600);
  Serial.println("Initializing Card");
  //CS Pin is an output
  pinMode(CS_Pin, OUTPUT);
  
  //Card will Draw Power from Pin 8, so set it high
  //Strange Arduino Idea... Arrrrggghhh them Software Designer they're so naughty!
  //This May not survive in Final Design
  //pinMode(PowerPin, OUTPUT);  
  //digitalWrite(PowerPin, HIGH);
  
  if (!SD.begin(CS_Pin))
  {
      Serial.println("Card Failure");
      return;
  }
  Serial.println("Card Ready");
  
  // Read DMX Sequence information (CSVTest1.csv)
  File CSVCommandFile = SD.open("CSVTest1.csv");
  if (CSVCommandFile)
  {
    Serial.println("Reading DMXSequence File");
    
    AvailableCounter = CSVCommandFile.available();
    Serial.print("Available Character(s): ");
    Serial.println(AvailableCounter);
   
   // As long as there are characters and that the Index is less than 80 characters read the file 
   while(CSVCommandFile.available() && (IterationCounter <= 79))
   {
     IntermediateBuffer = CSVCommandFile.read();
      // Look for the first exception, 0x0D or carriage return. May be absent in a Mac generated CSV file
     if (IntermediateBuffer == 0x0D)
     {
       continue; // If found, re-iterate the while loop
     }
     // Look for the second exception, 0x0A or line feed 
     else if (IntermediateBuffer == 0x0A)
     {
       CSVStringBuffer[IterationCounter] = NULL;
       
       
       // Actual Parsing of CSVStringBuffer to find Tokens
       // This is where the magic happens :-)
       SimpleToken = strtok(CSVStringBuffer, ",");
       while(SimpleToken)
       {
         // Do something with this token
         DMXValues[DMXValueIndex] = atoi(SimpleToken);
         
//         Serial.print(DMXValues[DMXValueIndex]);
//         Serial.print(" was read into Integer Array at position ");
//         Serial.println(DMXValueIndex);
         
         DMXValueIndex++;
         
         // Get the next token... 
         SimpleToken = strtok(NULL, ",");
       }
       
       // Various progress messages 
//       Serial.println("Done Acquiring Character from 1 Line in CSVTest1.csv");
       
       Serial.print("Integer Array is: ");
       for(int j = 0; j <= (DMXValueIndex - 1); j++)
       {
         Serial.print(DMXValues[j]);
         Serial.print(" ");
       }
       Serial.println(" ");
       
       // Reset Index for Values Array before exiting and re-looping for the next line in CSVCommandFile
       DMXValueIndex = 0;

       // Reset Index for CharBuffer before exiting and re-looping for the next line in CSVCommandFile
       IterationCounter = 0;
     }
     
     // This section is where the CSVStringBuffer is filled with the characters from the file
     else
     {
       CSVStringBuffer[IterationCounter] = IntermediateBuffer;
//       Serial.println(CSVStringBuffer[IterationCounter]);
       //Serial.print("Iteration Counter = ");
       //Serial.println(IterationCounter);
       IterationCounter++;
     }
   }
  }
  
  // If upon exit of while loop i hasn't been reset there was a problem. Should test to determine the problem and print to log file
  if (IterationCounter != 0)
  {
    Serial.println("Aborted Character Acquisition");
    Serial.print("IterationCounter= ");
    Serial.println(IterationCounter);
//    Serial.print("Available Character(s): ");
//    Serial.println(AvailableCounter);
  }
  // Succes message
  else
  {
    Serial.print("Read total of ");
    Serial.print(AvailableCounter);
    Serial.println(" characters");
//    Serial.print("DMXValueIndex= ");
//    Serial.println(DMXValueIndex);
    Serial.println("Operation Succesful");
  }
}

void loop()
{
}

I want to read in the text "46,110,173,237;"
and store it as a String "46,110,173,237;"

Then it's slightly different to pshoule's approach ...

I guess your rules are:

  • There are max 16 characters ( 4*3 digits + separators ). So define a
    ** **char text[17];** **
    as minimum for those characters and a trailing 0 end indicator.
  • The last one is the only semicolon ( ';' ) in the file, and it's always there as an end indicator. (If you're at the end of the provided space, treat it as a wrong file)
  • There are no line feed ( 0x0A ) or carriage return (0x0D) characters in the file before the semicolon. (If there are, treat it as a wrong file)

If you were interested in the numbers, there were other rules, of course. ...and pshoule's approach was worth having a closer look :wink:

You just want to read a character, store it in your text array at the correct location and check if all is still ok. If ok and no end indicator found yet, do it again with another character to the next location.
When done, add a trailing 0 after the last character you just stored ( the ';' ), so that Serial.println(text); works well.

Ah haah, thanks. you've just helped me understand some crucial things!

how's this? -

int byteCount = 0;
      while (myFile.available()) 
      {
        testBuff[byteCount] = myFile.read();
        byteCount++;
      }
      testBuff[byteCount] = NULL;

Can I set the size of char testBuff[] dynamically, increasing by 1 every time I need to add a new byte, or do you have to just initialise the array to be larger than it can ever be?

just initialise the array to be

...as large as it can ever be!
And check if that's true ( never write beyond the array size )

how's this?

Fine

Great.

One more thing - As each comma separated value is describing one value of 0-255, is it possible to just send single bytes in sequence with a delimiter at the end?

If so, how would these appear in a text file? I'm using another piece of software to generate the text files and make it do anything. I'm just not sure what a single byte should 'look like'

This would reduce each file size by 75%, which would be great as I'm trying to send lots of values very quickly.

Thanks again

mrboni:
Is it possible to just send single bytes in sequence with a delimiter at the end?

If so, how would these appear in a text file? I'm using another piece of software to generate the text files and make it do anything. I'm just not sure what a single byte should 'look like'

Sure, it's not a text file any more but some binary file, without any structure.
You cannot read it with notepad, but it would open and read in your sketch were the same.
Every byte you read would have a specific meaning.

mrboni:
This would reduce each file size by 75%, which would be great as I'm trying to send lots of values very quickly.

File size does not really matter, and won't affect speed too much, I think. It's not a Serial connection at 2400 baud.
What is "lots of values" ? You cannot store 2k bytes in an Arduino UNO anyway. If it's 200 bytes, it does not really matter if the file size is 200 or 1000 bytes.
I've got a very old SD card of 32 MB, which is horribly oversized for my Arduino project.

As a compromise: what about storing the data as hex strings. perhaps without comma delimiters but newline to structure it and make it readable?

"46,110,173,237;" would become
"2E6EADED"

Readable in a text editor, and easily convertible to binary in Arduino.

Hi Michael.

I'm using a Mega2560 with an ethernet shield.

I've created text files, each one representing one 'frame' of 512 8bit values. The plan is to read these load each frame into arduino ram, update these 512 values (which get sent as dmx), then load the next frame.

I need to read 512 8bit values at as close to 30 fps as possible.

Below is the test code I'm using to open then close 30 of these text files

Where do you think the bottleneck is?

#include <SD.h>

File myFile;

void setup()
{
  Serial.begin(57600);
  Serial.print("Initializing SD card...");
  // On the Ethernet Shield, CS is pin 4. It's set as an output by default.
  // Note that even if it's not used as the CS pin, the hardware SS pin 
  // (10 on most Arduino boards, 53 on the Mega) must be left as an output 
  // or the SD library functions will not work. 
   pinMode(53, OUTPUT);
   
  if (!SD.begin(4)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  
  String filenamePre = "fr";
  String filenamePost = ".txt";
  String filenameStr = "fr21.txt";
  char filename[13];  
  char testBuff[513];
  
  
  Serial.println("beginning read");
  Serial.println(millis());
  
  int j = 0;
  while(j < 30)
  {
  
    // re-open the file for reading:
    filenameStr = filenamePre + j + filenamePost;
    filenameStr.toCharArray(filename, 13);
    myFile = SD.open(filename);
    if (myFile) {
      //Serial.println(filename);
      
      // read from the file until there's nothing else in it:
      int byteCount = 0;
      while (myFile.available()) 
      {
        testBuff[byteCount] = myFile.read();
        byteCount++;
      }
      testBuff[byteCount] = NULL;

      
      // close the file:
      myFile.close();
    } else {
    	// if the file didn't open, print an error:
      Serial.println("error opening test.txt");
    }
    
    
   // Serial.println(testBuff);
    
  j++;
  }
  
  Serial.println();
  Serial.println(millis());
  
  
  
}

void loop()
{
	// nothing happens after setup
}

Where do you think the bottleneck is?

Is this a trick question ? :wink:

SD card speed depends. I read that it gets slower when you delete / create many files often on a card, without re-formatting the card.

What's your result ?

BTW, I would not use String for the filenames, but

 static char filename[] = "fr0#.txt";
 
 for (int tens = '0'; tens< '3'; tens++) {
   for (int ones = '0'; ones <='9'; ones++) {
      filename[3] = ones;
      // open the file for reading:
      myFile = SD.open(filename);
      // etc.
   }
   filename[2]++;
 }

There might be enough space for 30 dynamic Strings besides the 512 byte data, but in a "run forever" scenario it's a bad idea...

And check that your byteCount does not exceed the limit (512) with a bad file

Is this a trick question ?

Do you mean 'How the * should I know?'. Was just hoping there might be an obvious way, different to my approach, to speed up the reading of each file on the card

It's currently taking 54ms to open,read, then close each (1024byte) file. This equates to a maximum of 18fps before any further time spent processing.

I came across Google Code Archive - Long-term storage for Google Code Project Hosting. which is an optimised sd library but it only supports fat16 standard sd cards and I don't have one to test. Any idea if this would speed up file opening and reading times?

Using hex instead of csv takes 2/3 of the time, but if there is no delimiter, how do I read in values less than 0xF? ie where there is only one digit... Can I just write 00 rather than 0 in the text file?

Thanks for your optimised filename code. I wasn't sure how to do it with chars. It's way better!

edit- also, how do I convert a char[] with a value of (for example) 'FF' into an int with a value of 255?

Using hex instead of csv takes 2/3 of the time, but if there is no delimiter, how do I read in values less than 0xF? ie where there is only one digit... Can I just write 00 rather than 0 in the text file?

That would have been my approach: Always two chars to one byte.

how do I convert a char[] with a value of (for example) 'FF' into an int with a value of 255?

byte b = bin(c[0])<<4+bin(c[1]);

byte bin(char c)
{
  if (c >='A') return c-'A'+10;
  else return c-'0';
}

This method bin() only works for '0' ...'9' , 'A' ... 'F' . Every other character, including spaces or 'a' ... 'f' produces garbage.

BTW: If it's for absolute speed optimization, my first approach (file size does not matter) might be wrong: You might prefer binary files over human readable files.

Thanks Michael

I've actually got it working using ascii codes, running at 25 fps for a 512byte (and 512 value) file

Attached code shows it controlling a string of ws2801 led driver chips

Seems to be a memory leak though as it stops running after a few seconds. So close :slight_smile:

#include <SD.h>
#include "SPI.h"
#include "Adafruit_WS2801.h"


int dataPin  = 2;    // Yellow wire on Adafruit Pixels
int clockPin = 3;    // Green wire on Adafruit Pixels

Adafruit_WS2801 strip = Adafruit_WS2801(170, dataPin, clockPin);



File myFile;

unsigned long millisPre;
unsigned long millisPost;
int testLoopCount;

void setup()
{
  strip.begin();
  // Update LED contents, to start they are all 'off'
  strip.show();
  
  
  Serial.begin(57600);
  Serial.print("Initializing SD card...");
  // On the Ethernet Shield, CS is pin 4. It's set as an output by default.
  // Note that even if it's not used as the CS pin, the hardware SS pin 
  // (10 on most Arduino boards, 53 on the Mega) must be left as an output 
  // or the SD library functions will not work. 
   pinMode(53, OUTPUT);
   
  if (!SD.begin(4)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  

    
  unsigned char testBuff[513];
  int testBuffInt[512];
  
  Serial.println("beginning read");
  millisPre = millis();
  //Serial.println(millisPre);
   
  
  
  testLoopCount = 30;
  
  for(int cycles = 0; cycles < testLoopCount; cycles++)
  {
    
    static char filename[] = "fr0#.txt";
 
   for (int tens = '0'; tens< '3'; tens++) 
   {
     for (int ones = '0'; ones <='9'; ones++) 
     {
        filename[3] = ones;
        // open the file for reading:
        myFile = SD.open(filename);
        if (myFile) 
        {
          //Serial.println(filename);
          
          // read from the file until there's nothing else in it:
          int byteCount = 0;
          while (myFile.available()) 
          {
            testBuff[byteCount] = myFile.read();
            //testBuffInt[byteCount] =(int)testBuff[byteCount];
            byteCount++;
          }
          testBuff[byteCount] = NULL;
          
          
          //set ws2801 pixels
          for(int i=0;i<512;i++)//channel position
          {
            strip.setPixelColor(i,(int)testBuff[i*3],(int)testBuff[i*3+1],(int)testBuff[i*3+2]);
          }
          strip.show();
          
          
          // close the file:
          myFile.close();
        } 
        else 
        {
          // if the file didn't open, print an error:
          Serial.print("error opening ");
          Serial.print(filename);
          Serial.println(" test.txt");
        }
      }
        filename[2]++;
    }
    
    filename[2] = '0';
    filename[3] = '#';

    
  }
   Serial.println();
   millisPost = millis();
   //Serial.println(millisPost);
   Serial.println((millisPost-millisPre)/testLoopCount); 
   

   
   
}



void loop()
{
	// nothing happens after setup
}

Your two testBuffs use 3/4 of your available SRAM. Probably the SD library has some buffer too, so I'm not surprised you ran out of memory. As far as I saw in a quick overview testBuffInt is not used but consumes half of your memory.

BTW: why storing the read values in an array and not directly putting them into the strip?

Thanks Pylon, that's a good point about the array.

Turns out it's not freezing though (yet) - I'd set it to only loop 30 times as a test. Duh