Sorting binary data from sdcard

system · February 20, 2011, 3:39am

Hi, I am trying to read data from sdcard (raw 0,1), and want to sort it so I can convert it to byte.

This is what I require:
The program, is simple. I have created one file, let say : 'pattern.txt', and save it in the sdcard.(Done)
Then the program will read all the raw data from sdcard, sort it to (8 bits -so I can have a byte), and declare it.
I have to declare it as byte because I will be shifting out this data through the shift register. Hope it's clear.

Let say, here's the raw data from the sdcard (pattern.txt):

00000000000000000000000000011100000000000000000000000000000000000000000000000000
00000000000000000000000000011100000000000000000000000000000000000000000000000000
01110000100001110011111100011100001111110000111111000011111110111110000111111000
01111001110011100111111110011100011111111001111111100011111111111110001111111100
00111001110011101111111110011100011111111001111111110011111111111111001111111110
00111011111011101110000111011100111100111011110001110011100111100111011110001110

as you can see, it's only raw data. So I will be polling the data one by one from the sdcard, and read it.
Here's the program that I test to group it into 8, and print it :

    in_char=file.read();              //Get the first byte in the file.
    while(in_char >=0){            //If the value of the character is less than 0 we've reached the end of the file.
    for(int y=0; y<8; y++){
      Serial.print(in_char);    //Print the current character
      in_char=file.read();      //Get the next character
    }
    delay(20);              // some delay

As you can see, I only read it one by one. My questions :

After I read the data (every 8 bits), how can I save it to variable, and declare it as a byte? Then read the next bit and do the same thing. (It required only temporary variable, as I will be shifting it out to the shift register right away).
How can I eliminate(remove or maybe neglect) any spaces in the raw data?

Let say:

00000000   00000000   0000000000011100000000000000000000000000000000000000000000000000

00000000000000000000000000011100000000000000000000000000000000000000000000000000
01110000100001110011111100011100001111110000111111000011111110111110000111111000
01111001110011100111111110011100011111111001111111100011111111111110001111111100
00111001110011101111111110011100011111111001111111110011111111111111001111111110
00111011111011101110000111011100111100111011110001110011100111100111011110001110

As you can see, there are few spaces between the raw data. Is it possible to eliminate/neglect it?
As from the testing, the program will still read the spaces when the program reach " in_char=file.read(); " line.
I just want the 0 and 1 data.

Thank you in advance.

system · February 20, 2011, 11:31am

After I read the data (every 8 bits), how can I save it to variable, and declare it as a byte?

    in_char=file.read();              //Get the first byte in the file.

You are already storing the data read from the file into a variable. Your snippet does not tell us what type the variable is, though. The code suggests that it must be an int, though, since you test that it is greater than 0 before using it. If it were a byte, it could not be less than 0.

How can I eliminate(remove or maybe neglect) any spaces in the raw data?

Simplest way is to not put them in in the first place.

system · February 20, 2011, 3:22pm

Dear Pauls,

Thank you for your reply.

I have tried to declare it, but the code given will only store one int( o or 1).

How can I store every 8 bits, and save it in another variable, and declare it as byte?

From our previous post, you have given me this code Pauls. Hope you remember.
And I think I am getting close to it with your help.

byte vals[255]; // reserve some space
byte valInd = 0;
void setup(){
   Serial.begin(9600);  
}

void loop()
{
   while(Serial.available() >= 8) // When there are 8 "bits" to reaed
   {
      byte val = 0;
      for(byte i=0; i<8; i++)
      {
         int bit = Serial.read() - '0'; // bit will be 0 or 1
         val *= 2;
         val += bit;
      }
       Serial.println(val, BIN);  
       
      if(valInd < 255) // Make sure there is room
      {
         vals[valInd] = val; // Save the val
         valInd++; // Point to next position in array
      }
   }
}

I have 2 questions:

What does this operation did: (what's the purpose of multiplying the value by 2. and minus the serial data with '0'?

 int bit = Serial.read() - '0'; // bit will be 0 or 1
val *= 2;
val += bit;

And this. (as it looks like doing nothing with the first for loop (reading 8 bits).)

if(valInd < 255) // Make sure there is room
      {
         vals[valInd] = val; // Save the val
         valInd++; // Point to next position in array
      }

I am quite new with c++. Hope that you can help. Thank you.

GrooveFlotilla · February 20, 2011, 3:30pm

Subtracting '0' from an ASCII digit converts it to decimal.
A '1' is 0x31 in ASSCII, and a '0' is 0x30.
0x31 - 0x30 = 1

The multiply by two is simply shifting the value left one binary palce.

If you were working in decimal, the multiplication would be by 10.

system · February 20, 2011, 4:00pm

@Groove, thank you for your reply.

Is there any reason why we need to convert it to decimal? (Decimal or binary?)

Thank you for explanation about multiplying. That makes sense.

Is it possible that I declare the 'val' variable as byte? So that I can shift that out to the shift register. Example :

  int bit = Serial.read() - '0'; // bit will be 0 or 1
         val *= 2;
         byte val += bit;
         shiftOut(dataPin, clockPin, val)   // shift out to shift register.

Is it possible to do this?

Another thing, how about this operation, what it does basically?

 if(valInd < 255) // Make sure there is room
      {
         vals[valInd] = val; // Save the val
         valInd++; // Point to next position in array
      }

it looks like it will search for the next array. (as we have read the previous 8 bits . Now we will be reading the next bit after 8)
Would I need it if I direct read it from the serial, group it per 8 bits(byte), and direct shift that out.

Thank you.

GrooveFlotilla · February 20, 2011, 4:05pm

Is there any reason why we need to convert it to decimal? (Decimal or binary?)

Because you can't represent an ASCII character in a single bit.

it looks like it will search for the next array

It doesn't "look" for anything, it just stores the value given in the next avalable place in the array and increments the pointer to the next slot.

system · February 20, 2011, 4:37pm

@Groove, thank you.

Now, let's go back to the sdcard.

Now my raw data in 'pattern.txt' stored in sdcard is:

00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000011100000000000000000000000000000000000000000000000000
00000000000000000000000000011100000000000000000000000000000000000000000000000000
00000000000000000000000000011100000000000000000000000000000000000000000000000000
01110000100001110011111100011100001111110000111111000011111110111110000111111000
01111001110011100111111110011100011111111001111111100011111111111110001111111100
00111001110011101111111110011100011111111001111111110011111111111111001111111110
00111011111011101110000111011100111100111011110001110011100111100111011110001110
00011011111111001111111111011100111100000011100001110011100111000111011111111110
00011111111111001111111111011100111100000011100001110011100111000111011111111110
00011111011110001111111111011100111100000011100001110011100111000111011111111110
00001111011110001111001111011100011110111111111011110011100111000111011110011110
00001111011110000111111110011100011111111001111111110011100111000111001111111110
00000110001100000111111110011100001111111000111111100011100111000111001111111100
00000110001100000001111000001100000111100000011110000011100111000111000011111000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000

and my code for reading it and display it is:

while(in_char >=0){            //If the value of the character is less than 0 we've reached the end of the file.
     byte val = 0;
   
      for(byte i=0; i<8; i++)
      {
         int bit = file.read() - '0'; // bit will be 0 or 1
         val *= 2;
         val += bit;
      }
       Serial.print(val, BIN);  
       delay(300);  // some delay
       
       
      if(valInd < 1000) // Make sure there is room
      {
         vals[valInd] = val; // Save the val
         valInd++; // Point to next position in array
      }
     
    }

as you can see, now the sdcard is reading the raw data from the sdcard from top to bottom. (left to right)

As the data is not in an array[], flash or progmem, so I can't address the data to be read from bottom to up. (right to left).
(my application require this way). As current will only just check whether there is character required to be read, in_char >=0, and just poll it.

How can I read from bottom to top?
Is it possible?

system · February 20, 2011, 5:11pm

Why don't you store the data on the SD card in the proper order?

system · February 20, 2011, 5:47pm

@Pauls

The main reason that I need to reorganize it in the arduino program because I don't want to make changes to the data outputted from the picture> ascii.

As if modifying the file data, and sort it beforehand (using macro, notepad++ or vim), it's completed. Now, I am going to try to automatic sort it using arduino.

As for the serial, we can set the pointer. Like an example:

while(in_char >=0){            //If the value of the character is less than 0 we've reached the end of the file.
     byte val = 0;
   
      for(byte i=0; i<8; i++)
      {
         int bit = serial.read() - '0'; // bit will be 0 or 1
         val *= 2;
         val += bit;
      }
       Serial.print(val, BIN);  
       delay(300);  // some delay
       
       
      if(valInd < 1000) // Make sure there is room
      {
         vals[valInd] = val; // Save the val
         valInd++; // Point to next position in array
      }

How can I use it for the sdcard? Can this pointer work also for the file.read?

while(in_char >=0){            //If the value of the character is less than 0 we've reached the end of the file.
     byte val = 0;
   
      for(byte i=0; i<8; i++)
      {
         int bit = file.read() - '0'; // bit will be 0 or 1
         val *= 2;
         val += bit;
      }
       Serial.print(val, BIN);  
       delay(300);  // some delay
       
       
      if(valInd < 1000) // Make sure there is room
      {
         vals[valInd] = val; // Save the val
         valInd++; // Point to next position in array
      }

Another thing, this code:

int codesize = sizeof(running)/ sizeof(int);
int lines= codesize; // numbers of coding lines

will actually calculate the codesize. This is successfully done if we are going to calculate size of array saved in flash or progmem. How can I calculate the codesize of the data in the sdcard?

So I can set the pointer where to start reading the data. (and rather than using i++, we will be using i-- ) read from back.

system · February 20, 2011, 7:13pm

How can I calculate the codesize of the data in the sdcard?

Read the whole file, and count the number of bits. Divide by 8 to get the number of bytes.

Can this pointer work also for the file.read?

That's not a pointer. It's an index. Yes, it can be used to index any kind of array.

You should create a function to read 8 values that represent one byte from the SD card, and return the byte that those 8 "bits:" represent.

You should create a function to read the number of "bytes" in the file. Then, allocate space to hold that data, then call the function to read a byte the required number of times, storing the result in the correct place in the array.

It isn't clear what sorting needs to be done, or how the data read from the card, which is unlikely to all fit in memory, is to be used.

system · February 21, 2011, 5:53am

@Pauls,

I've created this to read the total bits and byte.

void setup(void)
{     
    Serial.begin(9600);        //Start a serial connection.
    delay(2000);  // some delay so we can read that it's ready 
    pinMode(10, OUTPUT);       //Pin 10 must be set as an output for the SD communication to work.
    card.init();               //Initialize the SD card and configure the I/O pins.
    volume.init(card);         //Initialize a volume on the SD card.
    root.openRoot(volume);     //Open the root directory in the volume. 
    file.open(root,name,O_READ);    //Open the file in read mode.
   
   //check for how many numbers of bits and bytes 
    in_char=file.read();
    while(in_char >=0){
    in_char_total++;
    in_char=file.read(); 
    }
    Serial.print("in char values are : ");  
    Serial.println(in_char_total);  
    
    no_of_byte= in_char_total/8;
    Serial.print("Total bytes are : ");  
    Serial.println(no_of_byte);  
    file.close();    //Close the file so we can read it again
    in_char=0;  // set the character buffer zero again as we will read it again in loop()    
    file.open(root,name,O_READ);    //Open the file again in read mode. 
   
}

It has successfully read the numbers of bits and bytes.

Now, how can I read the pattern.txt , from bottom to up. (left to right).
Here's the code that I tried to read the index, but I think it does not give any changes. Please let me know where I did it wrong?

 in_char=file.read(); 
    while(in_char >0){            //If the value of the character is less than 0 we've reached the end of the file.
    
      for(int x=0; x<10; x++)
      {
      for(int i=0; i<8; i++)
      {
         int bit = in_char - '0'; // bit will be 0 or 1 // CONVERT TO DECIMAL
         val *= 2;
         val += bit;     
         in_char=file.read(); 
      }
      
      Serial.print(val,BIN);  // print the val in binary (a byte)  
      delay(20);  // some delay
      
  
      
     /*
      if(valInd < 500) // Make sure there is room
      {
         vals[valInd] = val; // Save the val
         valInd++; // Point to next position in array
      }
      */
     
     }

For the index in the previous code given, I've tried to comment it out. Still the result output is the same.
As If I can make this work, I can use this index to start read the val from the last bits position. And valInd--

Any details on how can I pinpoint/read the sdcard file.read() from the last bits?

system · February 21, 2011, 10:15am

Now, how can I read the pattern.txt , from bottom to up. (left to right).

You can't.

You can only read the file from beginning to end.

What exactly are you planning to do with the data that you read from the SD card?

system · February 21, 2011, 12:17pm

Not possible Pauls?
This is what I want to do with the result.

 file.open(root,name,O_READ);    //Open the file in read mode.
  
  
    in_char=file.read(); 
    while(in_char >0){            //If the value of the character is less than 0 we've reached the end of the file.
    
      for(int x=0; x<10; x++)
      {    
      digitalWrite(latchPin, 0);  
      for(int i=0; i<8; i++)
      {
         int bit = in_char - '0'; // bit will be 0 or 1 // CONVERT TO DECIMAL
         val *= 2;
         val += bit;     
         in_char=file.read(); 
      }
      shiftOut(dataPin, clockPin, val);   // shift out to shift register
      Serial.print(val,BIN);  // print the val in binary (a byte)  
      digitalWrite(latchPin, 1);
      delay(delaytime);  // some delay
      
  
      if(valInd < 500) // Make sure there is room
      {
         vals[valInd] = val; // Save the val
         valInd++; // Point to next position in array
      }

As you can see above, once saved in val as byte, it will be shifted out to the shift register.

I am actually just trying to do some program, as now the results from the picture> ascii is printed out in that form. (which I can't change at the moment).

So, my idea is that, I can use that software (which will provide me with the .txt file), then I can just put it in SDcard, and done. I don't have to put any syntax or make any changes, the program will do the rest.

As for using the array and saving the data from (progmem), I don't have any problem as I can read the number of arrays, and read it from back.

Let say.

I stored all the 010101 pattern in

const uint16_t running[] PROGMEM= {
B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,
B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,B00000000,
}

so, as I know the position of the last array, by scanning it using

int codesize = sizeof(running)/ sizeof(int);
int lines= codesize; //54  numbers of coding lines

I can code this function to read it back from the back.

 for (int j = lines; j >=0; j=j-noofmodules) {
    digitalWrite(latchPin, 0);
      
    //load the light sequence you want from array
    for(int i=noofmodules-1; i>=0; i--){
    byte k= pgm_read_byte(&(running[j+i]));
    shiftOut(dataPin, clockPin, k);
    }
       //return the latch pin high to signal chip that it 
    //no longer needs to listen for information
    digitalWrite(latchPin, 1);
    delay(delaytime);
    }
  }

As for now, the shift register library only allows me to shift out the MSB or LSB.
But my application require me to shift out the data from back to bottom as well. (as the pattern will be read from bottom to top).

Or else, it will be a 360degrees rotated pattern.

FYI I'm using 10 shift registers for this program.
Please let me know if there are any ways I can solve this.

As you can see, in the setup file, I've managed to read the number of bits and bytes from the sdcard data.

 in_char=file.read();
    while(in_char >=0){
    in_char_total++;
    in_char=file.read(); 
    }
    Serial.print("Total bits are : ");  
    Serial.println(in_char_total);  
    
    no_of_byte= in_char_total/8;
    Serial.print("Total bytes are : ");  
    Serial.println(no_of_byte);

Are there any ways that I can use this data to read the index from the back?

It will print out result like this:

Total bits are: 240
Total bytes are: 30

From here, what i want to do is, let say.

file.read[240] - which will read the value (0 or 1) at index number 240 of the sdcard (which I know it's the last bit in the sdcard).
then, file.read[239].. file.read[238].. and so on.

Save the first 8 last bits in a byte variable, and shift that out.

system · February 21, 2011, 12:51pm

I don't really see any way to do what you want to do. The data in the file is arranged in one order and you want it in a different order. The capabilities of the SD library are fairly limited, allowing only reading the file sequentially.

Given enough memory, you could read the whole file into memory, and then shift out the bytes in the correct order(the reverse of what you read from the file). However, it is unlikely that you have enough memory to do that.

Your best approach is to write a PC application to read the as-defined file, and output a to-be-displayed file.

system · February 21, 2011, 1:07pm

@Pauls, Thank you.

There are no point to use the sdcard if I will be reading and saving it back to the progmem or flash.
Anyway, thank you.

Just want some declaration about the coding that you gave.

byte vals[255]; // reserve some space
byte valInd = 0;

void loop()
{
   while(Serial.available() > 8) // When there are 8 "bits" to reaed
   {
      byte val = 0;
      for(byte i=0; i<8; i++)
      {
         int bit = Serial.read() - '0'; // bit will be 0 or 1
         val *= 2;
         val += bit;
      }

      if(valInd < 255) // Make sure there is room
      {
         vals[valInd] = val; // Save the val
         valInd++; // Point to next position in array
      }
   }
}

My questions:

Are there any ways that I can make valInd as a variable rather than const value? What's the limitation size for that array?
Looking back to the code,

while(Serial.available() > 8) // When there are 8 "bits" to reaed
   {
      byte val = 0;  // do we need to declare it here? or at the setup()?
      for(byte i=0; i<8; i++)
      {

Why do we declare byte val=0 every time the serial.available >8? As after putting it in byte, supposed to be the index should be 9.
This is referring to this code:

if(valInd < 255) // Make sure there is room
      {
         vals[valInd] = val; // Save the val
         valInd++; // Point to next position in array
      }

Please let me know if I wrongly interpret your code. As am confused here.

system · February 21, 2011, 2:19pm

Why do we declare byte val=0 every time the serial.available >8? As

To clear out anythng that may already be in there before filling it with a new value, bit-by-bit.
In this case, because you are always putting in eight bits, and it is an eight bit wide variable, it probably isn't necessary, but is good practice.

system · February 21, 2011, 2:32pm

There are no point to use the sdcard if I will be reading and saving it back to the progmem or flash.

Since you can't write to progmem, there might still be a need to read the data from the SD card.

Are there any ways that I can make valInd as a variable rather than const value?

valInd IS a variable. Are you referring to the 255? That is the size of the array that is being indexed. That value can be stored in a variable, too.

What's the limitation size for that array?

On which Arduino? There is a limited amount of memory on each Arduino. All arrays need to fit in that memory.

Why do we declare byte val=0 every time the serial.available >8? As after putting it in byte, supposed to be the index should be 9.

val is created from a collection of 8 "bits". It is supposed to be reset to 0 after each 8 "bits" is read, so that the next 8 "bits" are added to a clean field.

system · February 23, 2011, 3:55pm

Thanks both of you.

I've managed to get the pattern.txt to be read from sdcard, and shifted it out to the shift register. (but it's all in reverse order)

Now, the only thing is that, is it possible for me to read the sdcard data index from the last to the first.

I've read the sparkfun microsd tutorial, MicroSD Shield and SD Breakout Hookup Guide - SparkFun Learn

and found this.

while(root.readDir(directory)>0){
    file.open(root, root.curPosition()/32-1, O_READ);
    in_char=file.read();              //Get the first byte in
                                        the file.
    //Keep reading characters from the file until we get an
     error or reach the end of the file. (This will output the 
     entire contents of the file).
    while(in_char >=0){            //If the value of the
                                     character is less than 0 we've
                                     reached the end of the file.
        Serial.print(in_char);    //Print the current character
        in_char=file.read();      //Get the next character
    }
    file.close();    //Close the file
    Serial.println();
}

From here, it seems like by using the sdfat library, it's possible to search for the folder index position.

I am thinking how I can use the file.read[] the same as the serial.read[].
As if there are buffers there, I might use it to point it to the last byte in the sdcard.

Where can I find more details about this? Anybody?

system · February 23, 2011, 4:08pm

The other thing, if I read the pattern.txt and save it in buffer like this,

  // copy file to serial port
  int16_t n;
  uint8_t buf[7];// nothing special about 7, just a lucky number.
  while ((n = file.read(buf, sizeof(buf))) > 0) {
    for (uint8_t i = 0; i < n; i++) Serial.print(buf[i]);
  }

Would it affect my arduino flash memory as well?
I am looking for file.read() details. It would be good if file.read(32), let say, can be implemented so that it will directly polling data at index 32 in the sdcard. and if I call another time file.read(), it will read index 31.

E.G: like now, if we call file.read(), it will +1 index, and read the next byte in the sdcard.

system · February 23, 2011, 4:16pm

uint8_t buf[7];// nothing special about 7, just a lucky number.

If you are going to read multiple bytes at a time from the file, you should read 8 bytes at a time, since the bytes are really bits, and you need 8 of them to make a real byte.

How do you define where the shifted out data does on the receiving device. Perhaps that means of defining the location could be changed, so that you read the file from top to bottom, left to right, and draw the picture from bottom to top, right to left.

Topic		Replies	Views
SD card problem with byte opperation Storage	3	1113	May 6, 2021
How to sort data in a String Programming	9	744	September 27, 2023
How to read/write binary data from a file on an SD card Programming	8	16771	May 5, 2021
can we sort a data file (.txt / .csv) from an sd card in Arduino ? Programming	6	1072	May 5, 2021
Isue with reading data of sd card General Guidance	12	521	August 28, 2022

Sorting binary data from sdcard

Related topics