Seek SD file without reading each byte

Other Hardware Storage
1

Hi,

I'm trying to use a lookup table stored on a SD card.
At the moment I works well but it's very slow, because I read each byte en parse everything until I find the correct value. Is there a way to do that in a more efficient and quick way?

Here is the reading part of my code :

do {
    lkpLat = lookup.parseFloat();
    lkpLon = lookup.parseFloat();
    lkpDeclDeg = lookup.parseFloat();
    lkpDeclRad = lookup.parseFloat();

    if (lkpLat == minLat && lkpLon == minLon)
      g0 = lkpDeclRad;
    else if (lkpLat == minLat && lkpLon == maxLon)
      g1 = lkpDeclRad;
    else if (lkpLat == maxLat && lkpLon == minLon)
      g2 = lkpDeclRad;
    else if (lkpLat == maxLat && lkpLon == maxLon)
      g3 = lkpDeclRad;

    ligne++;

  } while (lookup.available() && (lkpLat < maxLat || lkpLon < maxLon));

and here it's how my lookup table look like (just a small part of it) :

...
50;40;8;0.14
50;45;9;0.16
50;50;9;0.16
50;55;9;0.16
50;60;9;0.16
50;65;9;0.16
50;70;9;0.16
50;75;8;0.14
50;80;7;0.12
50;85;5;0.09
50;90;3;0.05
50;95;1;0.02
50;100;-1;-0.02
50;105;-3;-0.05
50;110;-6;-0.10
50;115;-8;-0.14
50;120;-10;-0.17
50;125;-11;-0.19
50;130;-12;-0.21
50;135;-12;-0.21
50;140;-11;-0.19
50;145;-10;-0.17
50;150;-9;-0.16
50;155;-7;-0.12
50;160;-5;-0.09
50;165;-2;-0.03
50;170;0;0.00
50;175;1;0.02
50;180;4;0.07
55;-180;4;0.07
55;-175;6;0.10
55;-170;8;0.14
55;-165;11;0.19
55;-160;13;0.23
55;-155;15;0.26
55;-150;16;0.28
55;-145;18;0.31
55;-140;19;0.33
55;-135;19;0.33
55;-130;19;0.33
55;-125;19;0.33
55;-120;18;0.31
55;-115;16;0.28
55;-110;13;0.23
55;-105;10;0.17
55;-100;5;0.09
55;-95;0;0.00
55;-90;-5;-0.09
55;-85;-10;-0.17
55;-80;-15;-0.26
55;-75;-18;-0.31
55;-70;-21;-0.37
55;-65;-22;-0.38
55;-60;-23;-0.40
55;-55;-22;-0.38
55;-50;-22;-0.38
55;-45;-20;-0.35
55;-40;-18;-0.31
55;-35;-17;-0.30
55;-30;-14;-0.24
55;-25;-12;-0.21
55;-20;-10;-0.17
55;-15;-8;-0.14
55;-10;-5;-0.09
55;-5;-3;-0.05
55;0;-1;-0.02
55;5;0;0.00
55;10;1;0.02
55;15;3;0.05
55;20;5;0.09
55;25;6;0.10
55;30;8;0.14
55;35;9;0.16
55;40;10;0.17
55;45;11;0.19
55;50;12;0.21
55;55;12;0.21
55;60;13;0.23
55;65;12;0.21
55;70;12;0.21
55;75;11;0.19
55;80;9;0.16
55;85;7;0.12
55;90;5;0.09
55;95;2;0.03
55;100;0;0.00
55;105;-3;-0.05
55;110;-6;-0.10
55;115;-9;-0.16
55;120;-11;-0.19
55;125;-12;-0.21
55;130;-13;-0.23
55;135;-13;-0.23
...

Actually it takes around 5sec to read until line 2000. But if the value is closer to the end of the file, it will take a lot much more time...

Please help me :slight_smile:

Thanks !

2

Hi

If you can force each line in the file to be the same length by padding record with leading records or appending blanks you can use direct file techniques to jump to known records including direct to the last, or last ten records. You will need to seek to the correct byte position in the file.

I use direct file techniques using known record numbers to retrieve descriptive information.for specific records.

If a file is sorted you can use a short series of binary searched to find ant record in say 10 lookups.

Tell me more and I will see if I can help further.

Cheers

CatweazleNZ NZ

3

AloyseTech:
Hi,

I'm trying to use a lookup table stored on a SD card.

Please help me :slight_smile:

Thanks !

CatweazleNZ:
Hi

If you can force each line in the file to be the same length by padding record with leading records or appending blanks you can use direct file techniques to jump to known records including direct to the last, or last ten records. You will need to seek to the correct byte position in the file.
Cheers

CatweazleNZ NZ

AloyseTech,
Follow what CatweazleNZ has said. Write your file usings FIXED length records.
Instead of:

fil.print(int);
file.print(',');
file.println(float);

define a structure to hold a data record, then read and write the entire record. The file will no longer be human readable, it will be a binary file.

struct datstruct{
int firstnum;
int secondnum;
int thirdnum;
float floatvalue;
};

datstruct ds;

ds.firstnum = somevalue;
ds.secondnum = somevalue;
ds.thirdnum = somevalue;
ds.floatvalue = somevalue;


file.write(ds,sizeof ds);

// to access a specific record
// you have to know the record number  to jump to it!

unsigned long recpos= recnum * sizeof ds;
file.seek(recpos);
file.read(ds,sizeof ds);

// use the data in ds.

If you need the data to be human readable, it is a little more complex, you still use fixed size entries in the file, but you have to build them yourself.

char buffer[41]; // arbitrary record length of 40 with terminating zero.
char fmt[]="%10d,%10d,%10d,%10.2";

// fillbuffer
   sprintf(buffer,fmt,firstvalue,secondvalue,thirdvalue,floatvalue);

  file.write(buffer,40);
// now the file would have fixed size (41 byte, including CR) records so you could jump around.
// and still be human readable.

depending on how you parse the file, this might be cumbersome.

Chuck.

Check out my Kickstarter Project Memory Panes an expansion RAM Shield for Mega2560's. It adds 1MB of RAM for those projects where 8KB is not enough.

4

Hi!
Sorry for the late answer...
Thanks to both of you :slight_smile:

It is possible to parse the fixed-size-human-readable entries with the struct method ?
I like the 10-spaces "container" idea.

5

AloyseTech:
Hi!
Sorry for the late answer...
Thanks to both of you :slight_smile:

It is possible to parse the fixed-size-human-readable entries with the struct method ?
I like the 10-spaces "container" idea.

Nope.
The struct method uses binary format variable, the print/println function converts from binary format to human readable.

Chuck.

Check out my Kickstarter Project Memory Panes an expansion RAM Shield for Mega2560's. It adds 1MB of RAM for those projects where 8KB is not enough.

6

Chuck,

can you please explain this code snippet:

struct datstruct{
int firstnum;
int secondnum;
int thirdnum;
float floatvalue;
};

datstruct ds;

ds.firstnum = somevalue;
ds.secondnum = somevalue;
ds.thirdnum = somevalue;
ds.floatvalue = somevalue;

file.write(ds,sizeof ds);

// to access a specific record
// you have to know the record number to jump to it!

unsigned long recpos= recnum * sizeof ds;
file.seek(recpos);
file.read(ds,sizeof ds);

// use the data in ds.

Is this programmed in Arduino IDE, or in some other IDE?

I've tried to use this code snippet, but I got a lot of error messages when tried to compile.

Thank you!

7

metheoneandonly:
Chuck,

can you please explain this code snippet:

/* the following statements define a data structure,  the dataStructure has 4 elements, 3 int's, one Float.
the name of the structure is [b]datastruct[/b]
*/

struct datstruct{
int firstnum;
int secondnum;
int thirdnum;
float floatvalue;
};

/* this statement creates a memory instance of the structure with then name of [b]ds[/b]
*/

datstruct ds;

/* these statements show you how to access each individual value of the structure.
*/

ds.firstnum = somevalue;
ds.secondnum = somevalue;
ds.thirdnum = somevalue;
ds.floatvalue = somevalue;


/* this is an example of using the structure to have the SD library save the record (structure) to the SDcard.
*/

file.write(ds,sizeof ds);

// to access a specific record
// you have to know the record number  to jump to it!
unsigned long recpos= recnum * sizeof ds;
file.seek(recpos);
file.read(ds,sizeof ds);

/* file.seek(); moves the 'file pointer' to the n'th byte of the file.
   I had a file on the SDcard that was composed of theses records, and I wanted to read the
   15th record from the file I would use the following code.
*/

// calculate where the data for record 15 starts in the file
 
unsigned long recpos = 15 * sizeof(ds); // either sizeof ds; or sizeof(ds); one of these is evaluated during the compile stage, the other calls a function during runtime.

// move the 'file pointer' to this calculated offset, file needs to already be open for read.

file.seek(recpos);

// now to actually read the data in

file.read(ds,sizeof(ds));

// use the data in ds.

metheoneandonly:
Is this programmed in Arduino IDE, or in some other IDE?

I've tried to use this code snippet, but I got a lot of error messages when tried to compile.

Here is a complete program to read from the sd card. Modified from the Examples included with the IDE

Chuck.

rw.ino (4.8 KB)

8

Thank you very much, now I'll study it. :slight_smile:

9

OK, now I have an idea how it works. I need help with the following: is it possible to read/write such kind of file on SD card with, say, Visual Basic 2012 and then put a SD card into card reader/shield to read those files with Arduino? I need it for my project to simplify and accelerated adding data to be used with Arduino. Or maybe just to to a serial communication between VB2012 and Arduino, so the Arduino writes those files by itself?

Thanks in advance!

10

Yes, a file written using a program in any language on any other operating system can potentially be read by a another program written in another language on another operating system after you have moved, copied or transmitted the file.

You just need to ensure that you understand the format of the data written into the file will be readable by the program that reads the file later.

I maintain text files on my Windows 8 machine and then upload them to my Arduino system which successfully reads and processes them without any issues.

Cheers

Catweazle NZ

11

Well, that's the point: I don't understand the format Arduino is writing to. Chucktodd said that the struct method uses binary format variable. He also gave me an example on how to use it and as I see this works perfectly with Arduino, it prints such data correctly on the LCD, but how to read/write this particular format with VB2012? I'm still newbie with programming Arduino and related, So I need a lot of help to get a good grip with it. It seems I'll make serial connection between Arduino and VB2012 app to send data to be written on SD by Arduino. Or, maybe you can a least put here some directions on how to solve my problem?

Thanks in advance!

12

metheoneandonly:
Well, that's the point: I don't understand the format Arduino is writing to. Chucktodd said that the struct method uses binary format variable. He also gave me an example on how to use it and as I see this works perfectly with Arduino, it prints such data correctly on the LCD, but how to read/write this particular format with VB2012? I'm still newbie with programming Arduino and related, So I need a lot of help to get a good grip with it. It seems I'll make serial connection between Arduino and VB2012 app to send data to be written on SD by Arduino. Or, maybe you can a least put here some directions on how to solve my problem?

Thanks in advance!

Writing Binary files in Visual Basic
Visual Basic structures
Visual Basic Data types

Private Structure arduinorec
  Private firstnum As Short
  Private secondnum As Short
  Private thridnum As Short
  Private floatvalue As Single
End Structure

My.Computer.FileSystem.WriteAllBytes(
  "C:\MyDocuments\CustomerData", arduinorec, True)

lookup how to position the file pointer, Read a block into the record (structure), Open the file, Close the file.

build from here.

chuck.

13

Wow, Chuck, thank you very much! Now I have a good home work to do!