Extended Database Library & SD-Card / strange shifting on database load from SD

Dear community

After searching the net and this forum I need your help!

I am able to save a database to a Micro-SD card. When I create and save the database, everything seems to be working fine.
But when I just load the database I got some strange behaviour.

Example with demo-data. Date and time are created randomly, Timeframe is 1.

ID: 1
Date: 2009-11-20
Time: 01:18:46
Timeframe 1: 1
ID: 2
Date: 2009-04-25
Time: 01:32:07
Timeframe 1: 1
ID: 3
Date: 2009-07-06
Time: 11:38:05
Timeframe 1: 1

After resetting the board and just loading the database from the sd-card i get the following:

ID: 1
Date: 

Time: 009-11-20
Timeframe 1: 12288
ID: 2
Date: 1:18:46
Time: 009-04-25
Timeframe 1: 12288
ID: 3
Date: 1:32:07
Time: 009-07-06
Timeframe 1: 12544

Somehow it is shifted?

Here the complete code:

#include "Arduino.h"
#include <EDB.h>
#include <EEPROM.h>
#include <SD.h>
#include <string.h>
 
File dbFile;
 
#define TABLE_SIZE 512
#define RECORDS_TO_CREATE 10
 
struct LogEvent {
  int id;
  int temperature;
}
logEvent;

struct Transponder {
  char date[11];
  char time[9]; 
   int tf1;
} transponder;

int recno;
 
void writer(unsigned long address, byte data)
{
  dbFile.seek(address);
  dbFile.write(data);
  dbFile.flush();
}
 
byte reader(unsigned long address)
{
  dbFile.seek(address);
  return dbFile.read();
}

EDB db(&writer, &reader);
 
void setup()
{
  Serial.begin(9600);
  Serial.print("Initializing SD card...");
  pinMode(10, OUTPUT);
  
  if (!SD.begin()) {
    Serial.println("initialization failed!");
    return;
  }
  
  Serial.println("initialization done.");
  //SD.remove("transp.db"); //just for testing to delete db and recreate
  Serial.println("Opening transp.db ...");
  if (SD.exists("transp.db")) {
    Serial.println("transp.db exists. Open it...");
    dbFile = SD.open("transp.db");
    db.open(0);
  }
  else {
    Serial.println("transp.db doesn't exist. Create it...");
    dbFile = SD.open("transp.db", FILE_WRITE);
    db.create(0, TABLE_SIZE, sizeof(transponder));
    Serial.println("Creating Records...");
    for (recno = 1; recno <= RECORDS_TO_CREATE; recno++)
    {
      int m = random(1, 12);
    int d = random(1, 31);
    int h = random(1, 12);
    int i = random(59);
    int s = random(59);
    sprintf(transponder.date, "2009-%02d-%02d", m, d);
    sprintf(transponder.time, "%02d:%02d:%02d", h, i, s);

      transponder.tf1 = 1;
      db.appendRec(EDB_REC transponder);
    }
  }

  Serial.print("Record Count: "); Serial.println(db.count());
  for (recno = 1; recno < db.count(); recno++)
  {
    db.readRec(recno, EDB_REC transponder);
    Serial.print("ID: "); Serial.println(recno);
    Serial.print("Date: "); Serial.println(transponder.date); 
    Serial.print("Time: "); Serial.println(transponder.time);
    Serial.print("Timeframe 1: "); Serial.println(transponder.tf1); 
  }


  dbFile.close();
  delay(2000); //Test database with reopen...
  Serial.println("Opening transp.db ...");
  if (SD.exists("transp.db")) {
    Serial.println("transp.db exists. Open it...");
    dbFile = SD.open("transp.db");
    db.open(0);
  }
  Serial.print("Record Count: "); Serial.println(db.count());
  for (recno = 1; recno < db.count(); recno++)
  {
    db.readRec(recno, EDB_REC transponder);
    Serial.print("ID: "); Serial.println(recno);
    Serial.print("Date: "); Serial.println(transponder.date); 
    Serial.print("Time: "); Serial.println(transponder.time);
    Serial.print("Timeframe 1: "); Serial.println(transponder.tf1); 
  }
}

void loop()
{

Any idea? I saw that there is no command implemented to actually close the database?

Thanks a lot in advance
Dave

  for (recno = 1; recno < db.count(); recno++)
  {
    db.readRec(recno, EDB_REC transponder);

Are the records really numbered from 1? C/C++ starts at 0 for arrays.

Where do you actually read from the SD file, if it exists?

good point.. I took this from an example :)

If the file "transp.db" exists, i open it here:

 if (SD.exists("transp.db")) {
    Serial.println("transp.db exists. Open it...");
    dbFile = SD.open("transp.db");
    db.open(0);
  }

I'm playing around with the db-creation command, perhaps the shifting occurs because of a wrong length?

db.create(0, TABLE_SIZE, sizeof(transponder));

Now i created the database with db.create(0, TABLE_SIZE, sizeof(transponder)-3);and did get the following when loading the database after reset:

ID: 1
Date: 
Time: 

Timeframe 1: 0
ID: 2
Date: 001:32:07
Time: 2009-11-2001:32:07
Timeframe 1: 0
ID: 3
Date: 511:38:05
Time: 2009-04-2511:38:05
Timeframe 1: 0

So somehow some the content of a database entry gets mixed up?

If the file "transp.db" exists, i open it here:

But, you never read it.

I'm playing around with the db-creation command, perhaps the shifting occurs because of a wrong length?

No. You are creating a database with some number of rows, but you are NOT populating the rows with data.

So somehow some the content of a database entry gets mixed up?

There is nothing to get mixed up. You haven't populated the rows in the table.

Thanks PaulS

In my opinion I read it here:

Serial.print("Record Count: "); Serial.println(db.count());
  for (recno = 1; recno < db.count(); recno++)
  {
    db.readRec(recno, EDB_REC transponder);
    Serial.print("ID: "); Serial.println(recno);
    Serial.print("Date: "); Serial.println(transponder.date); 
    Serial.print("Time: "); Serial.println(transponder.time);
    Serial.print("Timeframe 1: "); Serial.println(transponder.tf1); 
  }

i tried starting at 0, but this didn’t work.

So how do i populate the data?

I am filling the rows here i think:

Serial.println("Creating Records...");
    for (recno = 0; recno <= RECORDS_TO_CREATE; recno++)
    {
      int m = random(1, 12);
    int d = random(1, 31);
    int h = random(1, 12);
    int i = random(59);
    int s = random(59);
      sprintf(transponder.date, "2009-%02d-%02d", m, d);
    sprintf(transponder.time, "%02d:%02d:%02d", h, i, s);
      transponder.tf1 = 1;
      db.appendRec(EDB_REC transponder);
    }

I went back to the beginning to this example with just an ID and an int value per database entry:
http://blog.brauingenieur.de/2014/01/20/extended-database-library-using-an-sd-card/

I let it run once, then switched from db.create to db.open. Then the ID showed the int-value and vice versa.
So perhaps there is something wrong with the library or what am I doing wrong?

Isn't there anyone using the extended database library with a SD-card?

This definitely seems to be too much for the arduino. I just ordered a Raspberry Pi, so I'm going to connect it to the arduino and use SQLlite for a real database layout. The extended database library with SD card support doesn't seem to be bugfree.

I tried the project with the SD slot of the Ethernet Shield. The output of the COM Port is:


Initializing SD card…initialization done.
Opening transp.db …
transp.db exists. Open it…
Record Count: 10
ID: 1
Date:

Time: 009-11-20
Timeframe 1: 12288
ID: 2
Date: 1:18:46
Time: 009-04-25
Timeframe 1: 12288
ID: 3
Date: 1:32:07
Time: 009-07-06
Timeframe 1: 12544
ID: 4
Date: 1:38:05
Time: 009-08-08
Timeframe 1: 12288
ID: 5
Date: 6:33:37
Time: 009-07-20
Timeframe 1: 12288
ID: 6
Date: 2:30:06
Time: 009-10-10
Timeframe 1: 12288
ID: 7
Date: 6:12:49
Time: 009-09-27
Timeframe 1: 12288
ID: 8
Date: 1:41:58
Time: 009-10-20
Timeframe 1: 12288
ID: 9
Date: 9:25:08
Time: 009-08-03
Timeframe 1: 12544
Opening transp.db …
transp.db exists. Open it…
Record Count: 10
ID: 1
Date:

Time: 009-11-20
Timeframe 1: 12288
ID: 2
Date: 1:18:46
Time: 009-04-25
Timeframe 1: 12288
ID: 3
Date: 1:32:07
Time: 009-07-06
Timeframe 1: 12544
ID: 4
Date: 1:38:05
Time: 009-08-08
Timeframe 1: 12288
ID: 5
Date: 6:33:37
Time: 009-07-20
Timeframe 1: 12288
ID: 6
Date: 2:30:06
Time: 009-10-10
Timeframe 1: 12288
ID: 7
Date: 6:12:49
Time: 009-09-27
Timeframe 1: 12288
ID: 8
Date: 1:41:58
Time: 009-10-20
Timeframe 1: 12288
ID: 9
Date: 9:25:08
Time: 009-08-03
Timeframe 1: 12544


I cannot open the “TRANSP.DB” file to see what is writen.
Can you please tell me how to open it?

Here is the solution just in case someone comes across this thread.

In EDB.cpp the open function never sets the value of the table pointer, this causes the table to be read thinking that the first record is at 0. Here is the fix.

EDB_Status EDB::open(unsigned long head_ptr)
{
  EDB_head_ptr = head_ptr;
  readHead();
  EDB_table_ptr = sizeof(EDB_Header) + EDB_head_ptr;
  return EDB_OK;
}