Help converting string data to struct... (solved!)

I'm reading from a somewhat large database (60k lines) stored in SPIFFS. It works well but I'm worried about efficiency and longterm stability, since my current method is to read the lines as String and then convert to structs. I'm wondering if there's a better way to do it?

My struct looks like this:

struct event {
  long seconds;
  char event_type; 
  float amount; 
};

Here's a sample few lines of my database file. I can format this anyway I want, if someone sees an improvement that could be made:

1480025589, Z, 35
1480026671, Z, 67
1480027464, F, 13
1480031512, R, 59
1480033603, F, 70
1480036092, R, 65
1480038692, Z, 87
1480045906, B, 15
1480047890, F, 54
1480050079, R, 10
1480050782, F, 3
1480051422, R, 35
1480055002, Z, 94
1480057014, R, 8
1480061853, Z, 53
1480062209, Z, 98
1480065485, B, 77
1480068269, F, 66
1480069687, F, 70
1480073558, Z, 61
1480081914, F, 72
1480083688, Z, 92
1480090945, B, 10
1480092367, R, 25
1480095948, F, 78
1480101772, B, 77
1480117751, Z, 40
1480119775, R, 37
1480121594, R, 53
1480123867, B, 94
1480127078, Z, 83
1480128373, R, 99
1480130635, R, 83
1480130705, Z, 38
1480147828, R, 62
1480148299, B, 31
1480149840, F, 37
1480150351, F, 75
1480155255, Z, 97
1480156190, R, 85
1480158097, R, 40
1480160837, R, 42

I'm currently iterating over each line in the DB, and stopping when I find a certain value. I then need to create an event (struct) for that line.

[Edited: see post #13 of this thread for the current issue]

You seem to be confusing strings and Strings.

Also, why is “amount” a float?

Good point about float, that should be an int.

I'm now trying to keep things as strings (lowercase).

I will repeat the comment I made in your earlier Thread on this subject, and which you have not yet responded to

I'm looking up the current time in the database to see if there's a record attached to it.

How do you envisage relating the current time to a record in the database since, by definition, the database must have been written when what is now the current time was some distance in the future?

And if you do have a text database with 1.6MB of data wouldn't it be an awful lot easier to process it with Python on an RPi? And if the data has some structure to it you could make life easier still by using a database program like SQLite.

...R

I know this would be easier on a Pi, but there are other considerations here.

The data in the DB is pre-generated since it's more complicated than would be possible on an Arduino.

You really do not have a dB, you have a flat file. Which microController are you using to get 60K SPIFF'?

Which microController are you using to get 60K SPIFF'?

Sorry, I should have mentioned, I'm using a Melife ESP32.

Will you be moving all the SPIFF's to structures or just a single structure that is used to store the data when read?

Are you using the Wear Levelling API? https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/storage/wear-levelling.html

Are you using the ESP32 OS, freeRTOS?

Are the SPIFF's received OTA?

For the moment I’m trying to keep things as simple as possible. Just reading the data and processing it.

Edited: removed an outdated code sample.

struct event {
  long seconds;
  char event_type;
  int amount;
};
setup()
{
log_i("size of structure: %d", sizeof(struct event) );
}

shows that the event structure does not take much ram. I'd consider a structure a good way to go. I'd put the management of the SPIFF file on one core and the data processing of the file on the other core; especially if you are not running all the other features of the ESP32. setup() and loop() run on core 1. By using freeRTOS you can put your tasks on different cores.

Thanks for that. That definitely sounds like the way to go eventually. In the meantime I don’t need to read the data very often (maybe once every few days) so I think the main processor can handle it, but we’ll see.

I’ve boiled my current issue down to the following sketch. At the top is the line:

char test_data[22]=“1477340278, F, 43.5”;

Which I’m trying to parse to the following struct:

struct event {
long seconds;
char event_type;
float amount;
};

I’ve successfully extracted the seconds (1477340278), but am having trouble extracting the event_type as a char (‘F’), and the amount (43.5) as a float. I don’t imagine anyone has any hints? I can easily do it by converting to Strings (capital S) using substring, but I’m trying to keep this as efficient as possible.

// the data that will be parsed into a struct below
char test_data[22]="1477340278, F, 43.5";

// the struct that will hold the data
struct event {
  long seconds;
  char event_type;
  float amount;    
};



void setup() {
   Serial.begin(115200);
}

void loop() {

  delay(500);

  char db_seconds_char[10];
  int db_seconds;
  char db_event_type;
  float db_amount;
  event test_event;
  
  // get the first 10 characters of the line, which is the seconds
  memcpy( db_seconds_char, test_data, 10 );

  // convert the db_seconds_char to int
  db_seconds = atoi(db_seconds_char);

  Serial.println(db_seconds);

  test_event.seconds=db_seconds;

  /*
  // how to get the event_type ('F') and amount (43.5)?
  test_event.event_type=db_event_type;
  test_event.amount=db_amount;

  if (db_event_type=='F') Serial.println("yes!");
  */

}

As long as that format is fixed width for the first two fields you can just do this:

// the data that will be parsed into a struct below
char test_data[22] = "1477340278, F, 43.5";

// the struct that will hold the data
struct event
{
  long seconds;
  char event_type;
  float amount;
};

void setup()
{
  Serial.begin(115200);
}

void loop()
{
  event test_event;
  test_event.seconds = atol(test_data);
  Serial.println(  test_event.seconds);
  test_event.event_type = test_data[12];
  Serial.println(test_event.event_type);
  test_event.amount = atof(&test_data[14]);
  Serial.println(test_event.amount);
  delay(1000);
}

If not, take a look at strtok.

Thank you! Works beautifully.

I'm now having a problem using strtok, so to keep it all in one place I thought I'd continue this thread.

I'm trying to process a line that looks like this:

43k21,Some Long Unpredictable Name, M,1.2,55

And convert it to a struct defined below. At the end it should look like:

itemID=43k21 itemName=Some Long Unpredictable Name itemType=M itemOffset=1.2 itemHeight=55

When compiling this code I get the error "incompatible types in assignment of 'char*' to 'char [6]'". Any advice for making that conversion? Or to set the code up differently?

struct item {
  char itemID[6];
  char itemName;
  char itemType[1];
  float itemOffset;
  int itemHeight;
};


void setup() {

  Serial.begin(115200);

  char array[] = "43k21,M,1.2,55";
  char *ptr = NULL;
  byte index=0;
  char *strings[100];
  while(ptr != NULL) {
    strings[index] = ptr;
    index++;
    ptr = strtok(NULL, ",");  // takes a list of delimiters
  }  

  item number1;
    
  number1.itemID=strings[0];
  number1.itemName=strings[1];
  number1.itemType=strings[2;
  number1.itemOffset=strings[3];
  number1.itemHeight=strings[4];

  Serial.println(current.itemID);
  
}

void loop() {
  // put your main code here, to run repeatedly:

}

You will need to use strcpy to populate your struct. Simple assignment won't do it.

Thanks for that. For anyone else coming this way, I pasted the updated code below. Use atof for floats, and atoi for integers.

My final hurdle is comparison with the char[1] type. This line gives the error "ISO C++ forbids comparison between pointer and integer [-fpermissive]":

if (number1.itemType=='M') Serial.println("Yes");

Any tips on making this comparison?

struct item {
  char itemID[6];
  char itemType[1];
  float itemOffset;
  int itemHeight;
};


void setup() {

  Serial.begin(115200);

  char array[] = "43k21,M,1.2,55";
  char *ptr = NULL;
  byte index=0;
  char *strings[200];
  ptr = strtok(array, ","); 
  while(ptr != NULL) {
    strings[index] = ptr;
    index++;
    ptr = strtok(NULL, ",");  // takes a list of delimiters
  } 

  item number1;
  //Serial.println(strings[0]);
  strcpy(number1.itemID,strings[0]);
  strcpy(number1.itemType,strings[1]);

  Serial.println(number1.itemType);

  if (number1.itemType=="M") Serial.println("Yes");
  //if (number1.itemType=='M') Serial.println("Yes2");
  //if ( !strcmp(number1.itemType, 'M') ) Serial.println("Yes3");

}

void loop() {
}

Use strcmp.
Or if you’'re only comparing single characters, use a subscript.
Or a single character, not an array.

Thank you, works perfectly.