Looking for help parsing data into a struct

I am still new to C so I am working on the assumption that I dont know something. Any help and pointers would be greatly appreciated.

I have a struct that contains configuration data. It consists of a couple of booleans, a bunch of Byte types, and three char arrays.

I am trying to figure out how to take the data from a HTTP POST and put it into the correct parts of the struct, without using all my Flash memory. lol

So the data comes in as a String that contains "data1=on&data2=off&byte1=128&byte2=256" etc.

The order is fixed, so it will always contain the same field=data in the same order. I need to parse that data out and put it into the correct places in the struct.

For example: data1=on goes into storage.data1 which is a boolean.

I started by indexing the String on "&" and then passing the entry to a function that indexes that String on the "=" followed by running it through a ton of IF statements. However that is very inefficient and sucks up memory.

Is there a programmatic way to do this?

So, to add more data to it.

struct Configuration {
  char version[4];
  bool dhcp;
  bool ctlpage;
  byte ip[4];
  byte gw[4];
  byte nm[4];
  byte dns[4];
  char hostname[9];
  char user1[22];
  char user2[22];
}

The received HTTP POST would be something along the lines of

"dhcp=on&ctlpage=off&ip1=192&ip2=168&ip3=1&ip4=120"etc,etc. continued out for all the entries.

I am doing this on an Arduino Mini (ATMega328)

Thanks for all the help and/or pointers.

So the data comes in as a String

No, it does not. It comes in as a series of characters. Quit stuffing them in a String. Use a NULL terminated array of chars, instead. That approach uses far less memory, AND makes using strtok() and atoi() possible and easy.

Again, Still real new to this but let me see if I understand what you are saying.

Rather than using a String, which is what all the example code I found uses, I should use a char array and strtok() to parse it.

so the code to capture the request would have to be changed to something like

char client_req[255];

 if (client.available()) {
        char c = client.read();
        strcat(client_req, c);
}

Instead of the examples that I found which use

String client_req;

    if (client.available()) {
        char c = client.read();
        client_req += c;
}

That being the case, I would have to rework all the "client_req.startsWith()" lines.

Is there a place that lists all the functions like strcat() so that I can determine what functions to use in place of the String functions?

Digging around and trying to think of a different way to do it I came up with this. Not complete code, just my though process of identifying that it is first a POST request, then when the later line comes in how to divide it up.

bool post
char data;
data = strtok(HTTP_req, " ");
if ( data == "POST" ) { 
post == true;
};
if ( data == "GET" ) { };
if (post == true && strtok(strtok(HTTP_req, "&"), "=") == dhcp) {
 //break up the Post data line
 post == false;
}

NOTE: I don't mind reworking all of it. I am working on learning both coding in C as well as the most efficient way to handle the work in the limited 30k flash and 2k SRAM.

I use this CplusPlus website for info on functions.

If you look at serial input basics you will see how to add characters into a char array and how to parse using strtok()

...R

Robin,

Thank you for the links. After a few hours of reading and trying stuff I did get it to break up. Now I just need to figure out how to stuff it into the struct. ;)

Pulling the data in via Serial for testing I came up with this to break up the data and print it. That is a lot closer.

void break_up_data() {
  char *p, *i;
  Serial.println((unsigned)strlen(receivedChars));
  p = strtok_r(receivedChars, "=, &", &i);
  while (p) {  
    Serial.print(p);
    Serial.print(" == ");
    p = strtok_r(NULL, "=, &", &i);
    Serial.print(p);
    Serial.println("");
    p = strtok_r(NULL, "=, &", &i);
    }
}

I then fed it the following line to test

dhcp=on&ctlpage=on&ip1=192&ip2=168&ip3=111&ip4=125&nm1=255&nm2=255&nm3=255&nm4=128&gw11=192&gw2=168&gw3=111&gw4=126&dns1=192&dns2=168&dns3=111&dns4=122&user1=123456789012345678901&user2=123456789012345678901

Serial.println((unsigned)strlen(receivedChars));

Why do you need to cast an unsigned int to an unsigned int? Is there ANY way that the length of a string can be less than 0? Of course not, so strlen() doesn't return an int.

Even if it did, there is no reason to cast it to unsigned to print it.

Who suggested using strtok_r()? No one in this thread, and for good reason. The strtok_r(0 version is far more complex, and is only needed in the case where multiple threads might be acting on the same block of memory. On an Arduino, with its single thread, that just is NOT possible.

HI paul,

First, I am a total newbie to Arduino C. Before this program I only did simple reads from sensors and output to serial or LCD display. To be honest, I am still trying to figure out what “char *p” means. I know it is a char type but why the *?

Why do you need to cast an unsigned int to an unsigned int? Is there ANY way that the length of a string can be less than 0? Of course not, so strlen() doesn’t return an int.

I found it as an example online and just copied and pasted it in with a change in the string to get the total count of characters in the line so I could adjust the “char recievedChars[255]” to just what was needed for the maximum length of the input.

strtok_r() was used in the examples provided by robin2 in the link serial input basics.

My current working test code is (Feeding it the same HTTP POST line I listed earlier.)

const byte numChars = 209;
char receivedChars[numChars];	// an array to store the received data
boolean newData = false;

#define CONFIG_VERSION "cf1"

struct Configuration {
  char version[4];
  bool dhcp;
  bool ctlpage;
  byte ip[4];
  byte gw[4];
  byte nm[4];
  byte dns[4];
  char hostname[9];
  char user1[22];
  char user2[22];
} storage = {
  CONFIG_VERSION,
  1,
  0,
  {10, 1, 1, 3},
  {10, 1, 1, 1},
  {255, 255, 255, 0},
  {10, 1, 1, 2},
  "hostname",
  "YWRtaW46YWRtaW4xMjMK",
  "dXNlcjp1c2VyMTIzCg=="
};


void setup() {
  Serial.begin(9600);
  Serial.println(F("<Arduino is ready>"));
  Serial.println(F("Current Config is"));
  printconfig();
  Serial.println();
  Serial.println(F("Ready to recieve new config"));
  Serial.println();
}

void loop() {
  recvWithEndMarker();
  showNewData();
}

void recvWithEndMarker() {
  static byte ndx = 0;
  char endMarker = '\n';
  char rc;
  
  while (Serial.available() > 0 && newData == false) {
    rc = Serial.read();

    if (rc != endMarker) {
      receivedChars[ndx] = rc;
      ndx++;
      if (ndx >= numChars) {
        ndx = numChars - 1;
      }
    }
    else {
      receivedChars[ndx] = '\0'; // terminate the string
      ndx = 0;
      newData = true;
    }
  }
}

void showNewData() {
  if (newData == true) {
    Serial.println(receivedChars);
    break_up_data();
    newData = false;
    printconfig();
  }
}

void break_up_data() {
  char *p, *i;
  char num[1];
  int a = 1;
  p = strtok(receivedChars, "=, &");
  while (p) {
    p = strtok(NULL, "=, &");
    if ( a == 1 ) storage.dhcp = atoi(p);
    if ( a == 2 ) storage.ctlpage = atoi(p);
    if ( a == 3 || a == 4 || a == 5 || a == 6 ) storage.ip[a - 3] = atoi(p);
    if ( a == 7 || a == 8 || a == 9 || a == 10) storage.nm[a - 7] = atoi(p);
    if ( a == 11 || a == 12 || a == 13 || a == 14) storage.gw[a - 11] = atoi(p);
    if ( a == 15 || a == 16 || a == 17 || a == 18) storage.dns[a - 15] = atoi(p);
    if ( a == 19 ) strcpy(storage.user1, p);
    if ( a == 20 ) strcpy(storage.user2, p);
    a++;
    p = strtok(NULL, "=, &");
  }
}

void printconfig() {
  // Print the settings
    Serial.print(F("Dhcp is: "));
    Serial.println( true == storage.dhcp ? "On" : "Off");
    Serial.print(F("Control Page login is: "));
    Serial.println( true == storage.ctlpage ? "On" : "Off");
    Serial.print(F("IP: "));
    Serial.print(storage.ip[0]);
    Serial.print(F("."));
    Serial.print(storage.ip[1]);
    Serial.print(F("."));
    Serial.print(storage.ip[2]);
    Serial.print(F("."));
    Serial.println(storage.ip[3]);
    Serial.print(F("Netmask: "));
    Serial.print(storage.nm[0]);
    Serial.print(F("."));
    Serial.print(storage.nm[1]);
    Serial.print(F("."));
    Serial.print(storage.nm[2]);
    Serial.print(F("."));
    Serial.println(storage.nm[3]);
    Serial.print(F("Gateway: "));
    Serial.print(storage.gw[0]);
    Serial.print(F("."));
    Serial.print(storage.gw[1]);
    Serial.print(F("."));
    Serial.print(storage.gw[2]);
    Serial.print(F("."));
    Serial.println(storage.gw[3]);
    Serial.print(F("DNS: "));
    Serial.print(storage.dns[0]);
    Serial.print(F("."));
    Serial.print(storage.dns[1]);
    Serial.print(F("."));
    Serial.print(storage.dns[2]);
    Serial.print(F("."));
    Serial.println(storage.dns[3]);
    Serial.print(F("Versions:  "));
    Serial.print(CONFIG_VERSION);
    Serial.print(F("=="));
    Serial.print(storage.version);
    Serial.println(F(".."));
    Serial.print(F("Hostname: "));
    Serial.println(storage.hostname);
    Serial.print(F("User1 Login:  "));
    Serial.println(storage.user1);
    Serial.print(F("User2 Login:  "));
    Serial.println(storage.user2);
  }

I am always open to pointers, suggestions, and any useful additions or improvements. It is a learn as I go thing and I have to admit it is a blast getting it working. :smiley:

The only languages I learned before this was BASIC (back on the C64), pascal (On the 286/386), COBOL (On an IBM Mainframe), Bash and Perl scripts which I use all the time.

COBOL (On an IBM Mainframe)

You are ruined, then. Forget trying to learn anything modern. 8)

strtok_r() was used in the examples provided by robin2 in the link serial input basics.

Hey, Robin, you really should fix that. strtok() is a much smaller, easier to use, function than strtok_r(), and is all that is needed on the Arduino.

CyberLink1:
Thank you for the links. After a few hours of reading and trying stuff I did get it to break up. Now I just need to figure out how to stuff it into the struct. :wink:

I don’t understand why you can’t use the parse code (in serial input basics ) with a very small modification to write the data to your struct?

…R

CyberLink1: Is there a place that lists all the functions like strcat() so that I can determine what functions to use in place of the String functions?

According to http://www.arduino.cc/en/Reference/HomePage

The Arduino language is based on C/C++. It links against AVR Libc and allows the use of any of its functions; see its user manual for details.

Follow the links, And you'll find http://www.nongnu.org/avr-libc/user-manual/modules.html and in particular http://www.nongnu.org/avr-libc/user-manual/group__avr__string.html .

PaulS:

COBOL (On an IBM Mainframe)

You are ruined, then. Forget trying to learn anything modern. 8)

Hey, Paul, I used to use Cobol (not on IBM though).

I converted. :)

Perhaps this isn't the moment to mention Fortran.

Or machine language. Never mind those namby-pamby assemblers for people who can't read straight hex codes. ;)

Robin2: I don't understand why you can't use the parse code (in serial input basics ) with a very small modification to write the data to your struct?

...R

Robin,

Is there something wrong with the function that I wrote?

It is based on the ideas and concepts expressed in your examples.

void break_up_data() {
  char *p, *i;
  char num[1];
  int a = 1;
  p = strtok(receivedChars, "=, &");
  while (p) {
    p = strtok(NULL, "=, &");
    if ( a == 1 ) storage.dhcp = atoi(p);
    if ( a == 2 ) storage.ctlpage = atoi(p);
    if ( a == 3 || a == 4 || a == 5 || a == 6 ) storage.ip[a - 3] = atoi(p);
    if ( a == 7 || a == 8 || a == 9 || a == 10) storage.nm[a - 7] = atoi(p);
    if ( a == 11 || a == 12 || a == 13 || a == 14) storage.gw[a - 11] = atoi(p);
    if ( a == 15 || a == 16 || a == 17 || a == 18) storage.dns[a - 15] = atoi(p);
    if ( a == 19 ) strcpy(storage.user1, p);
    if ( a == 20 ) strcpy(storage.user2, p);
    a++;
    p = strtok(NULL, "=, &");
  }
}

Hmm I just noticed there are a couple of char declared that I am not using. Other than that it appears ok to me.

CyberLink1: Is there something wrong with the function that I wrote?

May I turn that around and ask is there something wrong with the function I wrote?

You seem to be asking me to spend time writing another piece of code to do the same thing. It is much more time-efficient for me to help you with code I am already familiar with. Which leaves more time to help someone else as well as you.

...R

    if ( a == 3 || a == 4 || a == 5 || a == 6 ) storage.ip[a - 3] = atoi(p);

vs:

if( a>= 3 && a <= 6)
   storage.ip[a - 3] = atoi(p);

Which is easier to understand?

Robin2: May I turn that around and ask is there something wrong with the function I wrote?

You seem to be asking me to spend time writing another piece of code to do the same thing. It is much more time-efficient for me to help you with code I am already familiar with. Which leaves more time to help someone else as well as you.

...R

Well, your parse routine is great and I appreciate the work you have done and the time you have spent pointing me to the information I was looking for.

First, to answer your question, your parse uses a delimiter of "," and I am breaking an HTTP POST line that contains string=value pairs separated by an &.

Second, I am learning how it works. To me that facilitates the need to write it based on what I have learned and not to just copy examples.

Your post and what I learned from the information and examples you provided, cut 10k of program storage out of my code. You have my deepest gratitude for the help you have provided.

To PaulS,

I did not even think of doing that. Thank you, it is much easier to read.

CyberLink1: Well, your parse routine is great and I appreciate the work you have done and the time you have spent pointing me to the information I was looking for.

First, to answer your question, your parse uses a delimiter of "," and I am breaking an HTTP POST line that contains string=value pairs separated by an &.

I would run through the parse process twice. First time I would divide things based on & following which I would divide the resulting parts based on =.

...R

Robin2: I would run through the parse process twice. First time I would divide things based on & following which I would divide the resulting parts based on =.

...R

I tried that with my first iteration of the routine. Before I found out that strtok could do it in a single line by identifying multiple delimiters.

The way I did it at first was to try to call strok on the output from strok, that did not work at all. I then did a loop to put the data into variables the ran a second loop to split the data and attach it to the correct part of the struct. That worked but the alternate design I used above required less program storage.

CyberLink1: I tried that with my first iteration of the routine. Before I found out that strtok could do it in a single line by identifying multiple delimiters.

Interesting. I did not know it would do that - I'm no C/C++ expert (I avoid it if I can :) )

Doing the two at once seems to introduce a risk if the incoming data was incorrect that everything would get out of step without the ability to recover - for example if there was a single = or & missing.

...R