char array to string array problem

Hey guys,

I'm writing a data parser that is converting and transmitting a large text files of comma-separated instructions (100-10,000 instructions per file) to an Arduino for execution on phsyical hardware. The instructions each occupy a line in the text file, ending with a '\n' but also starting and ending with special characters. FYI they look like this '<4,4294,8269,4181,9150,0,;'

I've written a Processing program that segments the file into chunks of 4-10 instructions which are transmitted over Serial and interterpreted by the Arduino. The processing part is going fine but I'm running into some issues in Arduino.

Essentially, I do my parsing by reading Serial data into a char array buffer while using the start and end characters of each instruction to copy the contents into entries in a string array, this action resets the buffer. This is convienient as I can later split and execute the contents of each array entry in another section of the code and can keep the buffer relatively small in the case that I need to transmit a bunch of instructions in one go.

This sounds like it should work but when I try and print the string array out of the arduino serial buffer I just get empty lines. I know that the data is being received as the buffer shows the correct contents and the arduino is counting the number of instructions correctly.

I'm wondering if there are some issues copying from the char array to a string (i.e., I can't use strcpy()) and was considering using a 2 dimensional char arrays (a list of arrays is what I'm after), but couldn't seem to get the syntax correct (or find an example) for copying the data in the form.

char c2dArray[30][200];
char buf[200] = "<4,4294,8269,4181,9150,0,;";
c2dArray[1][1:200] = buf; // copy buffer contents to entry one, using all the characters necessary

Any help and suggestions appreciated. This is driving me nuts! :astonished:

Here's my parsing function:

String lines[30];            // array of instructions from incoming serial data
char buf[200];             // a place to store incoming data

    case 0:              // waiting for data / recieving data mode    
      while (Serial.available()>0 && i < 200)  {    // if there is data and we're not exceeding the index
        char incomingByte = Serial.read();        // read the first byte of the message

         // store the incoming char in buffer
         buf[i] = incomingByte;  
         buf[i+1] = '\0';  // keeps the string null terminated       
         
         if (incomingByte == '>'){      // start of normal instructions
             num_commands++;            // increment number of instructions
         }
         
         if (incomingByte == '<'){     // start of last instuction in group
            num_commands++;            // increment number of instructions
            last_line = true;          // set last line flag as high
         }
         
         if (incomingByte == ';'){     // last character of each line            
            lines[num_commands-1] = buf;  // store buffer in string array
          
            i = -1;                       // reset position at start of buf array
            
            // if end of last line in current instruction block reached
            if (last_line == true){
                 Serial.write("\n num_commands = ");
                 delay(10);
                 Serial.println(num_commands);
                 
                 delay(10);
                 Serial.println(buf);  // print current buffer contents
                 
                 // repeat contents of string array
                 for (int j=0; j<num_commands;j++){
                   Serial.println(lines[j]);                   // returns empty strings over serial
                 }
           
                 Mode = 1;    // change to data splitting mode
            }
            
          } // incomingByte = ';'
         
         i++;
      }
      break;
char buf[200];             // a place to store incoming data

This 10% of your memory, unless you are using a Mega.

String lines[30]; // array of instructions from incoming serial data

Advice: Ditch the String class.

String lines[30];            // array of instructions from incoming serial data
char buf[200];             // a place to store incoming data

    case 0:              // waiting for data / recieving data mode    
      while (Serial.available()>0 && i < 200)  {    // if there is data and we're not exceeding the index

Are the arrays actually defined inside a switch statement?

The comment on the case 0 line is wrong. This code does not wait for serial data. If there is no data to read, it ends right away.

                 Serial.write("\n num_commands = ");

Serial.write() to print a string? Why? This should be Serial.print().

This sounds like it should work but when I try and print the string array out of the arduino serial buffer I just get empty lines. I know that the data is being received as the buffer shows the correct contents and the arduino is counting the number of instructions correctly.

As proof of this, you provided no evidence.

I'm wondering if there are some issues copying from the char array to a string

First, there is a huge difference between string and String.

and was considering using a 2 dimensional char arrays (a list of arrays is what I'm after),

You need to do more than consider it. You need to do it.

but couldn't seem to get the syntax correct (or find an example) for copying the data in the form.

What you really want is an array of pointers.

char *line[30];

line[n] = strdup(buf); // Make a copy of what's in buf.
c2dArray[1][1:200] = buf;   // copy buffer contents to entry one, using all the characters necessary

This is not how to copy an array of chars. You mentioned the strcpy() function, earlier. You need to learn to use it, if you use static arrays.

char c2dArray[30][200];

That's 6000 bytes in the 2000 byte SRAM. Most likely will not fit.

better don't use Strings as they are ram hogs. (I agree with PaulS comments BTW)

Wrote a new "stringless" parser - need such thing in a project soon anyway.

This parser reads lines formatted as <n,n,n,...n,>
If it found a line, the line is splits into fields
Then the fields are processed.

(runs on 0.22)

//
// FILE: csvLineParser.pde
//   BY: Rob Tillaart
// DATE: 2012-feb-11
//
char line[64];                   // contains one single line from <  to >  with instructions to be processed, increase if needed.
int  commandCount = 0;
char *command[10];         // 10 pointers to commands -- increase 10 if needed.

void setup()
{
  Serial.begin(115200);
}

void loop()
{
  if (lineAvailable())
  {
    commandCount = splitLine();
    for (int i = 0; i< commandCount; i++)
    {
      processCommand(command[i]);
    }
  }
}

// reads chars from serial until it has a <...> line, then it returns true.
// Note: not 100% fool proof
boolean lineAvailable()
{
  static bool avail = false;  // statics are only iniitialized once
  static int idx=0;

  if (idx == 0) // to reset after last line complete
  {
    avail = false;
  }

  if (Serial.available() > 0)
  {
    int c = Serial.read();
    switch(c)
    {
    case '\n':  // ignore linebreaks
      break;
    case '<' :  // new line with commands starts
      idx = 0; 
      break;
    case '>':   // line ready
      avail = true;
      idx = 0;
      break;
    default:
      line[idx++] = c;
      line[idx] = '\0';
    }
  }
  return avail;
}

// assumes the commands are separated by , 
int splitLine()  
{
  int idx = 0;
  int i = 0;
  command[idx] = &line[0];

  while (line[i] != '\0')
  {
    if (line[i] == ',')  // separator found
    {
      line[i] = '\0';
      i++;
      command[++idx] = &line[i];
    }
    i++;
  }    
  return idx+1;
}

void processCommand(char *s)
{
  Serial.println(s);  // replace by your own processing
}

The String class is #1 on my list of things I never use with Arduino Five things I never use in Arduino projects | David Crocker's Solutions blog because of the memory issues it causes. String arrays are even worse. Rewrite your code to not use them.

Thanks for the prompt replies. Some very helpful comments in there that made me decide

  1. To ditch the Strings (interesting post dc42)
  2. To re-evaluate the necessity of storing the separate char instructions at all for later parsing, it looked like a waste of memory and seemed to be the cause of my dissappearing data issues.

Rob - thanks so much for the parser example. I modified it to fit my requirements and now have the following, which parses my data very robustly (let's hope the machine elves continue allowing it to do so tomorrow morning!). The switch statements in the main loop are necessary for the bigger picture this fits into into. It's actually going to take awhile to process each command (via fairly large physical actuators) so I need to do the processing independantly of the communications, hence storing the data.

// Data parser, based on the Arduino Forum example from Rob Tillaart
// 11-Feb-12 - Ad

char line[64];                   // contains one single line from >  to ;  with instructions to be processed, increase if needed.
int  commandCount = 0;
char *command[10];         // 10 pointers to commands -- increase 10 if needed.

boolean last_line = false;

int X1array[20];    // store instuctions in tables of XY co-ordinates
int Y1array[20];
int X2array[20];
int Y2array[20];
int Pen[20];

int local_index = 0;    // position in data arrays (refreshes on new transmission sets)

int Mode = 0;  // defines which mode we are in


void setup()
{
  Serial.begin(9600);
}

void loop()
{
  switch(Mode){
    case 0:                         // receiving data
      if (lineAvailable())          // if a line has been identified in the serial data
      {
        commandCount = splitLine();

        // convert string segments into elements in int arrays
//      int index;  // instruction number (from text file)
//      index = atoi(command[0]);  // don't need this data right now but might later
        X1array[local_index] = atoi(command[1]);
        Y1array[local_index] = atoi(command[2]);
        X2array[local_index] = atoi(command[3]);
        Y2array[local_index] = atoi(command[4]);
        Pen[local_index] = atoi(command[5])/100;
  
        local_index++;              // this will be used to overwrite the old data
  
        if (last_line == true){
          Mode = 1;
        } // if last_line = true
      }  // if lineAvailable()
   break;
   case 1:
         
         Serial.write("number of instructions = ");
         Serial.println(local_index);  // to account for zero index
         
         // print the contents of the arrays
         PrintData();
      
         last_line = false;  // reset last_line flag
         local_index = 0;    // reset local index (instuction count)
         Mode = 0;           // go back to listening
   break;
  }
}

void PrintData(){
// print current array contents
   for(int n = 0; n<local_index;n++){
      Serial.print(n);              // print the local index
      Serial.write(",");            
      Serial.print(X1array[n]);     // print X instruction
      Serial.write(",");            
      Serial.print(Y1array[n]);     // print Y instruction
      Serial.write(",");            
      Serial.print(X2array[n]);     // print X2 instruction
      Serial.write(",");            
      Serial.print(Y2array[n]);    // print Y2 instruction
      Serial.write(",");
      Serial.println(Pen[n]);         // print Pen instruction
      }//for
}

// reads chars from serial until it has a <...> line, then it returns true.
// Note: not 100% fool proof
boolean lineAvailable()
{
  // statics are only iniitialized once
  static bool avail = false;  // available
  static int idx=0;           // index

  if (idx == 0) // to reset after last line complete
  {
    avail = false;
  }

  if (Serial.available() > 0)
  {
    int c = Serial.read();
    switch(c)
    {
    case '\n':  // ignore linebreaks
      break;
    case '>' :  // start of new line
      idx = 0; 
      break;
    case '<' :  // start of last line
      last_line = true;  // set flag high
      idx = 0;
      break;
    case ';':   // end of line
      avail = true;
      idx = 0;
      break;
    default:    // all other characters
      line[idx++] = c;  
      line[idx] = '\0';
    }
  }
  return avail;
}

// assumes the elements are separated by , 
int splitLine()  
{
  int idx = 0;
  int i = 0;
  command[idx] = &line[0];

  while (line[i] != '\0')
  {
    if (line[i] == ',')  // separator found
    {
      line[i] = '\0';
      i++;
      command[++idx] = &line[i];
    }
    i++;
  }    
  return idx+1;
}

PaulS, this code snippet looks interesting, but I want to make sure I understand it correctly (for use in future projects)

char *line[30];

line[n] = strdup(buf); // Make a copy of what's in buf.

So line is a 30 element char array, but using strdup() effectively creates a new 30 element char array with index n? After these elements are created is it as easy as using regular arrays to access and modify the elements of line?

Thanks all,

Ad :slight_smile:

So line is a 30 element char array

No, it is an array of 30 pointers to chars. A pointer can point to a NULL terminated array of chars, of any length.

The strdup function allocates space for those chars, based on the actual number of chars in buf, and returns a pointer to the space it allocated. This is fortunate, since the assignment operator expects to store that pointer in a pointer variable or array.

After these elements are created is it as easy as using regular arrays to access and modify the elements of line?

Access, yes. Modify, maybe. The pointer can change where it is pointing to, and the pointed to space can change values. What can't happen is that the space can't change size (easily). Adding characters becomes a challenge. Removing characters is easy.

I would avoid using strdup(). Like the String class, it allocates dynamic memory, which can lead to fragmentation and hence out-of-memory. What I would do it leave the commands in the buffer but replace each command separator by '\0' and set up an array of pointers that point to the start of each command. Which is exactly what you appear to be doing in function splitline() in the code you posted in reply #4. You should also check that 'idx' remains a valid index into array 'command' so that your program doesn't crash if it receives too many commands in one line of input.

I want to thank Rob Tillaart also, He saved me a lot of headaches. I need to look at the code and see why it works and so many of the other scripts did not. I tried a few of them and they did nothing or gave the results of several Serial writes appended together.