Delimited String to byte array

Projects Programming
1

Hi All,

Firstly I appologise for posting this if it has already been asked before, but i might not be using the right key words as I'm pretty new to this.

I will have a number of relocatable/repurposeable arduino boards that I will want to change IP and MAC addresses (along with a number of ather variable) without having to recompile the source each time.

I have a need to read a IP or MAC string from a config file on an SD card before initialising the ethernet adaptor with byte arrays.

I have the SD reading down pat but am looking for away to convert my delimited string to a byte array. Code is below for reference. Any clues would be much appreciated

-Simon

config.ini:

[Global]
;This is a comment
prog=Track0imer V2
ver=2.1
name=Simon Pilepich
organization=Melton Motor Sports CLub
Unit Number=1

SerialOutput=1
networkOutput=1
networkDiag=0
logToFile=0

[Network Config]

mac=02-50-F2-00-00-01
ip=192.168.001.003
subnet=255.255.255.000
gateway=192.168.001.100
trackServer=192.168.001.001
timeServer=192.043.244.018
ntpPort=8888
framePort=63333
httpPort=80
diagPort=60666

[SD Card]
delimiter=,
basefilename=result

sketch:

/*
 *Autocross Laptimer/Sensor+Sender
 *Sep 2011
 *Simon Pilepich
 *simon.pilepich@gmail.com
 */
//Include Libraries
#include <Time.h>
#include <SPI.h>         
#include <Ethernet.h>
#include <Udp.h>
#include <SD.h>
#include <LiquidCrystal.h>

//Declare Default Global Variables 
unsigned int ntpPort = 8888;                          // local port to listen for UDP packets
unsigned int framePort = 63333;                       // local port to listen for UDP packets
unsigned int httpPort = 80;                           // local port to listen for UDP packets
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };  //default MAC
byte ip[] = { 192,168,1,2 };                          //default ip
byte timeServer[] = { 192, 43, 244, 18 };             //time.nist.gov NTP server
const int NTP_PACKET_SIZE= 48;                        // NTP time stamp is in the first 48 bytes of the message
byte packetBuffer[ NTP_PACKET_SIZE];                  //buffer to hold incoming and outgoing packets 
String configFileName="config.ini";                   //config file name to draw from sd card
File configFile;                                      //config file to draw from sd card
int triggerState;                                     //timer trigger state
long interval = 100;                                  // blink interval - change to suit
long previousMillis = 0;                              // variable to store last time LED was updated
long startTime;                                       // start time for stop watch
long elapsedTime;                                     // elapsed time for stop watch


/*TODO
 *Start Program
 */
void setup() {
  Serial.begin(9600);
  
  //Print out a description
  diagOut("Autocross Laptimer/Sensor+Sender V0.1");
  diagOut("*Sep 2011");
  diagOut("*Simon Pilepich");
  diagOut("*simon.pilepich@gmail.com");
  
  
  //Open the SD card session and call function to open then read settings from the config file if possible
  if (!SD.begin(4)) {
    diagOut("SD initialisation failed!");
    return;
  }
  else {
    configFile = SD.open(configFileName.toCharArray());
    if (configFile) {
      
// stuck here*********************************<<<<<<


       readini("ip")   --------->  mac[]


     
    }
    else {
      diagOut("Cannot Read File");
    }
  }
  
  //Start Ethernet after reading config file
  Ethernet.begin(mac,ip);
  Udp.begin(ntpPort);
  
  
}

/*TODO
 *Main Program
 */
void loop() {
}

/*
*Read value from ini config file
*/
String readini(String val) {
  String aString = "";                                                                                   // temporary string space   
  int aByte;                                                                                             // temporary byte/int space
  while (configFile.available()) {                          
    aByte = configFile.read();                                                                           // run through file, 1 char at a time
    if (aByte == 10 || aByte == 13) {                                                                    // break for eval at each line break
      if (aString.length() != 0 && aString.charAt(0) != ';' && aString.charAt(0) != '[') {               // ignore null lines, comments and headings
        if (aString.substring(0,aString.indexOf('=')) == val) {                                          // compare current line with lookup 
          return aString.substring(aString.indexOf('=') + 1,aString.length());                           // return value
        }
      }
      aString = "";
    }
    else {
      aString = aString + byte(aByte);
    }
  }
  // close the file:
  configFile.close();
}

void diagOut(String diagMSG) {
  Serial.println(diagMSG);
}

I

2

I have the SD reading down pat but am looking for away to convert my delimited string to a byte array. Code is below for reference. Any clues would be much appreciated

The code you posted does not lend credence to the "I have the SD reading down pat" part of your statement. All that I can see is that you can open the file on the SD card and call a function that gathers the characters from a line into a String object. That function does not return anything nor does it write to any global variables.

You need to ditch the use of the String class. You have so much going on in the sketch that you can not afford the overhead of the String class.

The records in the config file should be limited to a known, reasonable, length. Store the data in an array that is passed to the readini() function (by reference). Don't forget to properly NULL terminate the array as it is populated.

The strtok() function can then be used to extract tokens. Call strtok() with the array, initially, and "=" as the delimiter string. The first token returned, when called for "mac=02-50-F2-00-00-01", will be "mac". Then, call strtok() with NULL as the first argument (to keep parsing the same string) and "-" as the delimiter. Call it in a while loop. Each call will return a token or a NULL. If it returns a token, use atoi() to convert that token to an int, and store it in the next position in a byte array (making sure it fits, of course) (both in terms of value and array size).

3

Thx PaulS....

What's the overhead String? And at what point does it become useful??

Where can i find out how exactly to use these functions, I am pretty new to this and the arduino reference doesn't have any reference to these functions you mention. Is there an API as such?

Also, you mention that i should have the config lines "limited to a known, reasonable, length". I don't understand this. What is the potential problem.

I used string return from the read ini because i thought it would be more flexible on the length. I would love to have the readini function to set the global variables, but i didn't think that possible?!?!?. Doesn't the variable's lable get destroyed on compilation? Any how, can i get it to return a char array instead?

Damn I'm lost.... I think I have bitten off more than I can chew here. To many questions i know. don't feel obliged to answer.....

4

What's the overhead String?

The String class fragments memory pretty badly. You don't have much to begin with on the Arduino. By the time the String class gets done helping you read 30 characters from the SD card, you could be out of memory (at least in terms of contiguous bytes of memory).

Where can i find out how exactly to use these functions

Google comes to mind.

I am pretty new to this and the arduino reference doesn't have any reference to these functions you mention.

The Arduino reference is for Arduino-specific functions. The strtok() function is not Arduino-specific.

Also, you mention that i should have the config lines "limited to a known, reasonable, length". I don't understand this. What is the potential problem.

Is there any reason to allow someone to put:
thisIsSomeVariableThatIWantToAssignAValueTo = "SomeRidiculouslyLongValueThatIsToBeAssignedGoesOnThisSideOfTheEqualsSign.ThereCanBeABazillionCharactersInTheValue"
in your config file? Is it not reasonable to say that a line in the file is limited to 40 characters? if so, then you can size the array needed to read the file, without needing to make it ridiculously large.

I used string return from the read ini because i thought it would be more flexible on the length.

It is, but that flexibility is not free.

I would love to have the readini function to set the global variables, but i didn't think that possible?!?!?

Why not? On the other hand, why? The readini() function should find the data that corresponds to the named variable and return that data. Otherwise, you need a readiniandparsemac() function and a readiniandarseip() function, and a host of others.

Doesn't the variable's lable get destroyed on compilation?

I don't understand this question.

Any how, can i get it to return a char array instead?

The readini function shouldn't return anything. It should take three arguments - the name of the line to locate, the address of an array to write to, and the size of that space.

void readini(char *varName, char *varData, int arrLen)
{
}

In loop():

char macStg[30];
readini("mac", macStg, 30);

The readini function then puts the data that it reads from the SD card into the varData array, as long as there is space (which arrLen defines).

I think I have bitten off more than I can chew here.

The best way to learn to swim is to jump in the deep end...

5

Hi again everyone,

First big thanks to PaulS for not flaming me and being so supportive. As always, the more I learn, the more I realise the less I know. I took Pauls points away and have been reading alot and learning some stuff about C. I highly recomend that people(particularly other aussie that have trouble with accents like me) listen through the lectures of Richard Buckland@UNSW, they have helped me get my head around some of conceptshttp://www.youtube.com/results?search_query=Richard+Buckland+UNSW&aq=f

I have some working code for my initialisation routine now that could probably use some optimisation but I have confirmed it is actually working so far.. The sketch has a long way to go in terms of actual function, so if you have any comments I would love to hear them as I will be working on this for some time to come.

Cheers

Simon

/*
 *Autocross Laptimer/Sensor+Sender
 *Sep 2011
 *Simon Pilepich
 *simon.pilepich@gmail.com
 */
//Include Libraries
#include <Time.h>
#include <SPI.h>         
#include <Ethernet.h>
#include <Udp.h>
#include <SD.h>
#include <LiquidCrystal.h>



//Declare Default Global Variables 
unsigned int ntpPort = 8888;                          // local port to listen for UDP packets
unsigned int framePort = 63333;                       // local port to listen for UDP packets
unsigned int httpPort = 80;                           // local port to listen for UDP packets
byte mac[6];                                          //device MAC
byte ip[4];                                           //device ip
byte gateway[4];                                      //default gateway ip
byte timeServer[4];                                    //timeServer 
int triggerState;                                     //timer trigger state
long interval = 100;                                  // blink interval - change to suit
long previousMillis = 0;                              // variable to store last time LED was updated
long startTime;                                       // start time for stop watch
long elapsedTime;                                     // elapsed time for stop watch
LiquidCrystal lcd(8, 9, 4, 5, 6, 7);


/*
 *Setup Function - Initialisation  from SD card
 */
void setup() {
  Serial.begin(9600);                                      //Open Serial
  lcd.begin(16, 2);                                        // set up the LCD's number of columns and rows: 
  if (!SD.begin(4)) {                                      //Open the SD card session
    diagOut(1,"");              
    return;
  }
  else {
    ipAddr(readini("ip"),ip);                              // Get ip
    ipAddr(readini("gateway"),gateway);                    // Get gateway
    macAddr(readini("mac"),mac);                           // Get mac
  }  
  Ethernet.begin(mac, ip, gateway);                        // start the Ethernet connection:
}

/*TODO
 *Main Program
 */
void loop()
{

}

/*
*Read value from ini config file
 */
char *readini(char *val) {
  File configFile = SD.open("cfg.ini");
  char lineStr[80] = {""};                                                      //string line buffer
  char *result = NULL;                                                          //the pointer to the string for strtok
  byte readByte;                                                                //byte buffer for chachter reading
  char delims[] = "=";                                                          //set the delimiters(just equals for the ini
  int linePos = 0;                                                              //position register for line string
  while (configFile.available()) {                                              // while we are not at the end of the file
    readByte = configFile.read();                                                 // grab a byte
    if (readByte != 10 && readByte != 13 && linePos < 81 && readByte != 59) {   // and while for each line(limited to 80 characters and stop at comments)
      lineStr[linePos] = readByte;                                              // add string to line 
      linePos++;                                                                // move to the next character slot
    }
    else {                                                                      // When we get to the end of the line
      result = strtok( lineStr, delims );                                       // grab the first 
      if( strcmp (result,val) == 0) {                                           // if we have the requested value
        configFile.close();                                                     // Close the file
        return (strtok( NULL, delims ));                                        // return a pointer to the appropriate string
      }
      for (int i=0;i<80;i++) {                                                  
        lineStr[i] = NULL;                                                      // Reset the linebuffer to nothing
      }
      linePos = 0;                                                              // Reset the linebuffer position
    }
  }
  configFile.close();                                                           // Close the file
  return NULL;                                                                  // If we find nothing.
}


/*TODO
 *Diagnostic function
 */
void diagOut(int i, char* msg) {                                                           // Diagnostic output routine **To DO**
  switch (i) {
  case 0:
    lcd.println(msg);
    break;
  default:
    Serial.println(i);
  }
}

/*
*Delimit and Modify and IP    
 */
void ipAddr(char *inputString, byte *modIP) {                                                                 
  char delims[] = ".";                //set the delimiters
  char *result = NULL;                   //the pointer to the string for strtok
  int a = 0;                             //set the current return string pos
  result = strtok( inputString, delims );//grab the first section of the input string
  modIP[a] = byte(atoi(result));     //set the byte into the return string                            
  while (result != NULL) {               //Until we get to the end of the string  && a < 3
    a++;                                 //increment the array positino
    result = strtok(NULL, delims );      //split the subsequent substrings
    if (result != 0) {
      modIP[a] = byte(atoi(result)); //set the byte into the return string
    }                               
  }
}


/*
*Delimit and Modify and MAC    
 */
void macAddr(char *inputString, byte *modMAC) {                          
  char delims[] = "-:";                         //set the delimiters
  char *result = NULL;                         //the pointer to the string for strtok
  int a = 0;                                   //set the current return string pos
  result = strtok( inputString, delims );      //grab the first section of the input string
  modMAC[a] = byte((ascHexToInt(result[0]) * 16)+(ascHexToInt(result[1])));    //set the byte into the return string                            
  while (result != NULL) {                     //Until we get to the end of the string  && a < 3
    a++;                                       //increment the array positino
    result = strtok(NULL, delims );            //split the subsequent substrings
    if (result != 0) {                         //dont include null asci character (end lines)
      modMAC[a] = byte((ascHexToInt(result[0]) * 16)+(ascHexToInt(result[1])));       //set the byte into the return string
    }                               
  }
}


/*
*Quick and dirty ascii hex convertor
*/
int ascHexToInt(char inStr) {
  if (inStr < 58 ) {                           //For ascii numbers
    return (inStr - 48);                      //Subtract 48 to get int value
  }
  if (inStr < 71 ){                           //For Upper Case
    return (inStr - 55);                      //Subtract 48 to get int value
  }
  else {                                      //For Upper Case
    return (inStr - 87 );                     //Subtract 87 to get int value
  } 
}
6

I'm afraid that your readini function still has problems. The strtok function creates and returns a pointer to the (now mangled) array that you are parsing. You then return that pointer from readini.

Unfortunately, that pointer points to memory allocated for the local variable lineStr in the function. When the function ends, the variable goes out of scope. Any pointers to that space become invalid.

That space isn't reclaimed/reused immediately, so you've gotten away with using it, so far, but there is no guarantee that you will continue to get away with it.

The readini function should really be a void function, taking two arguments - an array containing the name to look for and an array to write the value to.

void readini(char *name, char *valu)
{
   // Lots of stuff snipped
   // Get a pointer from strtok
   char *ptr = strtok(NULL, delims);
   valu[0] = '\0';
   strcat(valu, ptr);
   // Lots of stuff snipped
}

In the readini function, the data pointed to by the pointer returned by strtok should be COPIED to the location pointed to be valu.

The function should then be called like so:

char ipArray[20];
readini("ip", ipArray);
7

LOL i just watched this^^

I get it now.

thx

Simon

8

I really like the versitilitly of "returning" a value.

Would it be frowned apon to create a excissively larger global to dump crap in to pull out later?

I really want to avoid side affects where possible, but stuggle to see an alternative.

9

I really like the versitilitly of "returning" a value.

Having a function say "Here is your value" is no different from having the caller say "put my value here, please", in terms of the function defining a value.

The caller has to allocate a variable to hold the returned value or allocate a variable to have the function write to, if the value is to be stored.

Having the function return a value, and using that function in place of the value, IS more convenient, but there are drawbacks. If more memory were available, readini could call strdup and return the pointer allocated by strdup. But, memory is constrained, so developers must adopt a more resource-constrained point of view.

Would it be frowned apon to create a excissively larger global to dump crap in to pull out later?

Yes. Anything that uses excessive amounts of memory is frowned upon. Some stuff is downright scowled at.

10

Thanks again Paul, I just about think we should rename this forum to askPaul..

Last question before i call it a night, do you think 80 Bytes of string/char array would be excessive? I am planning on holding some larger strings in the ini down the track but the should be less than 80 bytes.

11

do you think 80 Bytes of string/char array would be excessive

No. 80 characters is not unreasonable for reading/parsing text files.

12

PaulS:

Would it be frowned apon to create a excissively larger global to dump crap in to pull out later?

Yes. Anything that uses excessive amounts of memory is frowned upon. Some stuff is downright scowled at.

However, one "best practice" in resource constrained environments is to use a "scratchpad" that is memory that can be used by different parts of the code at different times. If you need a kilobyte to parse a configuration file, or run a path-finding algorithm, or run an image-analysis algorithm, but you never run those three systems at the same time, then it makes perfect sense to declare a single kilobyte scratch-pad, and pass that as the storage location to be used by each of those subsystems.

The main problem here, from a code structure point of view, is making sure that the systems that use the scratch pad don't expect the data they put in there to remain untouched forever, and (different side of same coin) make sure that no two systems try to use the scratchpad at the same time.