Pages: [1] 2   Go Down
Author Topic: SD read and store  (Read 1731 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Newbie
*
Karma: 0
Posts: 14
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hello,

I'm pretty new to Arduino code, though I've been doing Java for a while... makes this a little confusing. What I'm trying to do is read single lines from an SD and then split the data based on various delimiters and store them. For each line, there's a keyword and several associated words, so I want to store the key in a HashMap and have the value be a Queue of the other words.

I found premade HashMap and QueueList libraries (which have worked for me thus far) but I've run into a problem.  I can successfully read the data and store the key in the table and I can store the other words into a QueueList, but I can't seem to store the QueueList with the key.

For example, in the following code (***note database is my HashMap variable name, and split is function I have for splitting strings), the print statements at the bottom (resp.pop()) all print out the correct thing. I've also checked the HashMap and they key is successfully stored, but when I try to print something like database.valueAt(key).pop(), it prints a blank line.
Code:
void setup()
{
  Serial.begin(9600);
  Serial.println("Start");
  
  SD.begin(4);
  File file = SD.open("brain.txt");
  
  if (SD.exists("brain.txt")) {
    Serial.println("Exists.");
  } else {
    Serial.println("boo.");
  }
  
  char buff[500];
  memset(buff, 0, sizeof(buff));
  int idx = 0;
  while (file.available())
  {
    char c = file.read();
    if (c != '\n' && c != '\r')
    {
      buff[idx] = c;
      idx++;
    }
    else {
      QueueList<String> resp;
      int rNo = 1;
      for (int i = 0; i < strlen(split(buff, ":", 2)); i++) {
        if (split(buff, ":", 2)[i] == '/') {
          rNo++;
        }
      }
      for (int i = 1; i <= rNo; i++) {
        resp.push(split(split(buff, ":", 2), "/", i));
      }
      database[split(buff, ":", 1)] = resp;
      Serial.println(resp.pop());  
      Serial.println(resp.pop());
      Serial.println(resp.pop());  
      Serial.println(resp.pop());
      memset(buff, 0, sizeof(buff));
      idx = 0;
    }
  }
}

I'm afraid the issue is something with passing values or whatnot, but I'm not sure.

Thanks for your help!
Logged

Offline Offline
Edison Member
*
Karma: 17
Posts: 1041
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Well, you have 2048 bytes of ram. The SD card library uses 512 bytes, and your buf arrays uses another 500. I'm guessing the other two libraries you're using also use up quite a bit. So, I think the problem is here:

Quote
I've been doing Java for a while

You're going to need to give up some stuff to save memory when working on microcontrollers.

Eg:
You're going to need to get rid of buf and read one line at a time and find the delimiters in the stream.
Both resp and database seem to be storing huge amounts of things on the Heap little by little. This makes memory fragmentation run rampant. Get rid of them if at all possible.
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 547
Posts: 45935
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Posting code snippets only makes it very difficult to help you. How is database defined? what does the split() function do?

What does a sample line from your file look like? How many lines are there?
Logged

New Jersey
Offline Offline
Faraday Member
**
Karma: 48
Posts: 3389
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Given your apparent memory needs, you might find it easier to use a Mega (assuming you're not already) to get more RAM. I assume you're concentrating on the config load piece for now; what will you be doing once you have this working (i.e. what is your project supposed to do) and is there any way to store that config data in a less heap intensive way?
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 14
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hey thanks for the replies.

I'm working on a Mega and have already bought a QuadRAM attachment which will add 512 kilobytes of RAM. This was just a testing sketch to see if I could properly read in data... the split function is the only thing I didn't include and it looks as follows:

Code:
char* split (char* str, char *delim, int index)
{
  char *act, *sub, *ptr;
  static char copy[100];
  int i;

  // Since strtok consumes the first arg, make a copy
  strcpy(copy, str);

  for (i = 1, act = copy; i <= index; i++, act = NULL)
  {
    sub = strtok_r(act, delim, &ptr);
    if (sub == NULL) break;
  }
  return sub;
}

I found it somewhere else on the arduino forum. As for one line of the txt file.. thats formatted like:

Code:
keyword:word1/word2/word3

The reading in is fine and the splitting is fine (from what I've tested). The storing is even fine--Just not when I try to call a specific QueueList out of the HashMap. The QueueList is found here: http://arduino.cc/playground/Code/QueueList and the HashMap was taken from http://wiring.uniandes.edu.co/source/trunk/wiring/firmware/libraries/HashMap/

If you think its a problem with RAM, the part should arrive today or tomorrow and I can continue debugging.  I was just wondering if you saw anything obviously wrong in the code. Ultimately, this will be used to store responses to certain actions (and there are a lot of them) for a stand alone Arduino board, so I thought this would be the easiest way to do that since it won't be connected to a computer.  Learning the limitations of a microcontroller is definitely something I should do, but really this project is more about a proof of concept in a limited amount of time than being extremely efficient. Thanks!
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 547
Posts: 45935
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Code:
    sub = strtok_r(act, delim, &ptr);
Why are you using the thread-safe version of strtok() on a single-threaded processor?

What is it you expect split() to do? As I read it, it returns the last token in a string.

Code:
  for (i = 1, act = copy; i <= index; i++, act = NULL)
Assigning a pointer to point to the start of an array is silly. The strtok() function expects an array of characters that is NULL terminated.

The strtok() function is much simpler, having only two arguments.

Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 14
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Thanks for the quick reply. I just found that function somewhere on the Arduino forum and used it for testing.

I've rewritten the code as WizenedEE suggested and just split the input as each character comes in.  The following code is the updated version which at the end tells me how many response words were stored in the QueueList that is matched with each keyword in the HashMap and gives the right numbers.  Unfortunately, the actual things that are stored there are still gibberish.  That being said, if I pop() directly from the QueueList before it is put into the HashMap, the Strings are correct.

Code:
#include <SD.h>
#include <QueueList.h>
#include <HashMap.h>
#include <string.h>

HashMap<String, QueueList<String>, 50> database = HashMap<String, QueueList<String>, 50>();

void setup()
{
  Serial.begin(9600);
  Serial.println("Start");
 
  SD.begin(4);
  File file = SD.open("brain.txt");
 
  if (SD.exists("brain.txt")) {
    Serial.println("Exists.");
  } else {
    Serial.println("boo.");
  }
 
  char buff[150];
  memset(buff, 0, sizeof(buff));
  int idx = 0;
  while (file.available())
  {
    char key[25];
    memset(key, 0, sizeof(key));
    QueueList<String> resp;
    while (file.peek() != '\n' && file.peek() != '\r')
    {
      char c = file.read();
      if (c != ':' && c != '/')
      {
        buff[idx] = c;
        idx++;
      }
      else {
        if (c == ':')
        {
          strlcpy(key, buff, sizeof(key));
          memset(buff, 0, sizeof(buff));
          idx = 0;
        }
        else if (c == '/')
        {
          resp.push(buff);
          memset(buff, 0, sizeof(buff));
          idx = 0;
        }
      }
    }
    file.read(); //toss new line character
    resp.push(buff); //push last response
    database[key] = resp;
    memset(key, 0, sizeof(key));
    int N = resp.count();
    for (int i = 0; i < N; i++)
    {
      resp.pop();
    }
    memset(buff, 0, sizeof(buff));
    idx = 0;
  }
 
  for (int i = 0; i < database.size(); i++) {
    QueueList<String> test = database.valueAt(database.keys[i]);
    Serial.println(test.count());
  }


void loop() {}

Is the limitation on RAM the reason gibberish is popped when the QueueList is drawn out of the HashMap? Still waiting on that part...
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 547
Posts: 45935
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

You are creating a QueueList object on the heap. When the object goes out of scope, the storage space is reused. You are assigning that location to database[n], which should do a deep copy, but apparently does not.

I'd take a look at how the copy constructor for the HashMap class is implemented, if it is. If it isn't, that would explain the lack of a deep copy.

I think you are going to need to venture into the wonderful world of pointers and malloc() and free() (or new and delete).
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 14
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

That makes sense, PaulS--the code for the HashMap library is:
Code:
/* $Id$
||
|| @author         Alexander Brevig <abrevig@wiring.org.co>
|| @url            http://wiring.org.co/
|| @url            http://alexanderbrevig.com/
|| @contribution   Brett Hagman <bhagman@wiring.org.co>
||                 Timothy Trieu (add overloaded valueAt to accept
||                                key as an argument)
||
|| @description
|| | Implementation of a HashMap data structure.
|| |
|| | Wiring Cross-platform Library
|| #
||
|| @license Please see cores/Common/License.txt.
||
*/

#ifndef HASHMAP_H
#define HASHMAP_H

//for convenience
#define CreateHashMap(hashM, ktype, vtype, capacity) HashMap<ktype,vtype,capacity> hashM
#define CreateComplexHashMap(hashM, ktype, vtype, capacity, comparator) HashMap<ktype,vtype,capacity> hashM(comparator)

template<typename K, typename V, unsigned int capacity>
class HashMap
{
  public:
    typedef bool (*comparator)(K, K);

    /*
    || @constructor
    || | Initialize this HashMap
    || #
    ||
    || @parameter compare optional function for comparing a key against another (for complex types)
    */
    HashMap(comparator compare = 0)
    {
      cb_comparator = compare;
      currentIndex = 0;
    }

    /*
    || @description
    || | Get the size of this HashMap
    || #
    ||
    || @return The size of this HashMap
    */
    unsigned int size() const
    {
      return currentIndex;
    }

    /*
    || @description
    || | Get a key at a specified index
    || #
    ||
    || @parameter idx the index to get the key at
    ||
    || @return The key at index idx
    */
    K keyAt(unsigned int idx)
    {
      return keys[idx];
    }

    /*
    || @description
    || | Get a value at a specified index
    || #
    ||
    || @parameter idx the index to get the value at
    ||
    || @return The value at index idx
    */
    V valueAt(unsigned int idx)
    {
      return values[idx];
    }

    /*
    || @description
    || | Get a value at a specified key
    || #
    ||
    || @parameter key the key to get the value at
    ||
    || @return The value at key
    */
    V valueAt(K key)
    {
      unsigned int idx = indexOf(key);
      return valueAt(idx);
    }

    /*
    || @description
    || | Check if a new assignment will overflow this HashMap
    || #
    ||
    || @return true if next assignment will overflow this HashMap
    */
    bool willOverflow()
    {
      return (currentIndex + 1 > capacity);
    }

    /*
    || @description
    || | An indexer for accessing and assigning a value to a key
    || | If a key is used that exists, it returns the value for that key
    || | If there exists no value for that key, the key is added
    || #
    ||
    || @parameter key the key to get the value for
    ||
    || @return The const value for key
    */
    const V& operator[](const K key) const
    {
      return operator[](key);
    }

    /*
    || @description
    || | An indexer for accessing and assigning a value to a key
    || | If a key is used that exists, it returns the value for that key
    || | If there exists no value for that key, the key is added
    || #
    ||
    || @parameter key the key to get the value for
    ||
    || @return The value for key
    */
    V& operator[](const K key)
    {
      if (contains(key))
      {
        return values[indexOf(key)];
      }
      else if (currentIndex < capacity)
      {
        keys[currentIndex] = key;
        values[currentIndex] = nil;
        currentIndex++;
        return values[currentIndex - 1];
      }
      return nil;
    }

    /*
    || @description
    || | Get the index of a key
    || #
    ||
    || @parameter key the key to get the index for
    ||
    || @return The index of the key, or -1 if key does not exist
    */
    unsigned int indexOf(K key)
    {
      for (int i = 0; i < currentIndex; i++)
      {
        if (cb_comparator)
        {
          if (cb_comparator(key, keys[i]))
          {
            return i;
          }
        }
        else
        {
          if (key == keys[i])
          {
            return i;
          }
        }
      }
      return -1;
    }

    /*
    || @description
    || | Check if a key is contained within this HashMap
    || #
    ||
    || @parameter key the key to check if is contained within this HashMap
    ||
    || @return true if it is contained in this HashMap
    */
    bool contains(K key)
    {
      for (int i = 0; i < currentIndex; i++)
      {
        if (cb_comparator)
        {
          if (cb_comparator(key, keys[i]))
          {
            return true;
          }
        }
        else
        {
          if (key == keys[i])
          {
            return true;
          }
        }
      }
      return false;
    }

    /*
    || @description
    || | Check if a key is contained within this HashMap
    || #
    ||
    || @parameter key the key to remove from this HashMap
    */
    void remove(K key)
    {
      int index = indexOf(key);
      if (contains(key))
      {
        for (int i = index; i < capacity - 1; i++)
        {
          keys[i] = keys[i + 1];
          values[i] = values[i + 1];
        }
        currentIndex--;
      }
    }

    void setNullValue(V nullv)
    {
      nil = nullv;
    }

  public:
    K keys[capacity];
  protected:
    V values[capacity];
    V nil;
    int currentIndex;
    comparator cb_comparator;
};

#endif
// HASHMAP_H

I don't think the constructor makes deep copies, so I tried using new and delete for the QueueList object, though I got the same result. Perhaps (more like probably) I did it wrong.

Code:
#include <SD.h>
#include <QueueList.h>
#include <HashMap.h>
#include <string.h>

HashMap<String, QueueList<String>, 50> database = HashMap<String, QueueList<String>, 50>();

void setup()
{
  Serial.begin(9600);
  Serial.println("Start");
 
  SD.begin(4);
  File file = SD.open("brain.txt");
 
  if (SD.exists("brain.txt")) {
    Serial.println("Exists.");
  } else {
    Serial.println("boo.");
  }
 
  char buff[150];
  memset(buff, 0, sizeof(buff));
  int idx = 0;
  while (file.available())
  {
    char key[25];
    memset(key, 0, sizeof(key));
    QueueList<String> resp;
    QueueList<String> *ptr = new QueueList<String>;
    ptr = &resp;
    while (file.peek() != '\n' && file.peek() != '\r')
    {
      char c = file.read();
      if (c != ':' && c != '/')
      {
        buff[idx] = c;
        idx++;
      }
      else {
        if (c == ':')
        {
          strlcpy(key, buff, sizeof(key));
          memset(buff, 0, sizeof(buff));
          idx = 0;
        }
        else if (c == '/')
        {
          resp.push(buff);
          memset(buff, 0, sizeof(buff));
          idx = 0;
        }
      }
    }
    file.read(); //toss new line character
    resp.push(buff); //push last response
    database[key] = resp;
    memset(key, 0, sizeof(key));
    int N = resp.count();
    for (int i = 0; i < N; i++)
    {
      Serial.println(resp.pop());
    }
    delete ptr;
    memset(buff, 0, sizeof(buff));
    idx = 0;
  }
 
  QueueList<String> test = database.valueAt("%ha");
  while (!test.isEmpty()) {
    Serial.println(test.pop());
  }



void loop() {}

Thanks
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 547
Posts: 45935
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Code:
    QueueList<String> resp;
    QueueList<String> *ptr = new QueueList<String>;
    ptr = &resp;
You still have a QueueList on the heap. Then, you dynamically create another one, pointer to by ptr. But, then you discard the address where the dynamic memory allocation occurred, and point to the first QueueList on the heap. That memory just leaked, and you can never recover it.

Get rid of resp and the reassignment of ptr. Change all the "resp." strings to "ptr->".
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 14
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I see. I read that syntax from a tutorial somewhere, but I've changed it to what you suggested.  It compiles, but still gives me the same output as before. I think the problem may be with the way I stored the QueueList into the HashMap. To get it to compile I did:

Code:
database[key] = *ptr;

But I'm not sure if that's the right syntax. Or maybe something else is wrong.

Code:
// keys can only be max 25 chars

#include <SD.h>
#include <QueueList.h>
#include <HashMap.h>
#include <string.h>

HashMap<String, QueueList<String>, 50> database = HashMap<String, QueueList<String>, 50>();

void setup()
{
  Serial.begin(9600);
  Serial.println("Start");
 
  SD.begin(4);
  File file = SD.open("brain.txt");
 
  if (SD.exists("brain.txt")) {
    Serial.println("Exists.");
  } else {
    Serial.println("boo.");
  }
 
  char buff[150];
  memset(buff, 0, sizeof(buff));
  int idx = 0;
  while (file.available())
  {
    char key[25];
    memset(key, 0, sizeof(key));
    QueueList<String> *ptr = new QueueList<String>;
    while (file.peek() != '\n' && file.peek() != '\r')
    {
      char c = file.read();
      if (c != ':' && c != '/')
      {
        buff[idx] = c;
        idx++;
      }
      else {
        if (c == ':')
        {
          strlcpy(key, buff, sizeof(key));
          memset(buff, 0, sizeof(buff));
          idx = 0;
        }
        else if (c == '/')
        {
          ptr->push(buff);
          memset(buff, 0, sizeof(buff));
          idx = 0;
        }
      }
    }
    file.read(); //toss new line character
    ptr->push(buff); //push last response
    database[key] = *ptr;
    memset(key, 0, sizeof(key));
    int N = ptr->count();
    for (int i = 0; i < N; i++)
    {
      Serial.println(ptr->pop());  //testing
    }
    delete ptr;
    memset(buff, 0, sizeof(buff));
    idx = 0;
  }
 
  QueueList<String> test = database.valueAt("%ha");
  while (!test.isEmpty()) {
    Serial.println(test.pop());
  }



void loop() {}
« Last Edit: July 13, 2012, 12:13:53 pm by mrquacks » Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 14
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

So I've been testing but getting no where. Here is the txt file and the output I get, if that helps.

brain.txt
Code:
%ex:duration = _duration/defaultUser = _defaultUser/currentUser = _currentUser/prevUser = _prevUser
%ha:good/yay

For testing purposes, I only look at what was stored in the QueueList for %ha
Code:
Start
Exists.
duration = _duration
defaultUser = _defaultUser
currentUser = _currentUser
prevUser = _prevUser
good
yay

call out QueueList from HashMap
wa

My code is:
Code:
// keys can only be max 25 chars

#include <SD.h>
#include <QueueList.h>
#include <HashMap.h>
#include <string.h>

HashMap<String, QueueList<String>, 50> database = HashMap<String, QueueList<String>, 50>();

void setup()
{
  Serial.begin(9600);
  Serial.println("Start");
 
  SD.begin(4);
  File file = SD.open("brain.txt");
 
  if (SD.exists("brain.txt")) {
    Serial.println("Exists.");
  } else {
    Serial.println("boo.");
  }
 
  char buff[150];
  memset(buff, 0, sizeof(buff));
  int idx = 0;
  while (file.available())
  {
    char key[25];
    memset(key, 0, sizeof(key));
    QueueList<String> *ptr = new QueueList<String>;
    while (file.peek() != '\n' && file.peek() != '\r')
    {
      char c = file.read();
      if (c != ':' && c != '/')
      {
        buff[idx] = c;
        idx++;
      }
      else {
        if (c == ':')
        {
          strlcpy(key, buff, sizeof(key));
          memset(buff, 0, sizeof(buff));
          idx = 0;
        }
        else if (c == '/')
        {
          ptr->push(buff);
          memset(buff, 0, sizeof(buff));
          idx = 0;
        }
      }
    }
    file.read(); //toss new line character
    ptr->push(buff); //push last response
    database[key] = *ptr;
    memset(key, 0, sizeof(key));
    int N = ptr->count();
    for (int i = 0; i < N; i++)
    {
      Serial.println(ptr->pop());  //testing
    }
    delete ptr;
    memset(buff, 0, sizeof(buff));
    idx = 0;
  }
 
  Serial.println("\ncall out QueueList from HashMap");
  QueueList<String> test = database.valueAt("%ha");
  while (!test.isEmpty()) {
    Serial.println(test.pop());
  }



void loop() {}
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 547
Posts: 45935
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Code:
HashMap<String, QueueList<String>, 50> database = HashMap<String, QueueList<String>, 50>();
I'd like to see a link as to where you got this library, to confirm something.

This looks to me like you are trying to create a HashMap instance of a templated class, where the instance uses a String, a QueueList (that is an instance of a templated class that takes a String) object,  and int argument.

I think the 2nd argument to the templated class needs to be a pointer to an type, not a type.
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 14
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I got the HashMap from

https://github.com/krohling/ArduinoPusherClient/tree/master/HashMap

I would've gotten it directly from the person who made it, but I think there was a bug that this guy fixed so I used this version instead.
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 14
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I'm still not exactly sure what you meant in the last post, but I tried adding a * after the QueueList<String> argument. Still got the same result though.

Code:
// keys can only be max 25 chars

#include <SD.h>
#include <QueueList.h>
#include <HashMap.h>
#include <string.h>

HashMap<String, QueueList<String>*, 50> database = HashMap<String, QueueList<String>*, 50>();

void setup()
{
  Serial.begin(9600);
  Serial.println("Start");
 
  SD.begin(4);
  File file = SD.open("brain.txt");
 
  if (SD.exists("brain.txt")) {
    Serial.println("Exists.");
  } else {
    Serial.println("boo.");
  }
 
  char buff[150];
  memset(buff, 0, sizeof(buff));
  int idx = 0;
  while (file.available())
  {
    char key[25];
    memset(key, 0, sizeof(key));
    QueueList<String> *ptr = new QueueList<String>;
    while (file.peek() != '\n' && file.peek() != '\r')
    {
      char c = file.read();
      if (c != ':' && c != '/')
      {
        buff[idx] = c;
        idx++;
      }
      else {
        if (c == ':')
        {
          strlcpy(key, buff, sizeof(key));
          memset(buff, 0, sizeof(buff));
          idx = 0;
        }
        else if (c == '/')
        {
          ptr->push(buff);
          memset(buff, 0, sizeof(buff));
          idx = 0;
        }
      }
    }
    file.read(); //toss new line character
    ptr->push(buff); //push last response
    database[key] = ptr;
    memset(key, 0, sizeof(key));
    int N = ptr->count();
    for (int i = 0; i < N; i++)
    {
      Serial.println(ptr->pop());  //testing
    }
    delete ptr;
    memset(buff, 0, sizeof(buff));
    idx = 0;
  }
 
  Serial.println("\ncall out QueueList from HashMap");
  QueueList<String>* test = database.valueAt("%ha");
  while (!test->isEmpty()) {
    Serial.println(test->pop());
  }



void loop() {}
Logged

Pages: [1] 2   Go Up
Jump to: