Optimizing memory for a 3D array with arrays of various size

Hi!

I am writing code for a cnc wire cutting machine with 5 stepper motors. I am sending machine code to the arduino from the computer as a string and then parsing it into individual commands using indexOf() and substring(). I have the code working fine, but the issue is that sometimes one motor will have many coordinate pairs, and others only a few. I feel like I am wasting a ton of memory by declaring an array with many empty sub-arrays.

the code basically takes a string like this:

100,200:300,400:1700,50:800,3999/15,49:13,22/1111,2500:666,420/5,9700:666,1256:33333,1500/12,6000:3333,5000:6644,1234:6457,12:1100,3

and parses it into an int array[5][6][2] like this:

{[100,200][300,400][1700,50][800,3999][0,0][0,0]}
{[15,49][13,22][0,0][0,0][0,0][0,0]}
{[1111,2500][666,420][0,0][0,0][0,0][0,0]}
{[5,9700][666,1256][33333,1500][0,0][0,0][0,0]}
{[12,6000][3333,5000][6644,1234][6457,12][1100,3][0,0]}

(the punctuation is just for the Serial.print, and isn’t part of the actual array.)

this version is minimized for demonstration purposes, but sometimes certain motors will need dozens of coordinate pairs, while others need only two. What I really want is an array[5][4,2,2,3,5][2], that can change depending on the string fed to it.

here is my code, including a print function:

int const maxcuts = 6;
long int  matrix[5][maxcuts][2];
String  strTemp;
int cuts[5];

void parsePos() {
  int indexM = 0;
  for (int i = 0; i < 5; i++) {
    int indexMnew = strTemp.indexOf('/', indexM);
    String strTempM = strTemp.substring(indexM, indexMnew);

    int strMLen = strTempM.length();                                       //length of string M
    int cut;
    int y = 0;
    for (int x = 0; x < strMLen; x++) {                                     //#of commas = #of cuts = #of items in 2nd array dimension
      if (strTempM[x] == ',') {
        y++;
        cut = y;
      }
    }

    int indexC = 0;
    for (int j = 0; j < cut; j++) {
      int indexCnew = strTempM.indexOf(':', indexC);
      String strTempC = strTempM.substring(indexC, indexCnew);

      int indexP = 0;
      for (int k = 0; k < 2; k++) {
        int indexPnew = strTempC.indexOf(',', indexP);
        String strTempP = strTempC.substring(indexP, indexPnew);

        matrix[i][j][k] = strTempP.toInt();
        indexP = indexPnew + 1;
      }
      indexC = indexCnew + 1;
    }
    cuts[i] = cut;
    indexM = indexMnew + 1;
  }
}

void print3DMatrix(long int mat[5][maxcuts][2]) {
  for (int a = 0; a < 5; a++) {
    Serial.print("{");
    for (int b = 0; b < maxcuts; b++) {
      Serial.print("[");
      Serial.print(mat[a][b][0]);
      Serial.print(',');
      Serial.print(mat[a][b][1]);
      Serial.print("]");
    }
    Serial.println("}");
  }
}

void setup() {
  Serial.begin(9600);
}

void loop() {
  while (Serial.available() > 0) {
    strTemp = Serial.readString();
    Serial.println(strTemp);
    parsePos();
    print3DMatrix(matrix);
  }
}

I was thinking pointers might be the way to go, from what I’ve read, but I’m a novice and have no idea how to implement them. Also everything I’ve read on the subject says that everything but the first dimension must have bounds. I could add a loop that finds the longest array in the second dimension and size that dimension accordingly, so at least I’m not wasting space on every column. I’m not sure if this would help or slow things down.

I tried not giving the global variable “int matrix” dimensions, but it didn’t like that. Is is as simple as adding a pointer * ?

thanks for the help!

I am sending machine code to the arduino from the computer as a string and then parsing it into individual commands using indexOf() and substring().

No, you aren't. You are pissing away resources using the String class.

Also everything I've read on the subject says that everything but the first dimension must have bounds.

Every dimension must have bounds. The "except the first" bit only applies to arrays that are declared AND initialized in one statement, where the task of defining the first dimension can fall to the compiler.

I tried not giving the global variable "int matrix" dimensions, but it didn't like that.

Of course it didn't.

Is is as simple as adding a pointer * ?

No. You can either create separate arrays for each motor, and send data for each motor separately, or determine a maximum number of cuts that will be supported and size the 3D array accordingly.

Hi Paul,

thanks for the reply.

PaulS:
No, you aren't. You are pissing away resources using the String class.

I do convert the string to int's as soon as possible for later manipulation... what would you suggest I use instead? char array?

PaulS:
You can either create separate arrays for each motor, and send data for each motor separately,

This sounds like the best option, I will try it.

thanks again

I do convert the string to int's as soon as possible for later manipulation

A String object and a string are not the same thing. Do not keep blathering away as though they are.

what would you suggest I use instead? char array?

Of course.

PaulS:
Do not keep blathering away as though they are.

Wow, you don't have to be a dick about it. I'm new to programming and I'm just trying to figure it out as I go. Belittling people may be fun for you, Mr High and Mighty, but it doesn't help foster an inclusive environment. Do you want to drive people away from the forum?

nonetheless, thank you.

Use a char array and study examples that use strtok(). It will save you some memory and, once you understand how the function works, it’s very simple to use. Note that strtok() is “destructive”, in that it chops the input string to bits while it works. If you need to preserve the input array for some reason, use strcpy() before strtok().

You have to cut Paul some slack. With over 65,000 posts, probably 3/4 of them are on the same issue and all of us may get a tad cranky explaining the same issues over and over. Still, with that many posts and that many karma points, he’s a valuable resource to have around.

obucklin:
Wow, you don't have to be a dick about it. I'm new to programming and I'm just trying to figure it out as I go. Belittling people may be fun for you, Mr High and Mighty, but it doesn't help foster an inclusive environment. Do you want to drive people away from the forum?

nonetheless, thank you.

^

I'd like to help you clear up your unused memory, but first, do you know at compile time which motors will have which length commands?

For example, will motor 1 always have 4 sets of command, motor always have 2 sets of command, etc?

If these are your constraints, then you can write your code such that it is more efficient.

What you describe is a relatively hard problem. The non-present values are still relevant, right - they just aren't in the text?

Actually, your text representation is pretty efficient. Depending on how the arrays need to be accessed, you could just do some pre-processing to get a list something like

100,200:300,400:1700,50:800,3999/15,49:13,22/1111,2500:666,420/5,9700:666,1256:33333,1500/12,6000:3333,5000:6644,1234:6457,12:1100,3

parsing to:

int semiparsedCoords[MAXSIZE][2] = {[100,200][300,400][1700,50][800,3999][-1,-1], 
[15,49][13,22][-1,-1],
[1111,2500][666,420][-1,-1], 
[5,9700][666,1256][33333,1500][-1,-1], 
[12,6000][3333,5000][6644,1234][6457,12][1100,3][-1,-1]

Essentially, doing the ascii-decimal conversion, and replacing the separator with a "recognizably illegal value" ([-1,-1])
(works reasonably well if you step through the list one at a time (save an index to the last stopping point), and not TOO badly even for random indexing by tupple number (just iterate and count separators. Should be plenty fast for coordinate lists shorter than typical AVR RAM sizes, and mechanical "real objects.")

Or, lookup "sparse arrays"; there are probably standard respected implementations; perhaps even something usable on Arduino. (although, your array might not really be sparse enough for those algorithms...)

would a linked list of dynamically allocated arrays work out better from a memory management point of view?

would a linked list of dynamically allocated arrays work out better from a memory management point of view?

Not on a microcontroller with severely limited quantities of memory and questionable abilities in the "dynamic allocation" area. :frowning: If you HAD good dynamic memory capabilities, you could just allocate each array at exactly the required size.

Not on a microcontroller with severely limited quantities of memory and questionable abilities in the "dynamic allocation" area. :frowning: If you HAD good dynamic memory capabilities, you could just allocate each array at exactly the required size.

Well thats unfortunate, Still new to the micro controllers but in hindsight I can see how that could be computationally expensive for arduino. Maybe if I get some time this weekend I'll see if I can play around with some dynamically allocated arrays. Thanks for the answer

A minimal implementation of a slab allocator? :smiley:

Than you everyone who replied with helpful suggestions since last week!

BigBobby:
I'd like to help you clear up your unused memory, but first, do you know at compile time which motors will have which length commands?

For example, will motor 1 always have 4 sets of command, motor always have 2 sets of command, etc?

Unfortunately not, it depends on the geometry of each brick I am cutting, so from unit to unit it will vary.

westfw:
What you describe is a relatively hard problem. The non-present values are still relevant, right - they just aren't in the text?

Not sure what you mean by non-present.

westfw:
Actually, your text representation is pretty efficient. Depending on how the arrays need to be accessed, you could just do some pre-processing to get a list something like

100,200:300,400:1700,50:800,3999/15,49:13,22/1111,2500:666,420/5,9700:666,1256:33333,1500/12,6000:3333,5000:6644,1234:6457,12:1100,3

parsing to:

int semiparsedCoords[MAXSIZE][2] = {[100,200][300,400][1700,50][800,3999][-1,-1],
[15,49][13,22][-1,-1],
[1111,2500][666,420][-1,-1],
[5,9700][666,1256][33333,1500][-1,-1],
[12,6000][3333,5000][6644,1234][6457,12][1100,3][-1,-1]




Essentially, doing the ascii-decimal conversion, and replacing the separator with a "recognizably illegal value" ([-1,-1])
(works reasonably well if you step through the list one at a time (save an index to the last stopping point), and not TOO badly even for random indexing by tupple number (just iterate and count separators. Should be plenty fast for coordinate lists shorter than typical AVR RAM sizes, and mechanical "real objects.")

Or, lookup "sparse arrays"; there are probably standard respected implementations; perhaps even something usable on Arduino. (although, your array might not really be sparse enough for those algorithms...)

So the sooner I can convert the string to a decimal array, the better. I will take a whack at it with these suggestions.

Thanks again!