Corrupted array?

Hello everyone!

So far I have managed to solve all my problems just by looking at different threads but I’ve ran into something very strange that’s been blowing my mind for days now, so I’ve decided to sign up and maybe someone might give me an insight…

So basically I’ve written a massive program with tons of global arrays of floats (unfortunately I need them all) and I’ve tried optimising everything, so I fit in the “hard disk” and SRAM parameters of the UNO board. Somewhere towards the end of the main loop I move down the values of several of these arrays, using a for loop:

for (uint8_t i=0; i< 14; i++){
Array1[i]=Array1[i+1];
Array2[i]=Array2[i+1];
Array3[i]=Array3[i+1];
Array4[i]=Array4[i+1];
Array5[i]=Array5[i+1];
}

I expect for each of these arrays to be shuffled down by one element, so I can add a new element at the end (ArrayX[15]) after the loop. Indeed that does happen, but after several iterations of the main loop the first eight or nine elements of the middle array become ‘nan’. This happens every single time. And only to that array - the others are being shifted down properly. Furthermore, the problematic array is still being shifted down, but only the last 5 or 6 elements. To illustrate, here is the serial output of that particular array:

-0.94 , -0.94 , -0.94 , -0.94 , -0.94 , -0.94 , -0.93 , -0.94 , -0.94 , -0.94 , -0.94 , -0.94 , -0.93 , -0.93 , -0.94 ,
-0.94 , -0.94 , -0.94 , -0.94 , -0.94 , -0.93 , -0.94 , -0.94 , -0.94 , -0.94 , -0.94 , -0.93 , -0.93 , -0.94 , -0.94 , 
-0.94 , -0.94 , -0.94 , -0.94 , -0.93 , nan , nan , nan , nan , -0.94 , -0.93 , -0.93 , -0.94 , -0.94 , -0.94 ,
-0.94 , -0.94 , -0.94 , -0.93 , nan , nan , nan , nan , nan , -0.93 , -0.93 , -0.94 , -0.94 , -0.94 , -0.93 , 
-0.94 , -0.94 , -0.93 , nan , nan , nan , nan , nan , nan , -0.93 , -0.94 , -0.94 , -0.94 , -0.93 , -0.94 , 
-0.94 , -0.93 , nan , nan , nan , nan , nan , nan , nan , -0.94 , -0.94 , -0.94 , -0.93 , -0.94 , -0.94 , 
-0.93 , nan , nan , nan , nan , nan , nan , nan , nan , -0.94 , -0.94 , -0.93 , -0.94 , -0.94 , -0.94 ,
nan , nan , nan , nan , nan , nan , nan , nan , nan , -0.94 , -0.93 , -0.94 , -0.94 , -0.94 , -0.94 , 
nan , nan , nan , nan , nan , nan , nan , nan , nan , -0.93 , -0.94 , -0.94 , -0.94 , -0.94 , -0.94 , 
nan , nan , nan , nan , nan , nan , nan , nan , nan , -0.94 , -0.94 , -0.94 , -0.94 , -0.94 , -0.93 ,
nan , nan , nan , nan , nan , nan , nan , nan , nan , -0.94 , -0.94 , -0.94 , -0.94 , -0.93 , -0.94 ,

My best guess is it just becomes ‘corrupted’, though I am not sure of the meaning of the word. I have looked into issues caused by running out of SRAM and judging by this http://playground.arduino.cc/Main/CorruptArrayVariablesAndMemory, it doesn’t seem like I am having that problem. My .hex file reads:

text	   data	    bss	    dec	    hex
           26456	    368	    956	  27780	   6cea

So based on the link above I should have plenty of SRAM.

Also it may be worth noting that I use a a couple of methods/functions that use a lot of SRAM (in the form of huge temporary arrays), but I have solved that by constraining the temporary variables in a one-time for-loop, assuming that the memory is cleared at the end of the for loop. I think that’s not the best thing to do, as it leaves the memory with holes like Swiss cheese (I read this somewhere), so some clarity on that will be quite useful.

I find it really strange and frustrating because all these arrays are pretty much the same in terms of how and where I use them and I’ve double and triple checked if I’ve made some sort of a typo just for that particular array.

Any suggestions towards fixing this would be extremely helpful!

EDIT: I’ve put the whole code two posts below. The problem I’m experiencing is with the zAcc[15] array.

Your post is a classic example of why you need to post all of your code, not just a small portion of it.

Exactly how big are the arrays ? What data type are they ? How big is "massive" ? How many arrays are there in all ?

It sounds like you are overwriting the contents of one or more of your arrays but without seeing all of your code, who knows.

Proper index management should mean you wouldn't have to move data around like this, but without seeing more of your code, my guess is that you're running short of RAM.

UKHeliBob:
Your post is a classic example of why you need to post all of your code, not just a small portion of it

As said before the code is pretty big and I did not see a point of posting it, but if you insist, here it is:

#include <Wire.h>
#include <MatrixMath.h>

#define STATUS_LED 13

float var[3] ={sq(0.5647/180*PI), sq(0.5674/180*PI), sq(0.5394/180*PI)};
float g_offset[3]={-15.12, 34.34, -11.32};

float aAcc[4] = {1, -2.9529, 2.9069, -0.9540};
float bAcc[4] = {0.1597/100000, 0.4792/100000, 0.4792/100000, 0.1597/100000};

float aMag[4] = {1, -1.7347, 0.7660};
float bMag[4] = {0.0078, 0.0156, 0.0078};

float P_Upd[4][4]={{2, 0, 0, 0}, {0, 2, 0, 0}, {0, 0, 2, 0}, {0, 0, 0, 2}};
float Q[4][4]={0};

float last_qObs[4][1]={{1},{0},{0},{0}};
float qObs[4][1]={{0.5},{0.5},{0.5},{0.5}};

float last_qUpd[4][1]={{1},{0},{0},{0}};
float qUpd[4][1]={{0.5},{0.5},{0.5},{0.5}};

float curAcc[3][1]={0};
float curMag[3][1]={0};
float curGyro[3][1]={0};
float lastGyro[3][1]={0};

float f_Acc[3][1]={0};
float xAcc[15]={0};
float yAcc[15]={0};
float zAcc[15]={0};

float f_Mag[3][1]={0};
float xMag[15]={0};
float yMag[15]={0};
float zMag[15]={0};

float angles[3]={0};

uint32_t timer;

void setup(){
  Serial.begin(115200);
  
  pinMode (STATUS_LED,OUTPUT);
  Wire.begin();
  digitalWrite(STATUS_LED,LOW);
  
  delay(500);
  
  acc_init();
  mag_init();
  gyro_init();
  
  delay(500);
  //gyro_varX(var[0]);  // These need 409 bytes
  //gyro_varY(var[1]);
  //gyro_varZ(var[2]);
  // Not sure how to calculate variance
  //gyro_offsets();
  
  for (uint8_t t=0; t<15; t++){
    read_acc();
    read_mag();
    read_gyro();
    v_norm(curAcc);
    v_norm(curMag);
    xAcc[t]=curAcc[0][0];
    yAcc[t]=curAcc[1][0];
    zAcc[t]=curAcc[2][0];
    xMag[t]=curMag[0][0];
    yMag[t]=curMag[1][0];
    zMag[t]=curMag[2][0];
    
    for (uint8_t i=0; i<3; i++){
      lastGyro[i][0]=curGyro[i][0];
    }
    
    delay(20);
  }
  
  calc_Q(var, Q);
  
  digitalWrite(STATUS_LED,HIGH);
  
  timer = micros();
}

void loop(){
//  test();
  // Acquire new data
  read_acc();
  read_mag();
  read_gyro();
  // Normalise acc and mag
  v_norm(curAcc);
  v_norm(curMag);
  
  filter(3, bAcc, aAcc, xAcc, yAcc, zAcc, f_Acc);
  filter(2, bMag, aMag, xMag, yMag, zMag, f_Mag);
  v_norm(f_Acc);
  v_norm(f_Mag);
  
  calc_GN(last_qObs, f_Acc, f_Mag, qObs);
  q_norm(qObs);
  
  Kalman_filter();
  
  q_norm(qUpd);
  calc_angles(qUpd, angles);
  Serial.println(F("Angles: "));
  for (uint8_t an=0; an<3; an++){
    Serial.print(angles[an]);
    Serial.print(F(" , "));
  }
  Serial.println(F(""));
  
  Matrix.Copy((float*)qObs, 4, 1, (float*)last_qObs);
  Matrix.Copy((float*)qUpd, 4, 1, (float*)last_qUpd);
  
  for (uint8_t i=0; i<3; i++){
    lastGyro[i][0]=curGyro[i][0];
  }
  
  for (uint8_t i=0; i<14; i++){
    xAcc[i]=xAcc[i+1];
    yAcc[i]=yAcc[i+1];
    zAcc[i]=zAcc[i+1];
    xMag[i]=xMag[i+1];
    yMag[i]=yMag[i+1];
    zMag[i]=zMag[i+1];
  }
  xAcc[14]=curAcc[0][0];
  yAcc[14]=curAcc[1][0];
  zAcc[14]=curAcc[2][0];
  xMag[14]=curMag[0][0];
  yMag[14]=curMag[1][0];
  zMag[14]=curMag[2][0];
  
  timer = micros();
}

@UKHeliBob: It does say in my original post the data type is float. I have looked very carefully at indexing of these arrays and as said before, this only happens to one of the arrays.

AWOL:
Proper index management should mean you wouldn’t have to move data around like this, but without seeing more of your code, my guess is that you’re running short of RAM.

I do not understand what you mean by that - since I am feeding new data through a sensor, i do not see another way of keeping the last 15 data points in an array. If I am running short of RAM, why would that happen only to that particular array, and furthermore, just half of it?

With the entire code posted maybe it’s important to clarify some aspects of it a bit more. The whole program’s intended to use an inertial measurement unit to get new data and it should be able to do Kalman filtering on the microcontroller.

The filter filter(uint8_t order, float b[], float a[], float xstate[], float ystate[], float zstate[], float data_output[3][1]) function DOES NOT change the values of xstate, ystate, zstate, it only modifies the data_output[3][1] array.

since I am feeding new data through a sensor, i do not see another way of keeping the last 15 data points in an array.

Leave the data where it is, and move the pointers to it - it takes less time.

I did not see a point of posting it,

Why not? We're not psychic.

AWOL: Leave the data where it is, and move the pointers to it - it takes less time.

I apologise for being slow, maybe it's the fact I'm fairly new to C and the whole pointers concept, but I don't have the data. A new 'data point' comes every iteration of the loop(), so I need to save it to the array because the next iteration it will be a different value, that needs to be saved as well and so on. I am a bit confused.. :~

I used the term "pointer" in a very loose sense, so perhaps I should have written "array index" instead. There is rarely any need to move data around like this, it's a waste of instructions and memory bandwidth ( not all that important here, but it is good to learn good habits early)

Imagine your readings are bits of paper with numbers on them, and your array is a set of pigeon holes. As you're given a new bit of paper, you wouldn't shuffle up or down all the other contents of the pigeon holes, would you? You'd say " pigeon hole x is my new latest reading, and the next reading goes into pigeon hole x+1"

What is the purpose of a 2D 3 x 1 array? How does that differ from a 1D array?

The names you used in your snippet are not the names that you use in the real code. You say that only one of the array is becoming corrupted, but you haven't said which one. That's like going to the tire store and saying "One of my tires has a slow leak. Please fix it. I'll be back tomorrow", and tehn walking away. When you get back, you'll either have 4 new tires and a big bill or you'll still have a slow leak.

@AWOL: Wouldn't that constantly increase my array (pigeon hole) size? What happens when I have a new bit of paper and all holes are filled up? I still don't really get it..

@PaulS: The purpose of the 3x1 array is that I am using matrices ("MatrixMath") for the filtering and it's easier. Basically it's the way my program is structured. The problem occurs with zAcc, you're right I'm gonna edit the previous post and put the info there as well. I prefer staying in the shop because I'm sure I haven't given said enough for people to understand. :)

Wouldn't that constantly increase my array (pigeon hole) size?

Of course it would. It's up to you to ensure it doesn't happen, because you know when you're going to write off the end of your array, so you don't do that, and start back at the beginning again.

@AWOL: Okay! Now I understand what you mean. But I need them ordered as well, with the most recent ones at the back. What I understand is that for example if I have an array with 15 elements and I fill it up with consecutive numbers, when it's full it's gonna look like 1, 2...14, 15 and then 16 comes in and you're saying to put it in the beginning, so it becomes 16,2,3...14,15, but I need it 2,3,...15,16.

Maybe there is a way to read 16,2,3...14,15 from the second element till the end and then adding the first element, but I think that's gonna be a pain when I try to implement it, that's why I chose to simply shuffle it down.

To illustrate, here is the serial output of that particular array:

That, to my mind, is rubbish. There is NOTHING in that output that gives anyone a clue as to what that batch of numbers corresponds to in the code. There is nothing in the code that gives a clue as to where those numbers were printed.

The thing that you need to determine is WHERE the array gets corrupted. Is it in the mysterious filter() function that you didn't show? Or is it in the moving the arrays?

@PaulS: Dude, chill. I didn’t show the “mysterious” filter() function as it does not meddle with it, but I can show it to you if you insist. The serial output is taken just before the arrays are input in the filter() function.

Here is the main loop with the serial output included:

void loop(){
//  test();
  // Acquire new data
  read_acc();
  read_mag();
  read_gyro();
  // Normalise acc and mag
  v_norm(curAcc);
  v_norm(curMag);
  for (uint8_t i=0; i<15; i++){
    Serial.print(xAcc[i]);
    Serial.print(F(" , "));
  }
  Serial.println();
  for (uint8_t i=0; i<15; i++){
    Serial.print(yAcc[i]);
    Serial.print(F(" , "));
  }
  Serial.println();
  for (uint8_t i=0; i<15; i++){
    Serial.print(zAcc[i]);
    Serial.print(F(" , "));
  }
  Serial.println();
  for (uint8_t i=0; i<15; i++){
    Serial.print(xMag[i]);
    Serial.print(F(" , "));
  }
  Serial.println();
  for (uint8_t i=0; i<15; i++){
    Serial.print(yMag[i]);
    Serial.print(F(" , "));
  }
  Serial.println();
  for (uint8_t i=0; i<15; i++){
    Serial.print(zMag[i]);
    Serial.print(F(" , "));
  }
  Serial.println();
  filter(3, bAcc, aAcc, xAcc, yAcc, zAcc, f_Acc);
  filter(2, bMag, aMag, xMag, yMag, zMag, f_Mag);
  v_norm(f_Acc);
  v_norm(f_Mag);
  
  calc_GN(last_qObs, f_Acc, f_Mag, qObs);
  q_norm(qObs);
  
  Kalman_filter();
  
  q_norm(qUpd);
  calc_angles(qUpd, angles);
  Serial.println(F("Angles: "));
  for (uint8_t an=0; an<3; an++){
    Serial.print(angles[an]);
    Serial.print(F(" , "));
  }
  Serial.println(F(""));
  
  Matrix.Copy((float*)qObs, 4, 1, (float*)last_qObs);
  Matrix.Copy((float*)qUpd, 4, 1, (float*)last_qUpd);
  
  for (uint8_t i=0; i<3; i++){
    lastGyro[i][0]=curGyro[i][0];
  }
  
  for (uint8_t i=0; i<14; i++){
    xAcc[i]=xAcc[i+1];
    yAcc[i]=yAcc[i+1];
    zAcc[i]=zAcc[i+1];
    xMag[i]=xMag[i+1];
    yMag[i]=yMag[i+1];
    zMag[i]=zMag[i+1];
  }
  xAcc[14]=curAcc[0][0];
  yAcc[14]=curAcc[1][0];
  zAcc[14]=curAcc[2][0];
  xMag[14]=curMag[0][0];
  yMag[14]=curMag[1][0];
  zMag[14]=curMag[2][0];
  
  timer = micros();
}

And here is the filter() function; it is based on a simple IIR filter and returns the last element of the filtered data input, but does not change the data in any way:

void filter(uint8_t order, float b[], float a[], float xstate[], float ystate[], float zstate[], float data_output[3][1]){
  float temp[15]={0};
  IIR_filter(order, b, a, 15, xstate, temp);
  data_output[0][0]=temp[14];
  IIR_filter(order, b, a, 15, ystate, temp);
  data_output[1][0]=temp[14];
  IIR_filter(order, b, a, 15, zstate, temp);
  data_output[2][0]=temp[14];
  Serial.println(freeRam());
}

void IIR_filter(uint8_t ord, float b[], float a[], uint8_t np, float x[], float y[]){
  uint8_t i,j;
  y[0]=b[0]*x[0];
  for (i=1;i<ord+1;i++){
    y[i]=0.0;
    for (j=0;j<i+1;j++)  y[i]=y[i]+b[j]*x[i-j];
    for (j=0;j<i;j++)  y[i]=y[i]-a[j+1]*y[i-j-1];
  }
/* end of initial part */
  for (i=ord+1;i<np+1;i++){
    y[i]=0.0;
    for (j=0;j<ord+1;j++)  y[i]=y[i]+b[j]*x[i-j];
    for (j=0;j<ord;j++)  y[i]=y[i]-a[j+1]*y[i-j-1];
  }
}

EDIT: I think the array gets corrupted when moving it. I just can’t explain why this happens only to that array! :~

@PaulS: Dude, chill.

OK. I'll check back in a couple of weeks, to see how you are doing.

@PaulS: What’s with all the hate? It’s my first thread - I’m still learning, sorry if I haven’t provided enough information. I’ve put everything I thought was relevant and if something is unclear I try to clarify it. If you’ve had enough of me, I understand. Thanks for trying to help though!

What's with all the hate?

What hate?

You asked a programming question in the programming forum, clearly without reading the sticky at the top of the forum. The one that says post ALL of your code.

Each time that there has been a request for more information, you've gotten more defensive. Fine. That's your right. It's also my right to walk away when you do that.

@PaulS: I did read the sticky. Very carefully actually. And it seems you have as well, since you were the first one to comment on it. And this is why I did not not post my entire code:

With coding problems, if possible post a “minimal” sketch that demonstrates the problem - not hundreds of lines of code.

So I do not believe you are right when saying that I had to post ALL my code, I though I posted the relevant bit, giving a generic example of my problems. You are a lot more experienced than I am and I just didn’t see why you had to use words like ‘rubbish’ and the whole superiour attitude, instead of just asking nicely - this is the only reason I am being defensive.

Anyway, if you do not wish to help (and I can clearly see you don’t), we do not have to continue this silly argument.

So I do not believe you are right when saying that I had to post ALL my code

Read your first post in this thread. How much of your minimal code compiles and demonstrates the problem? Now do you understand our comments?

My initial idea was that someone might have experienced a similar problem before, that's why I didn't think it's necessary to be a compilable code. I've shown the bit where I think the problem occurs and I've described what I expect and what I get, so yes, I think I've demonstrated the problem. As to your comments, I understand that I should have posted everything from the beginning, but then you might have said that I've put too much unnecessary code and explaining it would have been hard.

So thanks for the help everyone! I strongly believe I am running out of SRAM for some reason, since my indexing is correct, otherwise what other reasons does an array have to get corrupted? The strangest thing is that it only happens only to that particular array, which probably makes everyone reading this think I've messed something up with it.

PS: I am sorry for all the drama.

What happens if you deal with the arrays in a different order or declare them in a different order or both. Does the problem occur with the same array or the array declared third (or whatever) ?

As has been suggested it would seem to be better to use a moving array index rather than shuffling the array(s). Add to the array in the position indicated by its index, increment the index and if it has exceeded the upper bound of the array reset it to zero.

As to reading the data back in the order it was received, the index of the next data to be stored is held in its index, so the earliest data available was stored at that position. Read values forward from that point to the end of the array, reset the index to zero and read forward until the index is one less that the original index value, or do the whole thing in a for loop for a fixed number of reads.