Initialize Array Size and Serial Data Constraint

para23 · April 25, 2022, 5:25am

Hello, I'm having an issue understanding why I am not able to increase the size of my array without my code breaking (No errors or warnings but allocating incorrect values). I followed this thread: Serial Input Basics - updated - #2 by Robin2 in receiving and parsing data from a serial port as for my code I am receiving a string of values from a GUI. In their example they receive 32 bytes of data, however, for mine I will be receiving in a string similar in size to the one below with about 288 different variables:

<1, 1.0, 100.0, 0.2, 1.0, 1000.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 100.0, 0.2, 1.0, 1000.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 100.0, 0.2, 1.0, 1000.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 100.0, 0.2, 1.0, 1000.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 100.0, 0.2, 1.0, 1000.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 100.0, 0.2, 1.0, 1000.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 100.0, 0.2, 1.0, 1000.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 100.0, 0.2, 1.0, 1000.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 100.0, 0.2, 1.0, 1000.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 100.0, 0.2, 1.0, 1000.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 100.0, 0.2, 1.0, 1000.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 100.0, 0.2, 1.0, 1000.0, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0>

When trying to change the size of the const byte at the top of the code I can get it to roughly 128 bytes before it starts to break. I checked to see what different byte values changed how much memory was given and saw the pattern below with values of 50, 128, and 256 respectively. As can be seen it allocates the least amount of memory for the 256 unless I am misunderstanding this.

I also attempted change the const byte to an int type instead of the same size that had been working for my code (128). I did make sure to change the static byte to a static int for the index in the recvWithStartEndMarkers() function. However, this also seemed to break the outputs for my code.

//Global Variables
const byte numChars = 128;
char receivedChars[numChars];
char tempChars[numChars];        // temporary array for use when parsing

void recvWithStartEndMarkers() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    char startMarker = '<';
    char endMarker = '>';
    char rc;

    while (Serial.available() > 0 && newData == false) {
        rc = Serial.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }
}

If anyone could explain the difference between using const byte versus const int when initializing the array or what I could possibly do to fix this that would be much appreciated. The last idea of how I could "fix" this is by making more arrays of byte size 128 and then once one is full just move to the next one, however, this seems tedious and redundant so I'm hoping for a better solution.

red_car · April 25, 2022, 5:32am

What exactly does that mean?

Please post all your code (the entire sketch) not just bits of it.

para23 · April 25, 2022, 6:59am

My apologies, I just posted the code that I thought was relevant at least for filling that array. What I mean by not working is that I'm getting the incorrect value with the array when just changing the size, however if I change it back to say 128 then I get the correct value that I'm sending and expecting after parsing the data. I hope that explains it? If not I can try and rephrase it again. Here is the rest of the code portions are commentated out as I was only testing some of the functions at the time (in the main loop):

#include <Wire.h>
#include <math.h>
#include <String.h>

const byte numChars = 128;
char receivedChars[numChars];
char tempChars[numChars];        // temporary array for use when parsing

boolean newData = false;
boolean Finished = false;

const int Pin = 6;
const int pwmout = 10;

unsigned long startMillis;
unsigned long currentMillis;

int Time_Check;

//Standard Float Arrays
float Current_Val[12] = {0.0};
float RPM_Val[12] = {0.0};

//Ramping Float Arrays
float Time_Val_T[96] = {0.0};
float Current_Val_T[96] = {0.0};
float RPM_Val_T[96] = {0.0};

boolean Start = true;

void setup() {
  
  Serial.begin(9600);
  
  Wire.begin();       //Join i2c bus (Address option for Master)
  
}

void loop() {
//    if(Start == true)
//    {
//      analogWrite(pwmout, 255);
//      tone(Pin, 0);
//      Start = false;
//    }
    recvWithStartEndMarkers();
    if (newData == true)
    {
        // this temporary copy is necessary to protect the original data
        // because strtok() used in parseData() replaces the commas with \0
        strcpy(tempChars, receivedChars);

        //Parse Data
        parseData();

        if(Time_Check == 0)
        {
          //Convert Data (Characteristic Equation)
          ConvertData();

          //Send Data to other Arduinos
          //Send_Data();
          
          //Set Current
          Set_Current();
  
          //Set RPM
          Set_RPM();
          
          newData = false;
        }

        if(Time_Check == 1)
        {
          //Convert Data (Characteristic Equation)
          ConvertData_Timed();

          //analogWrite(pwmout, Current_Val_T[0]);
          //tone(Pin, RPM_Val_T[0]);
          
          //Send Data to other Arduinos
          //Send_Data_Time();
          
          //Set RPM and Current with Time
          Set_Values_Time();
          
          newData = false;
        }
    }
}

//=========================================================
//
//  Setting/Converting Data Values Functions
//
//=========================================================

void ConvertData()
{
  for(int i = 0; i < (sizeof(Current_Val)/sizeof(Current_Val[0])); i++)
  {
//    //Convert Current to PWM (Characteristic Equation)
//    if(Current_Val[i] > 1.9945)
//      Current_Val[i] = 1.9945;
//  
//    Current_Val[i] = (Current_Val[i] - 1.9945)*-97.71;

      //Convert Current to PWM (Characteristic Equation)
      if(Current_Val[i] > 1.95)
        Current_Val[i] = 1.95;
  
      Current_Val[i] = (Current_Val[i] - 1.95)*-100;

    //RPM to Frequency
    //RPM_Val[0] = RPM_Val[0];
  }
}

void ConvertData_Timed()
{
  for(int i = 0; i < (sizeof(Current_Val_T)/sizeof(Current_Val_T[0])); i++)
  {
    //Convert Current to PWM (Characteristic Equation)
    if(Current_Val_T[i] > 1.95)
      Current_Val_T[i] = 1.95;
  
    Current_Val_T[i] = (Current_Val_T[i] - 1.95)*-100;

    //RPM to Frequency
    //RPM_Val_T[i] = RPM_Val_T[i]/60;
  }
}

void Set_Current()
{
  analogWrite(pwmout, Current_Val[0]);  
}

void Set_RPM()
{
  tone(Pin, RPM_Val[0]);
}

void Set_Values_Time()
{
  unsigned long Period;
  for(int i = 0; i < 8; i++)
  {
    Finished = false;
    analogWrite(pwmout, Current_Val_T[i]);
    tone(Pin, RPM_Val_T[i]);

    Period = Time_Val_T[i]*60000;
    
    startMillis = millis();
    while(!Serial.available() && (Finished == false))
    {
      currentMillis = millis();
      if(currentMillis - startMillis >= Period)
      {
        Finished = true;
      }
    }
  }
  //analogWrite(pwmout, Current_Val_T[1]);
  //tone(Pin, RPM_Val_T[1]);
}

//=========================================================
//
//  I2C Sending Data
//
//=========================================================

void Send_Data()
{
    String TempVal;
    String Temp_RPM;
    String Temp_Current;

    for(int i = 0; i < (sizeof(Current_Val)/sizeof(Current_Val[0])); i++)
    {
      Temp_RPM = String(RPM_Val[i+1]);
      Temp_Current = String(Current_Val[i+1]);

      TempVal = "0, " + Temp_RPM + ", " + Temp_Current;

      Wire.beginTransmission(i);
      Wire.write(TempVal.c_str());
      Wire.endTransmission();

      delay(1);
    }
}

void Send_Data_Time()
{
  String TempVal;
  String Temp_RPM;
  String Temp_Current;
  String Temp_Time;

  int Count = 0;

  for(int i = 0; i < (sizeof(Current_Val_T)/sizeof(Current_Val_T[0])); i++)
  {
      Temp_RPM += String(RPM_Val_T[i]) + ", ";
      Temp_Current += String(Current_Val_T[i]) + ", ";
      Temp_Time += String(Time_Val_T[i]) + ", ";

      Count++;

      if(Count == 6)
      {
        Temp_RPM += String(RPM_Val_T[i]) + ", ";
        Temp_Current += String(Current_Val_T[i]) + ", ";
        Temp_Time += String(Time_Val_T[i]);
        
        TempVal = "1, " + Temp_RPM + Temp_Current + Temp_Time;

        Wire.beginTransmission(i);
        Wire.write(TempVal.c_str());
        Wire.endTransmission();

        Count = 0;

        delay(100);
      }
  }
}

//=========================================================
//
//  Recieving/Parsing Data Functions
//
//=========================================================

void recvWithStartEndMarkers() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    char startMarker = '<';
    char endMarker = '>';
    char rc;

    while (Serial.available() > 0 && newData == false) {
        rc = Serial.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }
}

//============

void parseData() {      // split the data into its parts

    char * strtokIndx; // this is used by strtok() as an index
    
    strtokIndx = strtok(tempChars,",");         // get the first part - the string
    Time_Check = atoi(strtokIndx);              // copy it to floatFromPC

    int i = 0;

    //Standard Setting Values
    if(Time_Check == 0)
    {
      while(strtokIndx != NULL)
      {
        strtokIndx = strtok(NULL,",");
        RPM_Val[i] = atof(strtokIndx);

        strtokIndx = strtok(NULL,",");
        Current_Val[i] = atof(strtokIndx);

        i++;
      }
    }

    //If it includes time (T) or ramping
    else
    {
      while(strtokIndx != NULL)
      {
        strtokIndx = strtok(NULL,",");
        Time_Val_T[i] = atol(strtokIndx);
        
        strtokIndx = strtok(NULL,",");
        RPM_Val_T[i] = atof(strtokIndx);

        strtokIndx = strtok(NULL,",");
        Current_Val_T[i] = atof(strtokIndx);
        
        i++;
      }
    }
}

red_car · April 25, 2022, 7:05am

Can you provide an example? What value of numChars doesn't work as expected and exactly what results to you get?

red_car · April 25, 2022, 7:19am

BTW.. the largest value a byte can contain is 255 (8 bits). Better to use unsigned int (uint16_t) for > 255.

Also.. what level of compiler warning do you have set. If I try and compile your code (with the level set to All) I get a bunch of warnings... these will likely point you at your issue.

sterretje · April 25, 2022, 7:38am

//Edit: this rply was based on an Uno So you're not necessarily low on memory.

Not behind a PC while typing this; i did however compile it earlier and you're low on memory.

I don't think that you need the tempChars array, you can work directly on receivedChars; that will save you 138 bytes of precious memory.

Further make all variables that will fit in a byte of type byte; that will possibly save some bytes.

Do you need 96 floats (times 2)? As said, not behind a pc at this moment so can't judge your program.

That's just some hints, will try to look later again

westfw · April 25, 2022, 10:22am

If you are using an Uno or other atmega328 board, you only have 2k of data memory. That probably won’t be enough for 288 floats (4 bytes each) plus serial buffers plus strings…

sterretje · April 25, 2022, 10:36am

OK, based on the screenshots of the memory usage, it's not an Uno; which board has 6k memory? Possibly a Nano Every?

david_2018 · April 25, 2022, 10:53am

As was pointed out, the index variable in the receive function will need to be larger than byte when using 256 bytes or larger in the buffer.

Are there any range limitations on the values of the data being send? I do not see anything in the sample data that would require floats for storage. The first number appears to be a single integer, followed by repeated groups of a small float number (single digit, decimal, one decimal place), a large float (with up to 4 digits before the decimal) and then another small float. That could be stored as byte, int, byte, respectively if the value is a fairly limited range (byte could contain 0 to 25.5, integer -3276.8 to +3276.7 or 0 to 6553.6 for unsigned int), just have to keep track of the decimal places yourself.

I would also try to eliminate the use of String, that tends to take a bit more memory than char arrays.

Your parsing function is not checking for validity after every use of strtok - that may get you into trouble.

para23 · April 25, 2022, 5:26pm

It is a nano every being used. When I'm changing numChars to const int of size 1024 it uses more dynamic memory (Also tested with size of 128 and it breaks there).

What happens when I'm trying to run it with both my index and numChars changed to an int is that I am getting the current value I'm sending but can not get my RPM value. I'm not sure how to tell what is actually the value that is being set for that array and not sure how to really check it. This is as I can't use a serial monitor while connected to the serial port with my GUI.

para23 · April 25, 2022, 5:32pm

First value being sent is just a 1 or 0 for what mode its in, second value is an time value which can be set up to 1440, third value is RPM which can be set up to 1000.0, and fourth value is Current which is set to max of 2.0. There is 288 of these values in total with that repeating pattern. What would you recommend for checking validity after strtok? I'm wondering if somehow that could be the issue as mentioned in my reply to sterretje. Also what do you mean by not using string as I am just using string on the GUI side but using char array on the arduino side, unless I'm misunderstanding.

para23 · April 25, 2022, 5:36pm

I tried to use int for a greater size but couldn't get that to work as mentioned in my reply to Sterretje. Also I did have my compiler warnings setup wrong but I don't believe that was the issue as I'm still getting the same result (warnings were just for some of my for loops in their data type but it still iterated for correct number of loops.

david_2018 · April 25, 2022, 11:30pm

Major problem, you are writing past the end of your arrays. When you reach the last float in the input text, after converting the text to a float and storing it in Current_Val, the while condition then checks for (strtokIndx != NULL), which will be true. At this point i will be 96, and the subsequent calls to strtok will return NULL, but there is no check for strtokIndx being equal to NULL at this point.


    //Standard Setting Values
    if (Time_Check == 0)
    {
      while (strtokIndx != NULL)
      {
        strtokIndx = strtok(NULL, ",");
        RPM_Val[i] = atof(strtokIndx);

        strtokIndx = strtok(NULL, ",");
        Current_Val[i] = atof(strtokIndx);

        i++;
      }
    }

david_2018 · April 26, 2022, 12:03am

Just how large have you made the Wire.write buffer????

void Send_Data_Time()
{
  String TempVal;
  String Temp_RPM;
  String Temp_Current;
  String Temp_Time;

  int Count = 0;

  for (int i = 0; i < (sizeof(Current_Val_T) / sizeof(Current_Val_T[0])); i++)
  {
    Temp_RPM += String(RPM_Val_T[i]) + ", ";
    Temp_Current += String(Current_Val_T[i]) + ", ";
    Temp_Time += String(Time_Val_T[i]) + ", ";

    Count++;

    if (Count == 6)
    {
      Temp_RPM += String(RPM_Val_T[i]) + ", ";
      Temp_Current += String(Current_Val_T[i]) + ", ";
      Temp_Time += String(Time_Val_T[i]);

      TempVal = "1, " + Temp_RPM + Temp_Current + Temp_Time;

      Wire.beginTransmission(i);
      Wire.write(TempVal.c_str());
      Wire.endTransmission();

      Count = 0;

      delay(100);
    }
  }
}

para23 · April 28, 2022, 2:10am

This was not the issue originally with the post but it is another problem that I didn't realize till this moment. I looked into it a little bit and see that I can change the buffer size in the actual header file but considering I'll have a size that is close to 100-200 chars (1 byte each) with that string I want to send.

Would you recommend a different approach to sending this string of data to other slaves as I've seen some people have issues with memory size in regards to increasing buffer size to something too large.

para23 · April 28, 2022, 2:29am

I did attempt to change the buffer size to various values to try to change what I actually received on the slave end but the last part got truncated (thus I didn't get my "end of transmission" character '>'). An image can be seen below of an example of what output I got and I'm missing about 3 more values. However, when changing the buffer size in the .h file as explained here https://support.arduino.cc/hc/en-us/articles/4406686928786-Modify-the-buffer-size-of-the-Wire-library I did not receive any different results

david_2018 · April 28, 2022, 3:02am

Is there some reason the data has to be sent to the slaves as ASCII characters? Could you use serial instead of I2C for communicating with the slaves?

In regards to the original serial input, you are going to need a buffer of almost 2k bytes if you have the maximum value in all fields simultaneously, which leaves very little memory for expanding buffers and working with Strings.

It would really help to have some general description of what all this is meant to do.

para23 · April 28, 2022, 3:53am

I'm sending them as ASCII characters as that's the best way I understand to send it and how I am initially sending the data serially to the master unit. I also already have code to parse that data in the master. (which I fixed that error you mentioned, thanks!). They do have to be connected through I2C based on my hardware design which connects multiple Arduino units together.

A general overview of my system is that I am sending a string of data from processing on my computer that holds data values such as time, rpm, and current. It uses these values to send varies PWM signals or setting frequencies for the rest of my hardware to use. I want to send the values to my other slave units so they get each their corresponding value based on the ID that I flashed them with. Each slave unit at a maximum at this time (No plans for future values at this moment but possibly in the future) will have 24 unique values that it will need to be sent. The slaves are just receiving this data and do not need to send back to the master.

para23 · April 28, 2022, 3:55am

Timing for the data to get to the other units is of slight concern but it can be within 10 seconds and still be plenty fine.

system · October 25, 2022, 3:55am

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
question about array Programming Questions	4	464	May 6, 2021
variable size array Programming Questions	25	26225	May 5, 2021
Error: variable-sized array may not be initialized Syntax & Programs	12	10297	May 6, 2021
Promlem with Arrays Programming Questions	9	822	May 5, 2021
Serial and array question Syntax & Programs	10	2705	May 6, 2021

Initialize Array Size and Serial Data Constraint

Related topics