Go Down

Topic: Unsigned long arrays and timeStamp (Read 203 times) previous topic - next topic

Dr_pepper

I'm using a Esp8266 as a remote weatherstation.
The webpage uses google graphs to display measurements with time on a graph.
The horizontal axis is time/date, I send unix timestamps to the webpage.
It was working ok, but because when the memory got full it shifted all the table entries up one this wore out the eeprom, so I changed the program so that the memory just rolls around, and a pointer keeps everything in order, so just one location gets changed not the whole buffer area.

These are the defines:

float *tempCdata;
float *pressData;
float *fdata;
float *ldata;
float *uvdata;
float *gtempdata;
unsigned long *timeStamp;

I run this during setup to declare the variables:

void allocateRam() {
  //We will allocate as much RAM as we can to track historical data. This means, we need to work out how much RAM we will needed for the TCP stack and enough for the ESP8266 to run.
  //Once we get this value, we can allocate the rest of the RAM to the array used to store the temperature and pressure values.
  //
  //Each TCP session expires every 2mins. (Default MSL is 2mins for Windows https://technet.microsoft.com/en-us/library/cc938217.aspx)
  //If using the refresh function on the main page with gauges, then there needs to be enough RAM left to accommodate the TCP sessions from the refresh as well as a little extra for the ESP8266 to run.
  //196KB RAM per refresh.
  //To calculate the amount to leave free;
  // (number of refreshes in 2 minutes x RAM per REFRESH) + SPARE RAM = ALLOCATED_RAM
  //
  //E.g.
  //For a refresh of every 1 second
  //(120x196KB) + 10,000 = 33,520
  //For a refresh of every 3 seconds
  //(40x196KB) + 10,000 = 17,840
  //
  //The less frequent the refresh will result in a smaller value needed to be reserved. And this also mean more RAM can be used to store historical data for a longer period.
  //If refresh isnt used, then leave ALLOCATED_RAM to 10,000

  //Get free RAM in bytes
  uint32_t free = system_get_free_heap_size() - ALLOCATED_RAM;

  //Divide the free RAM by the size of the variables used to store the data.
  //This will allow us to work out the maximum number of records we can store.
  //All while keeping some RAM free which is specified in ALLOCATED_RAM
  numberOfRows = free / (sizeof(float) * 6 + sizeof(unsigned long)); // 4 x float for temp pressure, humidity & lux.  Long for time.

  //re-declare the arrays with the number of elements
  tempCdata = new float [numberOfRows];
  pressData = new float [numberOfRows];
  fdata = new float [numberOfRows];
  ldata = new float [numberOfRows];
  uvdata = new float [numberOfRows];
  gtempdata = new float [numberOfRows];
  timeStamp = new unsigned long [numberOfRows];

  if ( timeStamp == NULL || tempCdata == NULL || pressData == NULL || fdata == NULL || ldata == NULL || gtempdata == NULL || uvdata == NULL)
  {
    numberOfRows = 0;
    Serial.println("Error in memory allocation!");
  }
  else
  {
    Serial.print("Allocated storage for ");
    Serial.print(numberOfRows);
    Serial.println(" data points.");

    Serial.print("Graph poll period in sec:");
    Serial.println(POLLPERIOD);

    Serial.print("This will equal "); Serial.print((POLLPERIOD * numberOfRows) / 60);
    Serial.print(" minutes of historical data ");
    Serial.print(((POLLPERIOD * numberOfRows) / 60)/60); Serial.println (" Hours of data");
  }
  delay(2000);
}

This is the part that stores data:

  // Check if poll period has expired
  if (millis() >= timeKeeper) {
    timeKeeper = millis() + (POLLPERIOD * 1000);   // Update timeKeeper with the latest time + poll period (multiply by 1,000 as millis() is milliseconds)
    unsigned long currentTime = now();
    // Update each row in the array until it is full
    if (!(isnan(currentTime)) || !(isnan(T)) || !(isnan(P)) || !(isnan(H)) || !(isnan(U)) || !(isnan(G)) || !(isnan(L))) { // Make sure that all values are a number before updating
      if (counters < numberOfRows) {
        tempCdata[counters] = T;   // Temperature
        pressData[counters] = P;   // Pressure
        fdata[counters] = H;
        ldata[counters] = L;
        uvdata[counters] = U;
        gtempdata[counters] = G;
        timeStamp[counters] = currentTime;  //Current time
        counters++;
      }
      else {
        for (int i = 0; i < (counters) ; i++) {       // Cycle the array. Move everything forward by one and add the new values to the end
          tempCdata = tempCdata[i + 1];
          pressData = pressData[i + 1];
          fdata = fdata[i + 1];
          ldata = ldata[i + 1];
          uvdata = uvdata[i + 1];
          gtempdata = gtempdata[i + 1];
          timeStamp = timeStamp[i + 1];
        }
        tempCdata[numberOfRows] = T;
        pressData[numberOfRows] = P;
        fdata[numberOfRows] = H;
        ldata[numberOfRows] = L;
        uvdata[numberOfRows] = U;
        gtempdata[numberOfRows] = G;
        timeStamp[numberOfRows] = currentTime;
      }
    }
  }

This is the part that reads the data out as a json, just the first couple:

void handleChartData () {

  String html = "HTTP/1.1 200 OK\r\n";
  html +=       "Content-Type:json\r\n";
  html +=       "Connection:keep-alive\r\n";
  html +=       "\r\n";
  server.sendContent(html);

  html = "{\"time\":[";
  server.sendContent (html);

  for (int i = 0 ; i < counters ; i++) {
  html = (String(timeStamp));
  if (i + 1 != counters) html += ",";
  server.sendContent (html);
  }

  html = "],\"Itemp\":[";
  server.sendContent (html);
 
  for (int i = 0 ; i < counters ; i++) {
  html = (String(tempCdata));
  if (i + 1 != counters) html += ",";
  server.sendContent (html);
  }
 
And this is the new version that rolls around without shifting up, storing data:

  // Check if poll period has expired
  if (millis() >= timeKeeper) {
    timeKeeper = millis() + (POLLPERIOD * 1000);   // Update timeKeeper with the latest time + poll period (multiply by 1,000 as millis() is milliseconds)
    unsigned long currentTime = now();
    // Update each row in the array until it is full
    if (!(isnan(currentTime)) || !(isnan(T)) || !(isnan(P)) || !(isnan(H)) || !(isnan(U)) || !(isnan(G)) || !(isnan(L))) { // Make sure that all values are a number before updating
      //if (counters < numberOfRows) {
        tempCdata[counters] = T;   // Temperature
        pressData[counters] = P;   // Pressure
        fdata[counters] = H;
        ldata[counters] = L;
        uvdata[counters] = U;
        gtempdata[counters] = G;
        timeStamp[counters] = currentTime;  //Current time
        counters++;
        if (counters > numberOfRows) counters = 0;
        Serial.println ("Data");
        Serial.println (timeStamp[counters]);
        Serial.println (counters);
      //}

And the json builder:

void handleChartData () {

  String html = "HTTP/1.1 200 OK\r\n";
  html +=       "Content-Type:json\r\n";
  html +=       "Connection:keep-alive\r\n";
  html +=       "\r\n";
  server.sendContent(html);

  html = "{\"time\":[";
  server.sendContent (html);

  for (int i = 0 ; i < numberOfRows ; i++) {
  html = (String(timeStamp[i+counters]));
  if (i + 1 != numberOfRows) html += ",";
  server.sendContent (html);
  Serial.println (timeStamp[i+counters]);
  Serial.println (counters);
  }

  html = "],\"Itemp\":[";
  server.sendContent (html);
 
  for (int i = 0 ; i < numberOfRows ; i++) {
  html = (String(tempCdata[i+counters]));
  if (i + 1 != numberOfRows) html += ",";
  server.sendContent (html);
  }

I think its something to do with unsigned long variables as the timestamp always comes out as 0, or a garbled value.






TheMemberFormerlyKnownAsAWOL

...and that's why you should read the instructions before posting.
Please don't PM technical questions - post them on the forum, then everyone benefits/suffers equally

blh64

You are reading beyond your array
Code: [Select]

if (counters < numberOfRows) {
  //....
}
else {
  for (int i = 0; i < (counters) ; i++) {       // Cycle the array.
    tempCdata = tempCdata[i + 1];
    pressData = pressData[i + 1];
    //...
  }
}

When counters reaches numbferOfRows, the else() portion executes.  you then iterator from 0 to counters-1 but access tempCdata[i+1] which will be 1 beyond the end of the array the last time through the for() loop.

As a side note, there is no reason to physically move all your data one index up or down.  Your index pointer should just wrap around like a circular buffer.  It is easier to adjust pointers than move all your data.

Dr_pepper

The strange thing is that the code posted in #2 worked and it was a linear buffer, I changed it so it was a circular buffer and then it didnt work.
The code filled up the buffer, then the else section you mentioned once the buffer was full shifted everything up one and bunged the next value on the end.
Shifting everything up is what wore out the eeprom, thats why I went circular.
I dont get why changong a few lines and remming something something out stopped everything from working.
I have a suspicion its an arrays or a pointer problem.





blh64

Then why don't you post your code that implements your circular buffer with pointers?  The code I quoted is what you posted.

Go Up