Using too much PROGMEM resets PWM timer setting

Hi everyone,

I have this very strange problem: If I use too much PROGMEM for my data (time series I want to output via a PWM pin), the PWM base frequency is set to a very slow value.

What I'm trying to do:

  • Add data to my program by storing it in byte arrays stored in PROGMEM.
  • Read out that data.
  • Output the data on one of the PWM pins, with a high enough frequency.

Some relevant code:

#include <avr/pgmspace.h>

#include "data.h"

// analog output
unsigned int dataPin = 12;

float framerate = 100;

void setup()
{
  Serial.begin(9600);      // open the serial port at 9600 bps

  init_data();

  for (unsigned int ii = 0; ii < digits; ++ii) {
    pinMode(bit8 - ii, INPUT);
  }
  pinMode(dataPin, OUTPUT);

  // Set PWM timer for pins 11 and 12
  // https://arduino-info.wikispaces.com/Arduino-PWM-Frequency
  TCCR1B = TCCR1B & B11111000 | B00000010;
};

void init_data() {
  data[0] = pgm_get_far_address(data0);
  data[1] = pgm_get_far_address(data1);
  data[2] = pgm_get_far_address(data2);
  data[3] = pgm_get_far_address(data3);
  data[4] = pgm_get_far_address(data4);
  data[5] = pgm_get_far_address(data5);
  data[6] = pgm_get_far_address(data6);
  data[7] = pgm_get_far_address(data7);
  data[8] = pgm_get_far_address(data8);
//  data[9] = pgm_get_far_address(data9);
//  data[10] = pgm_get_far_address(data10);
//  data[11] = pgm_get_far_address(data11);
//  data[12] = pgm_get_far_address(data12);
//  data[13] = pgm_get_far_address(data13);
  data[14] = pgm_get_far_address(data14);
  data[15] = pgm_get_far_address(data15);

}

void loop()
{
  unsigned char pos = position();
  play_data(pos);
}

void play_data(unsigned char data_index){
  float framerate = framerates[data_index];
  unsigned int delay_t = round(1000.0/framerate);
  uint32_t len = data_lengths[data_index];
  uint32_t base_ii = data[data_index];
  for (uint32_t ii = 0 ; ii < len; ++ii) {
    uint32_t pgm_pos = base_ii + ii;
    // sets the value (range from 0 to 255):
    unsigned char value = pgm_read_byte_far(pgm_pos);
    analogWrite(dataPin, value);
    // Serial.println(value);
    delay(delay_t);
  }
}

And the data input:

#ifndef DATA_H
#define DATA_H

const float framerates[16] = {
   1000, 1000, 1000, 1000,  // 0..3
   1000, 1000, 1000, 1000,  // 4..7
   1000, 1000, 1000, 1000,  // 8..11
   100, 10, 10, 10,         // 12..15
};


// 256kB / 16 = 16kB
const unsigned char data0[] PROGMEM = {
  #include "data/data1.h"
  //3, 1, 4, 1, 5, 9, 2, 6, 5, 8, 9, 7, 9
};
const unsigned char data1[] PROGMEM = {
  #include "data/data1.h"
};
const unsigned char data2[] PROGMEM = {
  #include "data/data2.h"
};
const unsigned char data3[] PROGMEM = {
  #include "data/data3.h"
};
const unsigned char data4[] PROGMEM = {
  #include "data/data4.h"
};
const unsigned char data5[] PROGMEM = {
  #include "data/data5.h"
};
const unsigned char data6[] PROGMEM = {
  #include "data/data6.h"
};
const unsigned char data7[] PROGMEM = {
  #include "data/data7.h"
};
const unsigned char data8[] PROGMEM = {
  #include "data/data8.h"
};
const unsigned char data9[] PROGMEM = {
  #include "data/data9.h"
};
const unsigned char data10[] PROGMEM = {
  #include "data/data10.h"
};
const unsigned char data11[] PROGMEM = {
  #include "data/data11.h"
};
const unsigned char data12[] PROGMEM = {
  #include "data/data12.h"
};
const unsigned char data13[] PROGMEM = {
  #include "data/data13.h"
};
const unsigned char data14[] PROGMEM = {
  #include "data/data14.h"
};
const unsigned char data15[] PROGMEM = {
  #include "data/data15.h"
};

// we need to use "pgm_get_far_address" to convert the numbers to
// addresses in the larger progmem address space.
uint32_t data[16];

const uint32_t data_lengths[16] = {
   sizeof(data0),  sizeof(data1),  sizeof(data2),  sizeof(data3),
   sizeof(data4),  sizeof(data5),  sizeof(data6),  sizeof(data7),
   sizeof(data8),  sizeof(data9), sizeof(data10), sizeof(data11),
  sizeof(data12), sizeof(data13), sizeof(data14), sizeof(data15)
};

#endif //DATA_H

What works, what does not?
It all works fine so far, but I run into trouble if I add more data. I tested it on the Mega2560, with plenty of flash memory still free. If I add more data (by uncommenting more lines in init_data()), the PWM frequency setting line (last line in setup()) loses its effect and the PWM frequency is set to something very, very slow. The last "good" setting uses slightly below 70kB (but well over 64kB) of flash memory, the first "bad" setting more than 70kB. I used the PWM divider settings from https://arduino-info.wikispaces.com/Arduino-PWM-Frequency, changing the PWM frequency generally works for me on this chip.

Now, what did I do wrong here, if anything? I do not allocate any memory dynamically, so I do not think that this can be a problem with insufficient SRAM.

Now, what did I do wrong here, if anything?

You posted at the wrong site. You should take your snippets to http://snippets-r-us.com.

const unsigned char data0[] PROGMEM = {
  #include "data/data1.h"
  //3, 1, 4, 1, 5, 9, 2, 6, 5, 8, 9, 7, 9
};
const unsigned char data1[] PROGMEM = {
  #include "data/data1.h"
};

If you have a guard set in data1.h then, since the guard will be set in populating data0, data1 will not get populated.

You could try including the data after your code (you might need some forward references).

stowite:
If you have a guard set in data1.h then, since the guard will be set in populating data0, data1 will not get populated.

Good point, but no, dataX.h are mostly auto-generated files with just the numbers. (Generated from numpy arrays with a small script.)

Did you consider my suggestion that you move the executable code to before the data? It seems to me a possibility that the data is pushing the executable code to beyond the 'far' boundary and that this is what is causing the problem. Moving the code to before the data may solve the problem. Of course it may make no difference what so ever but what does it cost you to try it out.

@stowite Yes, I tried to put the data to the back but failed with the forward declaration. Forward declaration of arrays of undefined size is not allowed by the compiler:

extern const unsigned char[] data0; // This will fail.

And if I try to use straightforward pointers, defining the content as an array later will also fail:

extern const unsigned char* data0;
const unsigned char[] data0 PROGMEM = {...}; // This will also fail.

Any help on this?

A little contorted but this worked for me -

extern const uint8_t* const data0 PROGMEM;
extern const size_t data0_len PROGMEM;

void setup() {
Serial.begin(19200); 

for (uint16_t i = 0; i < data0_len; i++)
{
  const uint8_t v = pgm_read_byte_far(data0+i);
  Serial.println(v);
}
}

void loop() {
  
}


const uint8_t data0x[] PROGMEM = {1,2,3,4,5}; 

const size_t data0_len PROGMEM = sizeof(data0x);
const uint8_t * const data0 PROGMEM = data0x;

You know storing small bits in PROGMEM is not worth it? So it's not that useful to put data0_len and data0 in PROGMEM.

And why can't you include the data on top of the file?

septillion:
You know storing small bits in PROGMEM is not worth it? So it's not that useful to put data0_len and data0 in PROGMEM.

And why can't you include the data on top of the file?

This is to attempt to get round a problem that the OP was having when putting his LARGE array of constant data at the top of the file! I suggested that placing it at the end and using forward references might solve the problem. It may not solve the OP's problem but ... .

Ahh, yeah, missed that part. But I doubt it matters...

septillion:
Ahh, yeah, missed that part. But I doubt it matters...

I agree but it might !

I have never done forward referencing of PROGMEM before so I learned something and found it interesting.

stowhite: Thanks for the hint, I will try this tomorrow. I have the impression, that, if I was able to reliably reproduce this, it might actually be a gcc bug...

I'm no longer convinced I'm actually creating valid PROGMEM forward references. I was working with a simple test harness containing just two PROGMEM arrays and it seemed to work OK but then adding a third produced silly results. This even without worrying about 'near' or 'far' pointers.

The reference I have been using is avr-libc: <avr/pgmspace.h>: Program Space Utilities but the notes have some worrying statements. For instance "This is normally not a big concern as the linker setup arranges any program space constants declared using the macros from this header file so they are placed right after the interrupt vectors, and in front of any executable code. However, it can become a problem if there are too many of these constants, or for bootloaders on devices with more than 64 KB of ROM. All these functions will not work in that situation." . To me this implies that even if one tries to place the PROGMEM arrays after the program code the the linker will place them before the program code! This, of course, will defeat the aim of the whole forward reference approach.

The problem still interests me so I shall continue looking at it but I for the moment I think it is beyond my expertise.

@quazgar, can you tell me how big is to big? Aka, what size array do you try to store in PROGMEM when PWM fails? Or even better, post a sketch that does (you can use the include file function).

stowite:
I'm no longer convinced I'm actually creating valid PROGMEM forward references. I was working with a simple test harness containing just two PROGMEM arrays and it seemed to work OK but then adding a third produced silly results. This even without worrying about 'near' or 'far' pointers.

Yes, when I use your suggestion (using data0, data0x and then setting the plain array data0 to data0x, thanks, that looks very creative to me :slight_smile: ), it does not work at all anymore. I assume that it also may have something to do with pgm_get_far_address(). According to the documentation on that site, it needs to resolve the address at linking time, and I'm not sure how that works out with the deferred initialization of the PROGMEM variables. I am afraid I will have to live with using only part of the available flash memory.

septillion:
@quazgar, can you tell me how big is to big? Aka, what size array do you try to store in PROGMEM when PWM fails? Or even better, post a sketch that does (you can use the include file function).

Yes, of course. It still works with a total program size of 69924 bytes. The error occurs when including the next data set, resulting in a size of 73176 bytes.

I am attaching a working copy of the program as a zip file for everybody to work on and also pushed my local repository to GitHub - quazgar/play_timeseries: Playing a timeseries on an Arduino.

play_timeseries.zip (113 KB)

That is indeed a hell lot of data... I mean, data0 alone is already almost 10k :o

Yes, nearly 10s at 1kHz. Originally I was looking forward to about 4 minutes at that resolution.