pgmspace array doesn't like lots of 255's ! bug?

Projects Programming
1

hi, this is my first posting - hope this is the right place...

i am coding a hyperbolic waveshaper to produce a distortion audio effect on sampled data. to save time i am hardcoding the transform as an array in program space. the array just maps sample value to output value. at one of of the array i have many zeros and at the other end many 255's. the array is 1024 bytes in size. i generate the values in excel and then copy & paste into my sketch.

when i upload onto my uno i get the error

avrdude: verification error, first mismatch at byte .... verifcation error; content mismatch"

this seemed strange as i had previously used program space successfully to replay raw sample data using an array much bigger. i re-uploaded this program and no such error.

after much tinkering about i discovered that if i limit the transform data to between 0 and 254 instead of 255 i don't get the error!!! so i conclude that having too many (consecutive) 255's in the PROGMEM array causes the code to tip over somehow?

all i am doing initially is printing out the values of the array in a loop -

Serial.println(pgm_read_byte(&hyptran[i]));

if i comment out this line the error message disappears, so it's not the PROGMEM declaration in itself that is the problem -though as i said, reducing the upper values of the array data to 254 instead of 255 clears the error.

any ideas why this is happening anybody?

2

Can you post either your code or a test case so we can replicate the issue?

3 
#include <avr/pgmspace.h>

prog_char hyptran[1024] PROGMEM = {
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,
2,2,3,3,3,3,3,3,3,4,4,4,4,4,5,5,5,5,6,6,6,6,7,7,7,8,8,9,9,9,10,10,
11,11,12,12,13,14,14,15,16,16,17,18,19,20,21,22,23,24,25,26,27,28,29,31,32,33,35,36,38,40,41,43,
45,47,48,50,52,55,57,59,61,63,66,68,71,73,76,78,81,84,87,90,92,95,98,101,104,107,110,113,117,120,123,126,
129,132,135,138,142,145,148,151,154,157,160,163,165,168,171,174,177,179,182,184,187,189,192,194,196,198,200,203,205,207,208,210,
212,214,215,217,219,220,222,223,224,226,227,228,229,230,231,232,233,234,235,236,237,238,239,239,240,241,241,242,243,243,244,244,
245,245,246,246,246,247,247,248,248,248,249,249,249,249,250,250,250,250,251,251,251,251,251,252,252,252,252,252,252,252,253,253,
253,253,253,253,253,253,253,253,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,255,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,
255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255
};

void setup() {
  int i;
  Serial.begin(9600);
  for(i=0;i<1024;i++) {
    Serial.println(pgm_read_byte(&hyptran[i]));
  }
};

void loop() {
};
4

Do you need those semicolons at the end of each function definition ?

5

I see the same problem with that code. It seems to be triggered by the presence of the value 255 from array index 761 onwards. The details of the mismatch given in the error message suggest that the 'faulty' data may be being compared to the value at some other location in the array, almost as if an offset counter was overflowing. It would need some careful experimentation to find out exactly what it was being compared with, and I don't have time to do that. If you could w2ork that out, it might give some clue about the mechanism causing it.

I wonder whether it may be related to the 'relocation truncated to fit' type errors where a linker section ends up bigger than expected and part of the toolset doesn't handle that correctly.

6

Thanks for replicating the error Peter. I was also getting the same index position. I have a bit more time to investigate but probably no more desire as the workaround of rescaling to 254 seems to be holding out... for now! I should really test a bunch of 254's in a larger array to see if something is still overflowing internally.

7

Another option might be to use 2 512 byte arrays or 4 256 byte arrays. It complicates the retrieval a little, but not much

8

Could the problem be that you have the array declared as size 1024, yet only have 992 elements in it?
I count 31 rows of 32 bytes, that's 992.

Try adding one more row of 32 elements,
or take the 1024 out of here:

prog_char hyptran[1024] PROGMEM = {

I don't know if this makes a difference, but I also add a comma after the last element when populating arrays:

255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255, // put a comma at the end?
};
9

The problem stems from what is essentially a bug in avrdude. The latest version is apparently fixed. Appending a 1025th non-255 value to your array should solve the problem.

10

Seen the amount of duplication in the values I would use a runlength encoded version of the array

in the example below an array of ( position, value) pairs is used
the position is searched for and then the associated value is returned.

result --> only 62 entries in the array :slight_smile:

int ar[62] =  //  position, value pairs
{
  384, 0,  // until position 384 the value == 0
  406, 1,
  415, 2,
  421, 3,
  425, 4,
  428, 5,
  431, 6,
  433, 7,
  434, 8,
  436, 9,
  437, 10,
  438, 11,
  439, 12,
  440, 14,
  441, 16,
  442, 239,
  443, 241,
  444, 243,
  445, 244,
  446, 245,
  448, 246,
  449, 247,
  451, 248,
  454, 249,
  457, 250,
  461, 251,
  467, 252,
  476, 253,
  498, 254,
  882, 255,
  -1,-1 // anchor
};

int getValue(int index)
{
  int i = 0;
  int pos = ar[0];

  while (index > pos && i < 60)  // search loop
  {
    i += 2;
    pos = ar[i];
    if (pos == -1) return -1; // out of array range
  }
  return ar[i+1];
}

void setup() 
{
  Serial.begin(115200);

  for(int i=0; i < 883; i++) 
  {
    Serial.print(i, DEC);
    Serial.print(", ");
    Serial.println( getValue(i), DEC);
  }
};

void loop() {
};

There are other run length encodings possible e.g. (count, value) pairs

11

prog_char

and what values can type char have?

12

Thanks coding, your answer seems to be the definitive one I was looking for. Was beginning to question my ability to count after the previous comment :slight_smile:

Project is time critical, don't mind the duplication. Error also occurs using uchar.

13

Apologies Rob if my last seemed a little dismissive, bearing in mind the effort you had taken to post the code. So an explanation - I am transforming audio in realtime at a sample rate of 32Khz. This gives me at best 31.25us for processing the samples (and everything else). So I can't afford to be looping anywhere really or taking up additional clock cycles. If time wasn't a factor I wouldn't even bother using an array as I could just use the tanh function to generate the values real time. However, if memory conservation does become a factor I'll probably just store the thresholds at which 0 and 255 occur and keep the rest of the array in tact. That way I've got only got an additional if statement and a couple of memory reads which shouldn't incur too much time.

14

no need to apologize, it was fun to write

if I look at your original array, I see it can be folded once reducing it in half. It would make the lookup a bit longer but only a tiny bit.

inline int getValue(int idx)
{
  if (idx > 512) return (255 - ar[1024 -idx]);
  return ar[idx];
}

by rearranging the numbers you can make both paths equal in duration (both paths have a subtraction)

inline int getValue(int idx)
{
  if (idx > 512) return (255 - ar[idx]);
  return ar[1024 - idx];
}

Or another variation to reduce the arraysize is to handle 2 big chunks separately. would remove 2x 382 = 760 bytes of the array

inline int getValue(int idx)
{
  if (idx < 382) return 0;
  if (idx > 499) return 255;  // something like 499  
  return ar[idx-382];
}
15

A quick test of the latter idea

uint8_t hyptran[] = {
1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,
2,2,3,3,3,3,3,3,3,4,4,4,4,4,5,5,5,5,6,6,6,6,7,7,7,8,8,9,9,9,10,10,
11,11,12,12,13,14,14,15,16,16,17,18,19,20,21,22,23,24,25,26,27,28,29,31,32,33,35,36,38,40,41,43,
45,47,48,50,52,55,57,59,61,63,66,68,71,73,76,78,81,84,87,90,92,95,98,101,104,107,110,113,117,120,123,126,
129,132,135,138,142,145,148,151,154,157,160,163,165,168,171,174,177,179,182,184,187,189,192,194,196,198,200,203,205,207,208,210,
212,214,215,217,219,220,222,223,224,226,227,228,229,230,231,232,233,234,235,236,237,238,239,239,240,241,241,242,243,243,244,244,
245,245,246,246,246,247,247,248,248,248,249,249,249,249,250,250,250,250,251,251,251,251,251,252,252,252,252,252,252,252,253,253,
253,253,253,253,253,253,253,253,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254
};


inline int getV(int idx)
{
  if (idx < 382) return 0;
  if (idx > 499) return 255;  // something like 499  
  return hyptran[idx-382];
}

volatile int x;

void setup() 
{
  Serial.begin(115200);

  uint32_t start = micros();
  for (int i=0; i< 1000; i++)
  {
    x = getV(i);
  }
  uint32_t stop = micros();
  Serial.println(stop - start);
}

void loop() {
}

output: 1408
so it takes 1.4 usec to do one lookup in the "compressed array" including the loop overhead, so the real number is more around the 1.0 usec. way below the 32 usec you have

you can add more "knowledge" in the function and strip the values 1 and 254 from the array too, reducing the array size with another 90 bytes.

16

some tweaking

uint8_t hyptran[] = {
  1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,
  2,2,3,3,3,3,3,3,3,4,4,4,4,4,5,5,5,5,6,6,6,6,7,7,7,8,8,9,9,9,10,10,
  11,11,12,12,13,14,14,15,16,16,17,18,19,20,21,22,23,24,25,26,27,28,29,31,32,33,35,36,38,40,41,43,
  45,47,48,50,52,55,57,59,61,63,66,68,71,73,76,78,81,84,87,90,92,95,98,101,104,107,110,113,117,120,123,126,
  129,132,135,138,142,145,148,151,154,157,160,163,165,168,171,174,177,179,182,184,187,189,192,194,196,198,200,203,205,207,208,210,
  212,214,215,217,219,220,222,223,224,226,227,228,229,230,231,232,233,234,235,236,237,238,239,239,240,241,241,242,243,243,244,244,
  245,245,246,246,246,247,247,248,248,248,249,249,249,249,250,250,250,250,251,251,251,251,251,252,252,252,252,252,252,252,253,253,
  253,253,253,253,253,253,253,253,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254,254
};


inline uint8_t getV(int idx)
{
  if (idx < 382) return 0;
  if (idx > 499) return 255;  // something like 499  
  return hyptran[idx-382];
}

volatile uint8_t x;

void setup() 
{
  Serial.begin(115200);

  uint32_t start = micros();
  for (int i=0; i< 1000; i++)
  {
    x = getV(i);
    x = getV(i);
  }
  uint32_t stop = micros();
  Serial.println(stop - start);

  start = micros();
  for (int i=0; i< 1000; i++)
  {
    x = getV(i);
  }
  stop = micros();
  Serial.println(stop - start);
}

void loop() {
}

output:
2088 // 2 calls + loop overhead
1248 // 1 call + loop overhead
meaning 1000 calls take 800 usec => 0.8 usec per call . // note this way I subtract the loop overhead !
Note I changed the return value to byte/uint8_t

you might want to measure the PROGMEM variation to reduce RAM usage.

17

I guess there's always a trade-off between speed and memory.

Always when you get the speed from pre-calculated values that are put in a cache, and often in other cases.

Most interesting is if you can find a different formula / approach to the problem that has performance with low memory.
(one often seen in embedded is to replace float math with fixed point integer instead versus using a lookup table)

As an aside, I remember writing a version of "life" many moons ago; for each cell I stored an array containing the memory locations of it's 8 neighbors. So i didn't need to calculate the neighboring positions, boundary checks etc. each iteration. Ran fast as hell... but used 9x times the memory!

Wow, that was no Arduino :wink: Do you remember the speed gain?