Compile array larger than 65k

Hello there. Lemme explain what's the deal.

I'm trying to store into the program's memory (PROGMEM), a really large array of bytes (actually a 2 seconds and 1 sec of 22 KHz 8-bit stereo samples); but this won't let me compile due to the following reason:

too many initializers for 'const unsigned char [798]'

And this is the part of the code that doesn't let me compile:

#include <avr/pgmspace.h>

const unsigned long far_len = 132554UL;
const unsigned char far[far_len] PROGMEM = {/* lots of bytes */}

I guess the compiler only can handle 16-bit pointers (thus below 65k is fine). Is it true?

So, is there a way can I compile it? Or should I appeal for a workaround in the code?

Best regards.

PD: to program such large array, I'm obviously using an Arduino Mega.

Tried it here and it compiles just fine. But the error message isn't complaining about the size of the array...

The problem is here:

Lucario448:

{/* lots of bytes */}

The compiler is complaining that "lots of bytes" is more then the 798 bytes you told it it is... So maybe do a recount :wink:

And again, it shows the forum rules are there for a reason... Don't post snippets!

septillion:
And again, it shows the forum rules are there for a reason... Don't post snippets!

Oh should I post the whole thing? Well, there it is in the attachment, because the arrays wouldn't let me post.

It's in a zip file; it contains the sketch itself, and the header files containing the large arrays.

It still might not compile but for other reasons I will later solve.

The sketch uses this library.

Thanks! :slight_smile:

AudioSamplesMega.zip (282 KB)

Update: if the previously provided library's link appears to not let compile the sketch, then try to download it here.
I totally forgot where I got it... :confused:

Thanks again.

Unless there's some sort of FAR pointer on the MEGA (which I doubt), pointers on the Arduino are 16 bits so it can't address more than 65536 bytes of memory.
Looks like you'll need to invest in a 32-bit beast.

Pete

el_supremo:
Unless there's some sort of FAR pointer on the MEGA (which I doubt), pointers on the Arduino are 16 bits so it can't address more than 65536 bytes of memory.
Looks like you'll need to invest in a 32-bit beast.

Pete

Only PROGMEM pointers on the mega have a far variant (24bit). But these are only used to reach data in flash that is located over the 65k boundary. The data types stored there are still limited to 16 bit widths.

#define ADDR(var)({uint_32t a; asm("ldi %A0, lo8(%l)\n ldi %B0, hi8(%l)\n ldi %C0, hlo8(%l)\n ldi %D0, hhi8(%l)": "=d"(a): "p"(&(var))); a;})

I found somewhere that this macro generates a 32-bit pointer; but I have no idea at all about how to use it.

pYro_65:
Only PROGMEM pointers on the mega have a far variant (24bit). But these are only used to reach data in flash that is located over the 65k boundary. The data types stored there are still limited to 16 bit widths.

Yeah, it will be funny if the 2560 microcontroller couldn't address all its program memory. :o

I think I have an idea: I could workaround the data type limitation by splitting up the arrays, but I'm afraid about if that makes a non-contiguous array writing (one followed by the other).

Also I guess I can use the pgm_read_byte_far function to deal with 24/32-bit pointers; however, how can I retrieve (or even create) a pointer of that size? (not a long* pointer)

Update: now I splitted up the arrays into ones of a length, at most, of 65535 (aka unsigned 16-bit maximum value).

However, compiler is still complaining; now it says that the "size of array 'nameOfArray' is too large".
What's the maximum length then? 32k? (32767)

I'll appreciate any answer...

Feel free to share any details you may deem necessary.

Lucario448:
What's the maximum length then? 32k? (32767)

I'll appreciate any answer...

Yes: https://arduino.land/FAQ/content/6/7/en/what-is-the-maximum-number-of-elements-an-array-can-have.html

All right thank you very much!

I also had to remove the macro usage, in order to finally compile my sketch, but... I can't listen anything

My idea is to output some audio by using a 8-bit-timer-driven PWM signal, changing the duty cycle by using the OCRxA and OCRxB registers.
I have the idea that 2560's timer0 is an 8-bit one; and it maps to the Arduino pins 4 and 13. Am I correct?

Furthermore, I tried to use a function that changes the prescaler of timer0, taken from the Arduino Playground (see code below); yet don't know if it's actually valid for the 2560's timer0.

Ladies and gentlemen, here's the code:

#include "close0.h"
#include "close1.h"
#include "close2.h"
#include "far0.h"
#include "far1.h"
#include "far2.h"
#include "far3.h"
#include "far4.h"
#include <TimerOne.h>
// Files and library required on this sketch

volatile unsigned long count = 0;

void setup() {
  //pinMode(2, INPUT_PULLUP);
  pinMode(4, OUTPUT); // Left
  pinMode(13, OUTPUT); // Right
  // setting up the pin.
  setPwmFrequency(5, 1); // I use 5 because this function is made for ATmega328P, but it should still set timer0's prescaler
  // Adjust the PWM frequency to the highest possible. Thus, the second argument must ALWAYS be 1.
  // Do not use the Timer0 (pins 5, & 6), otherwise "delay" and "millis" functions may not work properly.
  analogWrite(4, 1); // Initialize PWM mode on that pin (timer)
  analogWrite(13, 1);
  Timer1.initialize();
  cli();
  Timer1.attachInterrupt(writeSamples, 45); // 45 microseconds should yield an interrupt rate of 22 KHz,
// which is the intended sampling rate.
  sei();
  pgm_read_byte(&close0[0]);
  pgm_read_byte(&close1[0]);
  pgm_read_byte(&close2[0]);
  pgm_read_byte(&far0[0]);
  pgm_read_byte(&far0[0]);
  pgm_read_byte(&far1[0]);
  pgm_read_byte(&far2[0]);
  pgm_read_byte(&far3[0]);
  // Forces compiler to write these arrays, even if we apparently aren't using them.
  }

void loop() {}

void writeSamples() { // Thankfully TimerOne library allows me to declare a timer-based ISR more friendly
  if (count == 253952UL) // 253952 bytes is the 2560 total program memory, right?
  count = 0;
  
  OCR0B = pgm_read_byte_far(count); // I think the "far" function is for 24 or 32-bit pointers.
  count++;
  OCR0A = pgm_read_byte_far(count);
  count++;
  // Scan throughout the program memory to figure out where the heck are the samples (arrays) written.
  // Not by showing up, but by listening. I will later figure it out by dumping on a SD card.
}

void setPwmFrequency(int pin, int divisor) {
  // A function that modifies the PWM frequency of the specified pin.
  // Taken from http://playground.arduino.cc/Code/PwmFrequency
  byte mode;
  if (pin == 5 || pin == 6 || pin == 9 || pin == 10) {
    switch (divisor) {
      case 1: mode = 0x01; break;
      case 8: mode = 0x02; break;
      case 64: mode = 0x03; break;
      case 256: mode = 0x04; break;
      case 1024: mode = 0x05; break;
      default: return;
    }
    if (pin == 5 || pin == 6) {
      TCCR0B = TCCR0B & 0b11111000 | mode;
    } else {
      TCCR1B = TCCR1B & 0b11111000 | mode;
    }
  } else if (pin == 3 || pin == 11) {
    switch (divisor) {
      case 1: mode = 0x01; break;
      case 8: mode = 0x02; break;
      case 32: mode = 0x03; break;
      case 64: mode = 0x04; break;
      case 128: mode = 0x05; break;
      case 256: mode = 0x06; break;
      case 1024: mode = 0x7; break;
      default: return;
    }
    TCCR2B = TCCR2B & 0b11111000 | mode;
  }
}

If you are going to attempt to compile it, then ask me for the necessary files; or just delete all includes except TimerOne, and the pgm_read_byte lines on the setup.

Thank you one more time! :slight_smile:

PD: hope that the timer0 has been set to "fast PWM" mode by default, and not to "phase-correct" mode.

Hello again, I'm back right now.

Finally I solved the problem of the 32-bit pointers, and arrays sizes.
Everything looked good to go... but, there's another problem: I got no sound

This is the code I'm currently using:

#include "close0.h"
#include "close1.h"
#include "close2.h"
#include "far0.h"
#include "far1.h"
#include "far2.h"
#include "far3.h"
#include "far4.h"
// 8-bit LPCM stereo samples

#include <TimerOne.h>

#define GET_FAR_ADDRESS(var)                          \
  ({                                                    \
    uint_farptr_t tmp;                                \
    \
    __asm__ __volatile__(                             \
        \
        "ldi    %A0, lo8(%1)"           "\n\t"    \
        "ldi    %B0, hi8(%1)"           "\n\t"    \
        "ldi    %C0, hh8(%1)"           "\n\t"    \
        "clr    %D0"                    "\n\t"    \
        :                                             \
        "=d" (tmp)                                \
        :                                             \
        "p"  (&(var))                             \
                        );                                                \
    tmp;                                              \
  })
// Here's how I get 32-bit (or 24-bit) pointers

volatile unsigned int sampleCount = 0;
volatile byte pointerIdx = 0;
volatile boolean selector = HIGH;

uint_farptr_t cp[3];
uint_farptr_t fp[5];
const unsigned int cl[] = {close0_len, close1_len, close2_len};
const unsigned int fl[] = {far0_len, far1_len, far2_len, far3_len, far4_len};
// Covinient way to address the audio data

void setup() {
  pinMode(2, INPUT_PULLUP);
  pinMode(4, OUTPUT); // Left (OCR0B)
  pinMode(13, OUTPUT); // Right (OCR0A)
  cp[0] = GET_FAR_ADDRESS(close0);
  cp[1] = GET_FAR_ADDRESS(close1);
  cp[2] = GET_FAR_ADDRESS(close2);

  fp[0] = GET_FAR_ADDRESS(far0);
  fp[1] = GET_FAR_ADDRESS(far1);
  fp[2] = GET_FAR_ADDRESS(far2);
  fp[3] = GET_FAR_ADDRESS(far3);
  fp[4] = GET_FAR_ADDRESS(far4);
  // I tested it out, and it actually give me the pointers I need; because some of them point out of the 64 KB range

  TCCR0B = TCCR0B & 0b11111000 | 0x01; // Timer0, prescaler 1. Right?

  analogWrite(4, 1); // Initialize PWM mode on that pin (timer)... I guess
  analogWrite(13, 1);
  
  Timer1.initialize();
  cli();
  Timer1.attachInterrupt(writeSamples, 45); // attachInterrupt(ISR, 22050 Hz)
  sei();
}

void loop() {
  // Nothing yet...
}

void writeSamples() {
  if (selector) {
    OCR0B = pgm_read_byte_far(fp[pointerIdx] + sampleCount); // Even indices for left channel
    sampleCount++;
    OCR0A = pgm_read_byte_far(fp[pointerIdx] + sampleCount); // Odd indices for right channel
    sampleCount++;
    if (sampleCount == fl[pointerIdx]) { // End of the fragment?
      sampleCount = 0;
      pointerIdx++; // Next one 
      if (pointerIdx == 5) { // No more fragments?
        pointerIdx = 0; // Reset all counters back to zero, in order to create an endless loop
        selector = PINE & 0x10; // Fast digitalRead(2)
        // Or change the sound depending of the pin state
      }
    }
  } else { // Ditto (except for the fact that points to another sound)
    OCR0B = pgm_read_byte_far(cp[pointerIdx] + sampleCount);
    sampleCount++;
    OCR0A = pgm_read_byte_far(cp[pointerIdx] + sampleCount);
    sampleCount++;
    if (sampleCount == cl[pointerIdx]) {
      sampleCount = 0;
      pointerIdx++;
      if (pointerIdx == 3) {
        pointerIdx = 0; 
        selector = PINE & 0x10;
      }
    }
  }
  /*
     Fragmentation occurs due to this couple of reasons:

     1. Annoying 32767 bytes array's length limit on AVR
     2. Arrays are written contiguously, but not in the right order. By playing it all continuously, you would listen a messed up sound.
  */
}

But no clue what's going on.

Is not a hardware problem, because timer0 also maps to pin 13 (and so to the built-in LED); and I'm not getting any light out of there.

In a nutshell: pins 4 and 13 are not giving me any output, and the question is: why? :confused:

I find error in your code
This work for me

#define ADDR(var)({uint32_t a; asm("ldi %A0, lo8(%1)\n ldi %B0, hi8(%1)\n ldi %C0, hlo8(%1)\n ldi %D0, hhi8(%1)": "=d"(a): "p"(&(var))); a;})