Arduino Mega 2560 oddity: Large PROGMEM arrays break basic functionality

Hello! (My first forum post. Apologies in advance if this topic has been dealt with before!)

So, I have myself a Mega 2560 and am in the midst of a data-driven project. That is to say, I have a sketch with about 5kb of code and about 128kb of data spread across several dozen 4kb PROGMEM arrays. It all works awesomely, but for one thing: The good old pinMode()/digitalWrite() functions don’t work anymore. If I comment out my big data arrays, they work fine, but with the data in the sketch, they simply don’t do anything. They become no-ops. And there are no compiler/linker warnings of any kind.

I can directly manipulate the port registers just fine, so I’m not blocked, but this is sufficiently fishy that I figured I’d ask and see if my description rings any bells. I haven’t dumped out the ELF and examined the assembly listing, but that’s probably my next step. I can only imagine that my own code and the library code are ending up on opposite sides of my data arrays or something, and that the linker is doing the wrong thing with the long calls, but that would be a pretty fundamental bug, so my confidence is not high.

Thanks for your thoughts and advice!
Best regards,
Aaron

It might help if you post the code; or an simplified example that exhibits the behaviour.

Sure. Here's a simple example. As is, this will blink the onboard LED at 1Hz. Uncomment the first line, however, and it won't do anything. Working with Arduino 1.8.1, FWIW.

Directly manipulating PB7 works in either case. It's only the pinMode()/digitalWrite() stuff that stops working. cue spooky music

//#define MISBEHAVE

#ifdef MISBEHAVE
const uint8_t bigArray0[32767] PROGMEM = {0};
const uint8_t bigArray1[32767] PROGMEM = {0};
const uint8_t bigArray2[32767] PROGMEM = {0};
const uint8_t bigArray3[32767] PROGMEM = {0};
#endif

void setup()
{
#ifdef MISBEHAVE
  pgm_get_far_address(bigArray0);
  pgm_get_far_address(bigArray1);
  pgm_get_far_address(bigArray2);
  pgm_get_far_address(bigArray3);
#endif

  pinMode(13, OUTPUT);
}

void loop()
{
  digitalWrite(13, HIGH);
  delay(500);
  digitalWrite(13, LOW);
  delay(500);
}

These probably need to the the far versions for the 2560...

https://github.com/arduino/Arduino/blob/master/hardware/arduino/avr/cores/arduino/Arduino.h#L171-L184

Or, if possible, the pin / port mapping arrays need to be forced into the first 64 KiB.


I don't see an issue for this problem...

Ah! Yes, the port/pin mapping arrays. That would make a lot of sense. So, those calls may well be doing something, but there's no telling what. Yikes! That seems a nasty bug. Will examine your links when I'm back in front of a proper computer. Thanks a bunch!

Here is an open issue: 64K problems: pins_arduino.h _PGM arrays pushed out of range of pgm_read_byte() · Issue #174 · arduino/ArduinoCore-avr · GitHub

Thanks oqibidipo! Sanity restored. (Sorry; didn’t mean to make this a quest for anybody. Now that I see where the issues are kept, I’ll try my hand at finding them next time!)

So from what I read there, it seems that:
a) Nobody is too eager to fix this (it’s been known for at least 2-1/2 years), possibly because the proposed fixes rely on internal details of GCC’s section naming/sorting, and so could be broken again easily.
b) The best way for me to handle this near-term is to just twiddle bits in the port registers myself (because using private copies of public headers is pretty much always a bad idea)

Pondering this more, I’m really quite surprised that the GCC toolchain permits addresses to get truncated like this without offering so much as a warning (maybe those are being suppressed somewhere). It’s really quite dangerous to have code compile/link and yield a binary full of incorrect addresses.

Thanks again, all, for the quick help! Really appreciate it!!

Best,
Aaron

jaholmes:
a) Nobody is too eager to fix this (it's been known for at least 2-1/2 years),

I suspect creating a pull request would help.

possibly because the proposed fixes rely on internal details of GCC's section naming/sorting, and so could be broken again easily.

Doubt it. The most likely reason is the perceived number of people who have crossed paths with the problem (very very few) when compared with the possible solutions (just get a 32 bit board) coupled with the Arduino folks having to prioritize limited resources.

b) The best way for me to handle this near-term is to just twiddle bits in the port registers myself (because using private copies of public headers is pretty much always a bad idea)

You could try applying the modifications. If that works without side-effects that would give the Arduino folks a much needed test case (assuming you follow-up on that issue).

Thanks! I'll dig a little deeper when I have a chance and see if this makes sense. Hacking around this with section names still feels a bit...well...hacky. But perhaps that's the best that can be done to "fix" this in the near term.

Yeah, I pondered the 32-bit route, but decided to stay the 5V course because my project makes heavy use of vintage 5V memories and memory-like devices. The 2560 was at the intersection of many conveniences.

Thanks again,
Aaron

jaholmes:
Hello! (My first forum post. Apologies in advance if this topic has been dealt with before!)

So, I have myself a Mega 2560 and am in the midst of a data-driven project. That is to say, I have a sketch with about 5kb of code and about 128kb of data spread across several dozen 4kb PROGMEM arrays.

Is the AVR GCC compiler able to use more than 64 KB of PROGMEM constants on a single sketch, meanwhile?

When I tried about 3 years ago, it seemed to me as if there was a 64KB barrier and the PROGMEM segment was limited to a size 64 KB maximum.

At that thime (I think I used Arduino IDE version 1.0.5 at that time), it was very hard work for me to come around the 64 KB PROGMEM segment size and read all data as I wanted it.

While it was easy to declare and compile more than 64 KB of PROGMEM constants for a MEGA2560 board at a first glance, it was a nightmare to read all data from program code in the same way they were declared in the different PROGMEM arrays.

And now you can use a dozen of arrays and read about 128KB of PROGMEM without any problems?
Which IDE version?

If I were you, I'd not assume aproblem with digitalWrite, but I'd assume a problem with reading data from an over-sized PROGMEM data segment.

It's to do with 16 bit near pointers and 32 bit far pointers.

http://www.nongnu.org/avr-libc/user-manual/group__avr__pgmspace.html

But just how PROGMEM dices with those I dunno.

Just to throw a bit extra in, you could add external direct-address RAM to the 2560 chip family.
I have a Rugged Circuits 512K RAM card that plugs into the back double row of pin holes. It works out to 8 banks of 56K RAM since the lower 8K addresses are the internal RAM that get used as dedicated stack space (configurable) while the external banks become switchable heap. That card ran me about $28 including shipping.

jurs:
If I were you, I'd not assume aproblem with digitalWrite, but I'd assume a problem with reading data from an over-sized PROGMEM data segment.

There is no assumption to be made. I inspected the disassembled program. The problem is exactly what is described in that issue. The pin / port mapping arrays are being placed outside of the first 64 KiB.

GoForSmoke:
But just how PROGMEM dices with those I dunno.

Within a given section, the usual way linkers deal with such things is first-come-first-served. The first thing "seen" by the linker is placed first in memory.

The solution proposed by @westfw forces the linker's hand by moving the arrays to a different section.

jurs:
While it was easy to declare and compile more than 64 KB of PROGMEM constants for a MEGA2560 board at a first glance, it was a nightmare to read all data from program code in the same way they were declared in the different PROGMEM arrays.

And now you can use a dozen of arrays and read about 128KB of PROGMEM without any problems?
Which IDE version?

If I were you, I'd not assume aproblem with digitalWrite, but I'd assume a problem with reading data from an over-sized PROGMEM data segment.

I'm using 1.8.1, and no trouble with the 128+kb of PROGMEM, except for the issue noted in this thread. My main gripe is the necessity of prg_get_far_address(), which seems like a brutal hack. One really ought not to need to execute code at runtime in order to fetch the address of global data--which is known to the linker. So where you might be inclined to break a large array into several smaller arrays and then have another array of pointers to these, you have to remember that you can't initialize that array of pointers at build time. You have to populate it at runtime by calling prg_get_far_address() a bunch of times. I guess this just isn't a common enough problem to deserve first-class support in the compiler/linker, but ideally if you're taking the address of something and storing it as 32 bits, the tools would just see this and actually store all 32 bits (or >16 bits, anyway) rather than truncating.

[/quote]

GoForSmoke:
Just to throw a bit extra in, you could add external direct-address RAM to the 2560 chip family.

Yep, I've thought about it. In this project, I'm already using the external memory interface to communicate with some old FM synthesizer chips, and it works quite well (I have to drop the clock speed from 16 to 8MHz to get the wait states right, though). The data stored in the MCU's flash is music data. Ultimately, the cool thing to do will be to evict all of that from the MCU and stick it on an SD card or something. Not ready to advance to that quite yet, but probably will soon. The 256kb of flash in the Atmega2560 is only enough for a few minutes of synthy goodness. :slight_smile:

SD is a hell of a lot slower than flash or 2560 external RAM.

There is also SPI RAM if you don't mind the slowdown.

GoForSmoke:
SD is a hell of a lot slower than flash or 2560 external RAM.

There is also SPI RAM if you don't mind the slowdown.

Yeah, speed is definitely not a concern in this case. Data access is purely sequential, and the maximum data rate needed is only 1.5-2kb/s. The only real concern is data size. I've already set the clock divisor to 2 and cranked the XMEM wait states to max in order to deal with these old synth chips, so nothing about this really screams "performance" :slight_smile:

You don't have much experience with file operations?

GoForSmoke:
You don't have much experience with file operations?

Not on the Arduino specifically, no, but I've glanced at the SD library and it appears to expose a familiar set of standard C-style file APIs. I don't foresee any difficulty there. Rather, I simply wanted to get the meat of the thing working before I started polishing it. For now, I can stuff a few good tunes into the atmega2560's flash.

Tunes?

You can buy an SD shield or spend less on an SD module or spend even less and make one out of an SD adapter sleeve for micro-SD cards.

GoForSmoke:
Tunes?

Tunes!

That's about 259kb of commands and timing data Huffman-coded down to about 128kb. Lots of inefficiencies in the underlying byte stream, which I'll tackle at some point. I modified the MAME emulator to log out the interactions between the sound CPUs and the synths, and this project plays those logs back against the real hardware. Reverse emulation!

This is certainly not the first Arduino (Mega or otherwise) project to use these old 1980's-vintage Yamaha FM synthesizer ICs, but surprisingly, it may be the first to use the MCU's external memory interface, which you'd think would have been the obvious thing to do. shrug It reduces pages of bit-banging down to about six lines of code.

GoForSmoke:
You can buy an SD shield or spend less on an SD module or spend even less and make one out of an SD adapter sleeve for micro-SD cards.

I'll be making a PCB as soon as I finalize the analog bits, and will probably stick an SD slot on there. The Uno-style headers on the Mega will be left alone for now, as it's likely I'll mate this with a MIDI shield at some point.

If you could drve an ISA card you wouldn't have to make a PCB. There's plenty of old PC/XT & PC/AT soundcards for sale.