Problem uploadig over 100 kb to mega2560 board

Projects General Guidance
1

I´ve been working on a project with arduino and a 320x240 for a long time, and until now everything had worked as expected.
Las day I uploaded a new version of my sketch and it stopped working. The screen would be static in white color or the last image it loaded.
It seamed weird since all i did was add a couple arrays to my code, nothing that could prevent it from even starting (those arrays were going to be used a while after the code started, so they werent even part of the initialization code).
I started trying out some things to get around what happend and I discovered a couple things:
-If we eliminate 11 out of the 12 arrays the code always works, no matter which ot the 12 arrays is left untouched.
-As long as the memory ussage is under 101000 bytes it works, (even it there are two of those arrays set).
-I´ve tried with two different mega boards (which are suppsoed to have 250kb of programe memory) and i get same errors with both.

I´m sorry about not posting the full code, but as you may think it´s to big to fully post.
Still here is an example array (all 12 of them are pretty similar):

const unsigned short lvlh[70] = {piedra1,edge_end1,base_block,base_block,path,path,corner_end8,edge_end2,edge_end2,corner_end2,corner_end1,corner_end6,base_block,base_block,path,path,base_block,base_block,base_block,corner_end8,corner_end6,base_block,base_block,base_block,path,path,base_block,base_block,base_block,base_block,base_block,base_block,preasure_plate,base_block,path,path,path,path,path,path,corner_end5,base_block,base_block,base_block,base_block,base_block,base_block,base_block,base_block,base_block,corner_end4,corner_end5,base_block,base_block,corner_end7,corner_end5,base_block,base_block,base_block,corner_end7,piedra1,corner_end4,edge_end4,edge_end4,corner_end3,corner_end4,edge_end4,edge_end4,edge_end4,corner_end3};

As you see, there are 70 objects in this array (all of them 32x32 images, previously defined)
Right now the full code weitghs 114494 bytes, (45%). As i said there are no compailer errors, nor errors while uploading, and the changes don´t affect the start of the program, so at least something should be displayed before the possible problems started.
I´m using 3322 bytes (40%) of RAM, leaving 4870 bytes left.

Thanks in advanced for the help,
Nick.

2

(deleted)

3

Right, I forgot to mention it, I´m using 3322 bytes (40%), leaving 4870 bytes left.
But even changing all arrays to PROGMEM doesn´t change what happens.
All the images referenced in the arrays are stored in PROGMEM, but I have the arrays in RAM, just so it is easier to manage.

4

I´m sorry about not posting the full code,

I'm not at all sorry for not even bothering to attempt to answer this question.

5

@NickJCSQ, it helps if you ask the right question and give us enough information.

I think the uploading is not the problem. There is no error during uploading ? However, your question is about uploading. The Arduino Mega 2560 did have troubles in the past with the bootloader. When you have a recent official Arduino 2560 board, then you have a recent bootloader. I suppose that uploading is not the problem.

Having pointers in RAM (that point to PROGMEM) is not easier to manage. Perhaps you use the pointers in the wrong way, we have to see the whole sketch. You can upload your sketch to Github if that is easier for you.
When the data and tables with pointers are in PROGMEM, it is easy to make a large buffer in ram and copy the data to ram when it is needed. Do you know the memcpy_P() functions ?

Do you have the project in different files (different tabs in the Arduino IDE) ? That is no problem. We are used to see lots of code. Even at the first glance of your code we have a pretty good idea of your programming skills. Then we can give better advise what to do next.

When the PROGMEM section exceeds the 64k, there might be a problem because far pointers are needed. It is not easy to exceed the 64k limit. That is a lot of data.

Your sketch can probably tell more about what is going on. Just a white screen is very little information. Do you use the serial monitor to dump all kind of information ? There is many information from the compiler that can be shown at runtime. Since the ATmega2560 has enough flash memory, you could show some information, and show when the screen will be initialized and so on.

6

@Koepel I´ve attached all the code, it´s quite a lot and messy, its my first big project.
The sketch uploads correctly to arduino, but the screen that it is connected to stays white and doesn´t do what it is supposed to.
The main tab is "pantalla_tactil" the rest are tabs with functions or structures. The code has almost no commentary and the ones it has are on spanish, sorry for the inconvenience.
The arrays i´ve mentioned in this topic are on "B_lvl" and the objects they are pointing to are on "AA_textures_tiles" (maybe a couple are on "AA_textures_entities") And they are used in "C_lvlinfo" by the structer defined in "B_lvlinfo"

A_entities.ino (3.94 KB)

AA_textures_entities.ino (157 KB)

AA_textures_tiles.ino (192 KB)

B_entities.ino (1.38 KB)

B_fentities.ino (1.27 KB)

B_lvl.ino (10.9 KB)

B_lvlinfo.ino (1.97 KB)

C_entities.ino (163 Bytes)

C_lvlinfo.ino (2.39 KB)

Cf_A_drawBitmapT.ino (1.33 KB)

Cf_A_hud_change.ino (790 Bytes)

Cf_A_hud_setup.ino (799 Bytes)

Cf_B_cuadro_texto.ino (2.83 KB)

Cfs_C_enemies.ino (5.83 KB)

Cfs_C_entities.ino (9.42 KB)

Cfs_C_link.ino (18.4 KB)

Cfs_D_lvlinfo.ino (10.3 KB)

pantalla_tactil.ino (2.45 KB)

z_loop.ino (127 Bytes)

z_setup.ino (412 Bytes)

7

I don't know the libraries that you use, so I can't compile it.
Perhaps the libraries allocate memory from the heap, and perhaps they allocate too much.

Did you write the complete code, or did you copy it from a website ? if you did, which website ?

During setup, the text "frase4" is shown on the display in z_setup.ino.
If that is not working, then something before that is wrong.

I suggest to add more code. At least use the serial monitor to show what is going on.

void setup(){
  delay(500);       // some delay during startup might help

  Serial.begin(9600);
  Serial.println(F( "Sketch has started"));    // this will inform of a restart.
  Serial.print(F( "freeRam="));
  Serial.println( freeRam());
  Serial.print(F( "frase4="));
  Serial.println( frase4);

  //fase inicio pantalla
  myGLCD.InitLCD(); //inicia el LCD
  ...

The "freeRam()" function is the function at the bottom of this page: Arduino Playground - AvailableMemory.
Check if the text of "frase4" is still valid, and print it to the serial monitor.

In the function zsetup() is a while-loop that could stop the sketch.
Perhaps you can add some messages to that as well.

Serial.println(F( "before while (myTouch..."));
while (myTouch.dataAvailable() == false) {} 
Serial.println(F( "after while (myTouch..."));

Your PROGMEM size is about 77kbytes ? That is beyond the 64k border. The normal code compiled from the sketch is in a different segment than the PROGMEM data. The total can be 256kbytes, and that is no problem for code, but the PROGMEM segment beyond the 64k border can be a problem.

Perhaps you can try to change the pgm_read_word(), for example in DrawBitmapT() and collision_static(), to the 'far' function. It should do that automatically as far as I know, but perhaps that is not working.

I have done a test for large PROGMEM blocks years ago, but I can't find it right now.

Do you use the newest Arduino IDE 1.8.5 ?
Old Arduino IDE versions can not handle a PROGMEM block larger than 64k.

Question for others: What is the current status of PROGMEM larger than 64k ?

8

I use libraries UTFT and URTouch, to control de display and the touch control it has.
I wrote all the code myself.
"frase4" is not shown when I have more than a couple arrays set. If I only have one array or two it works, which doesn´t really make sense to me, since they dont interfere with the print of the first sentence.
The while loop you mention is made on purpose, so that after the sentence is printed the program wont continue until you touch the screen (as well as the other things this works perfectly with less than 2 arrays)
I think that what you say about the 64k border makes a lot of sense and could be the problem in my sketch. I´ll look if everuthing is up to date and try the far function you mentioned.
Thanks a lot for your help.

9

So, I´ve tried a couple things and everything is getting even stranger. Despite the screen being frozen when I have the 12 arrays defined the code still works. I´ve set the serial info so i get different messages on different parts of the code. It starts properly, then i get the message that the first sentence is printed (even though the screen is frozen) and if a touch the screen 4 times (the number of times i would need to touch it to finish the message and start the game) i get a serial message saying that the game is starting. That means the full code is running and moreover the touch inputs I give to the screen are working too.
Still, the screen stays frozen, but this only happens when I have 2+ arrays.
Strangely enaugh, everything seams to work, except from the screen.

10

Anything is possible when memory or pointers are overwritten.
The compiler optimizes a lot. That means that if there is code with a memory problem, then something else that might seem unrelated can go wrong.

I suppose that initializing the display is going wrong, and it is pure luck that the rest of the code still works.

When you mention libraries, please give a link to the ones that you have installed. When you use the library manager, a link is not needed.

This seems to be the most recent information: store more then 64k on progmem - Programming Questions - Arduino Forum.
User @AWOL says: "Just use 'far'"

That means you have to use the "pgm_read_word_far()" (see: avr-libc: <avr/pgmspace.h>: Program Space Utilities).
Avoid the memcpy_P() functions, that I mentioned before, they can't handle it. It seems that Arduino does not automatically switch to 'far' functions when the PROGMEM segment is larger than 64k.

The function drawBitmapT() uses something called "bitmapdatatype" for the pointer to PROGMEM. I don't know what kind of pointer that is and I don't know how to make it a far pointer.
I hope you don't have to rewrite the library.

11

http://www.rinkydinkelectronics.com/library.php?id=51 That´s UTFT
http://www.rinkydinkelectronics.com/library.php?id=92 That´s URTouch
http://www.rinkydinkelectronics.com/r_fonts.php And those are the fonts I use

On the links you have both the option to download the library or to acces a pdf where you can see its functions.

I´ve changed all the "pgm_read_word" with "pgm_read_word_far" even one that was in a function defined in the library but it still doesn´t work. It might be that I´m missing a "pgm_read_word" I haven´t changed in the library or my code or that adding far doesn´t solve the problem.

bitmapdatatype is given by the library UTFT. I use it in that fuynction since I wrote it based on the function drawBitmap of the library UTFT. It´s a similar function but I added some variations I needed for my sketch.

The library seams to work perfectly unless I have the 12 arrays defined at the same time.

Thanks for your help

12

So far I have not seen other problems than the PROGMEM segment beyond 64k. That is a real issue, therefor I will only focus on that.

In "UTFT/hardware/avr/HW_AVR_defines.h" is:

#define bitmapdatatype unsigned int*

I'm not sure what that will do. I think that is a near pointer, since normal data is near. Perhaps "uint_farptr_t" is needed.

Is there a way to do a small test, while still having all those PROGMEM defined ?

You could add a new PROGMEM array with numbers at the end of the other arrays. Hopefully that will put it beyond the 64k limit. Use pgm_read_word() and pgm_read_word_far() and print a few numbers to the serial monitor during setup(). See what happens, test if you can change the code for rubbish and good data.

The next step could be to copy the function "drawBitmapT()" to a new "TestdrawBitmapT()" and don't use the parameter to the array with data, but use the array name directly in the function itself with pgm_read_word_far().
If you can make that work, perhaps you can try to change the parameter 'bitmapdatatype' to 'uint_farptr_t'.
If that works, perhaps you can try to change the define in HW_AVR_defines.h to 'uint_farptr_t'.

We live in the Github era. When code is put on Github, others can create issues. Even if those issues are not solved, at least everyone can read the issues to see if they have encountered the same problem. The code in a ZIP file from a website could be bad. The RinkyDink code has been ported to other platforms and copied to Github by some. I don't know if someone has tried a PROGMEM segment beyond 64k.

13

Koepel:
Question for others: What is the current status of PROGMEM larger than 64k ?

Pointers to progmem (actually, pointers to anything) are 16 bit, hence the "64 K barrier".

The [u]pgm_get_far_address[/u] function gets a full 32 bit address of any PROGMEM object. Only "problem" is that it's not dynamic. It provides a static, compile time address for a PROGMEM object, but it can "reach" anywhere.

14

I´ve been experimenting a bit with what you´ve told me and this is what I´ve figured out.
-Changind everything to far and far-readable types doesn´t solve the problem
-Using two arrays (which allows the code to work) I´ve seen that pgm_read_word_far only works with certain objects (I guess that it works with the ones on the 64k+ memory and not with the ones under it) but it doesn´t work with other (presumably the ones under the 64k limit).
-If I use the old code with pgm_read_word (without specifying near or far) it reads everything correcly. So what I think is happening is that the compiler chooses to use near or far depending on the need for the certain object.
-So, in a working code I think I´m using both objects in PROGMEM under the 64k and over, accesing both with "pgm_read-word". The thing is, it still doesn´t work with all the arrays.
-In one of the links you posted I read that someone said that there is another limit (around 70k). If this was true, which I don´t know, it would mean that I´m able to acces memory under 64k (with pgm_read_word_near) over 64k and under seventy-something (with pgm_read_word_far) but not above this second limit. So pgm_read_word chooses between near or far, bu neither of them can go over 70k

This is just what I´ve come up with experimenting a bit.

15

@krupski, it can be used directly in the pgm_read_word_far() function, but not in a 32-bit variable ?

I don't know how to fix it. I don't have time at the moment to test it.
I think there is another way by defining multiple segments instead of one large PROGMEM segment.
Or you could compress the data and unpack it runtime.

16

@Koepel how do compress the data in arduino and then unpack it at runtime?

17

If you have the room, make your arrays 'static' - that way they won't be overwritten .

Allan

18

Koepel:
@krupski, it can be used directly in the pgm_read_word_far() function, but not in a 32-bit variable ?

The pgm_get_far_address macro gives you a 32 bit (actually 24 bit) value that points to the location of PROGMEM data.

For example, you would do this:

#define myData01 pgm_get_far_address(_myData01)

static const uint8_t _myData01[] PROGMEM = {
    0x22, 0x41, 0x41, 0x49, 0x36,
    0x18, 0x14, 0x12, 0x7f, 0x10,
    0x27, 0x45, 0x45, 0x45, 0x39,
    0x3e, 0x49, 0x49, 0x49, 0x30,
    0x61, 0x11, 0x09, 0x05, 0x03,
    0x36, 0x49, 0x49, 0x49, 0x36,
};

Now, "myData01" points to your data block and you would access it as such:

uint8_t siz = (sizeof (_myData01) / sizeof (*_myData01));
uint8_t x;
uint8_t value;

for (x = 0; x < siz; x++) {
    value = pgm_read_byte_far (myData01 + x);
}

OK?

19

Koepel:
@krupski, it can be used directly in the pgm_read_word_far() function, but not in a 32-bit variable ?

You could also copy the value into a 32 bit variable and then use that as a pointer (plus offset) to the PROGMEM data, but why when the #define gives you a usable pointer? Why waste 4 bytes of ram for a uint32_t you don't need?

20

@allanhurst I made my arrays static, but nothing has changed.
@krupski I tried the code you wrote but when I write the define statement you wrote I get this error:
"statement-expressions are not allowed outside functions nor in template-argument lists"