Arduino stucked- how to command to utilize of FULL MEM to display

Dear Experts

My sample codes provided by supplier of lcd can only display less than 64k of pic array data ,but i want to use the whole flash memory for my codes & many pictures display but now i am stucked with only can use less than 64k of my flash memory for pictures!

My Arduino Mega 2560 using PROGMEM array to display pictures on Lcd hangs when my sketch + PROGMEM arrays gets my sketch larger

I not sure if i need to change GUI_paint.cpp's "pgm_read_byte" to a 32bit commands to access the whole 256k addresses of flash.

My Lcd iS 2.4 inch Lcd.

My LCD_mega2560.ino file mainly uses these 4 files for display pics on Lcd :

  1. image.h
  2. image.cpp
  3. GUI_paint.h
  4. GUI_paint.cpp

I extracted parts of code for your easy viewing.
Codes below from GUI_paint.h

//pic
void Paint_DrawImage(const unsigned char *image,UWORD Startx, UWORD Starty,UWORD Endx, UWORD Endy);

Codes below from image.cpp

#include "image.h"
//HORIZONTAL 3D ICON
const unsigned char gImage_3D_batt1percent[9760] = { /* 0X00,0X0C,0X50,0X00,0X3D,0X00,0X00,0X1B, */
  0XFF, 0X0F, 0XFF, 0X0F, 0XFF, 0X0F, 0XFF, 0X0F, 0XFF, 0X0F, 0XFF, 0X0F, 0XFF, 0X0F, 0XFF, 0X0F,
  0XFF, 0X0F, 0XFF, 0X0F, 0XFF, 0X0F, 0XFF, 0X0F, 0XFF, 0X0F, 0XFF, 0X0F, 0XFF, 0X0F, 0XFF, 0X0F, . . . . . . . and a lot more to the end of array

I need kind assistance from dear experts online :slight_smile: as i tried for 1 week and no success.

You will have to keep some tables of the core below 64k (like translation tables for digitalWrite),
or your sketch will probably crash/malfunction.

Put your data chunks behind those, if I remember correctly,
with some attribute specifying the segment.

You will need to use far access for data beyond 64k.

I once made a small shell that can dump RAM, FLASH, and EEPROM,
maybe you can use some of it. dumpFF dumps from far PROGMEM.

#include <Wire.h>

#define AF(ptr) ((__FlashStringHelper*)ptr)

const byte maxMsgLen = 64;

const byte cbLF = 10;
const byte cbCR = 13;

void i2cScan();

typedef byte (*accessFunction)(const byte *adr);

void dumpR(const byte* adr, int len, const char* pfx = NULL);
void dumpF(const byte* adr, int len, const char* pfx = NULL);
void dumpFF(unsigned long adr, int len, const char* pfx = NULL);
void dumpE(const byte* adr, int len, const char* pfx = NULL);
void dumpAsc();
void dumpGen(const byte* adr, int len, accessFunction accs, const char* pfx, bool withAdr = true, int suppHex = -1);
//void withOneByte(const char* buf, void (*alAction)(uint8_t));
void withAdrLenPrefix(const char* buf, void (*alAction)(const uint8_t*, int, const char*), const char* pfx = NULL);
void withAdrLenPrefixLong(const char* buf, void (*alAction)(unsigned long, int, const char*), const char* pfx = NULL);
void dumpVA(unsigned long adr, int len, const char* pfx);

void setup() {
  Wire.begin();
  Serial.begin(250000);
  Serial.println(F("Minishell v1.02.001"));
}

void loop() {
  //char* x = nullptr;
  handleSerial();
}

void oneLineReceived(const char * buffer) {
  switch (*buffer) {
    case 0:
      Serial.println();
      return;
    case 'i':
      i2cScan();
      break;
    case 'd':
      switch (*++buffer) {
        case 'r':
          withAdrLenPrefix(buffer, dumpR, PSTR("R "));
          break;
        case 'f':
          withAdrLenPrefixLong(buffer, dumpVA, PSTR("F "));
          break;
        case 'e':
          withAdrLenPrefix(buffer, dumpE, PSTR("E "));
          break;
        case 'a':
          dumpAsc();
          break;
        default:
          Serial.println(F("d[rfe] adr, len/val | da"));
          break;
      }
      break;
    default:
      Serial.print(F("unknown command '"));
      Serial.write(*buffer);
      Serial.println(F("' try one of 'id'."));
      Serial.print(F(""));
  }
}

void handleSerial() {
  static uint8_t bIndex;
  static uint8_t buffer[maxMsgLen + 1];
  bool lineReady = false;
  if (Serial.available()) {
    uint8_t inChar = Serial.read();
    if (inChar != cbLF) {
      if (inChar == cbCR) {
        lineReady = true;
      } else {
        buffer[bIndex++] = inChar;
        lineReady = (bIndex == maxMsgLen);
      }
      if (lineReady) {
        buffer[bIndex] = 0;
        oneLineReceived((const char*)buffer);
        bIndex = 0;
      }
    }
  }
}

void withAdrLenPrefix(const char* buf, void (*alAction)(const uint8_t*, int, const char*), const char* pfx) {
  int len = 16;
  char* satis;
  uint8_t* res = (uint8_t*)strtol(++buf, &satis, 0);
  if (*satis == ',' || *satis == ' ') {
    len = (int)strtol(++satis, &satis, 0);
  }
  (*alAction)(res, len, pfx);
}

void withAdrLenPrefixLong(const char* buf, void (*alAction)(unsigned long, int, const char*), const char* pfx) {
  int len;
  char* satis;
  unsigned long res = strtoul(++buf, &satis, 0);
  if (*satis == ',' || *satis == ' ') {
    len = (int)strtol(++satis, &satis, 0);
  } else {
    len = 16;
  }
  (*alAction)(res, len, pfx);
}

void dumpVA(unsigned long adr, int len, const char* pfx) {
  if (((adr + len - 1) & 0xFFFF0000L) == 0) {
    dumpF((uint8_t*)adr, len, pfx);
  } else {
    dumpFF(adr, len, pfx);
  }
}

void i2cScan() {
  uint8_t count = 0;
  for (uint8_t i = 8; i < 120; i++) {
    Wire.beginTransmission (i);
    if (Wire.endTransmission () == 0) {
      Serial.print(F("Found address (0x"));
      if (i < 16) {
        Serial.write('0');
      }
      Serial.print(i, HEX);
      Serial.print(F(") "));
      Serial.println(i);
      count++;
    }
  }
  if (count) {
    Serial.print(F("Found "));
    Serial.print(count);
    Serial.print(F(" I2C device"));
    if (count != 1) {
      Serial.write('s');
    }
    Serial.println();
  } else {
    Serial.println(F("No I2C devices detected. Scanned adr 8 to 119."));
  }
}

byte accRAM(const byte *adr) {
  return *adr;
}

byte accPGM(const byte *adr) {
  return pgm_read_byte_near(adr);
}

byte accEEPROM(const byte *adr) {
  return eeprom_read_byte(adr);
}

void dumpGen(const byte* adr, int len, accessFunction accs, const char* pfx, bool withAdr, int suppHex) {
  byte idx;
  byte blanks;
  if (len) {
    for (; len > 0; len -= 16, adr += 16) {
      Serial.print(AF(pfx));
      if (withAdr) {
        phAdr(adr);
      }
      for (idx = 0; idx < 16; idx++) {
        if (idx < len ) {
          byte curr = (*accs)(adr + idx);
          if (suppHex < 0 || suppHex != curr) {
            phByte(curr);
          } else {
            Serial.write('.');
            Serial.write('.');
          }
          blanks = 1;
        } else {
          blanks = 3;
        }
        pBlanks(blanks);
      }
      Serial.write('\'');
      for (idx = 0; (idx < 16) && (idx < len); idx++) {
        byte curr = (*accs)(adr + idx);
        Serial.write(curr < 0x20 || (suppHex >= 0 && curr == suppHex) ? '.' : curr);
      }
      Serial.write('\'');
      Serial.println();
    }
  }
}

void phAdr(const byte* adr) {
  phByte(((uint16_t)adr) >> 8);
  phByte(((uint16_t)adr) & 0xFF);
  Serial.write(':');
  Serial.write(' ');
}

void phByte(byte byt) {
  if (byt < 16) {
    Serial.write('0');
  }
  Serial.print(byt, HEX);
}

void phBytesB(byte* mem, byte count) {
  mem += count;
  while (count--) {
    phByte(*--mem);
  }
}

void pBlanks(byte num) {
  while (num--) {
    Serial.write(' ');
  }
}

void dumpR(const byte* adr, int len, const char* pfx) {
  dumpGen(adr, len, accRAM, pfx);
}

void dumpF(const byte* adr, int len, const char* pfx) {
  dumpGen(adr, len, accPGM, pfx);
}

void dumpE(const byte* adr, int len, const char* pfx) {
  dumpGen(adr, len, accEEPROM, pfx);
}

void dumpAsc() {
  byte buff[16];
  byte val = 0;
  for (byte lines = 0; lines < 16; lines++) {
    for (byte idx = 0; idx < 16; idx++) {
      buff[idx] = val++;
    }
    dumpGen(buff, 16, accRAM, PSTR("ASCII: "), false);
  }
}

void dumpFF(unsigned long adr, int len, const char* pfx) {
  byte idx;
  if (len) {
    for (; len > 0; len -= 16, adr += 16) {
      Serial.print(AF(pfx));
      phBytesB((byte*)&adr, 3);
      Serial.write(':');
      Serial.write(' ');
      for (idx = 0; idx < 16; idx++) {
        if (idx < len ) {
          phByte(pgm_read_byte_far(adr + idx));
          Serial.write(' ');
        } else {
          pBlanks(4);
        }
      }
      Serial.write('\'');
      for (idx = 0; (idx < 16) && (idx < len); idx++) {
        byte curr = pgm_read_byte_far(adr + idx);
        Serial.write(curr < 0x20 ? '.' : curr);
      }
      Serial.write('\'');
      Serial.println();
    }
  }
}

Part of the problem is that a far PROGMEM address won't fit into a "char *". Pass the address as an "unsigned long":

void Paint_DrawImage(const unsigned long image, UWORD xStart, UWORD yStart, UWORD W_Image, UWORD H_Image)

Don't forget to use "pgm_read_byte_far()" instead of "pgm_read_byte()"

1 Like

You are in so much trouble :scream:

If you do this on a Arduino Mega: Serial.println(sizeof(size_t));
then it prints "2". That means it uses two bytes for the index and size of an array. That is okay if the array is in SRAM. That means it can not deal with data in Flash that is larger than 64kbyte.

If you have multiple sections in Flash (and each is less than 64kbyte), then all the F() and memcpy_P() functions can no longer be used. You have to check all the libraries that they don't use those functions.

There is a workaround, but it needs code that runs during runtime.
The "pgm_get_far_address()" returns a 32-bit value for the address during runtime. That value is not a pointer for the compiler, but it can be used as a pointer for pgm_read_byte_far() and strncpy_PF().

My own investigation is in Progmem_far.ino .

After writing this, I noticed that I wrote this already in the link of post #2 by Whandall. Well, better do something twice in a good way than half in a bad way.

As far as I know, the Wokwi simulator behaves the same for this situation as in the real world. What kind of display are you using ? Wokwi has a ILI9341 display.

This might be the right time to upgrade to a Raspberry Pi Pico or a ESP32.

1 Like

Hi Koepal

My Lcd is 2.4 inch Lcd.

It has Rpi code but the hardware is toooo big.
They dont have Pi Pico codes where the pi pico is tiny in size which is good.

< edit >
Sorry, still working on the code, what I posted is not working correctly.

Will repost later if I get it working.

Arduino runs on a Raspberry Pi Pico and on a ESP32. It should be mostly in the same way as with your Arduino Mega.

A Wokwi simulation with the display and a ESP32: https://wokwi.com/projects/342032431249883731.
I think that the Adafruit library is often used, but there is also a lcdgfx library: https://wokwi.com/projects/308022099088245312.
I don't know how to use the SPI bus with a Raspberry Pi Pico and if the Adafruit library is compatible with it. If everything is compatible then it should work.

Ok, lets try this again:

The new setup() that initializes the far addresses and used the new Paint_DrawImage_FP:

void setup()
{ 
    Image_Init();
    Config_Init();
    LCD_Init();
    LCD_Clear(0xffff);
    Paint_NewImage(LCD_WIDTH, LCD_HEIGHT, 0, WHITE);
    Paint_Clear(WHITE);
    
    Paint_DrawImage_FP(gImage_3D_batt1percent_farptr, 0, 210, 80, 61); 
    Paint_DrawImage_FP(gImage_3D_batt5percent_farptr, 20, 210, 80, 61); 
    Paint_DrawImage_FP(gImage_3D_batt100percent_farptr, 40, 210, 80, 61);
    Paint_DrawImage_FP(gImage_3D_batt85percent_farptr, 60, 210, 80, 61);
    Paint_DrawImage_FP(gImage_3D_batt75percent_farptr, 80, 210, 80, 61);
    Paint_DrawImage_FP(gImage_3D_batt50percent_farptr, 100, 210, 80, 61);
    Paint_DrawImage_FP(gImage_3D_batt25percent_farptr, 120, 210, 80, 61);
    Paint_DrawImage_FP(gImage_3D_batt10percent_farptr, 140, 210, 80, 61);
}

The Paint_DrawImage_FP() function:

void Paint_DrawImage_FP(const uint_farptr_t image, UWORD xStart, UWORD yStart, UWORD W_Image, UWORD H_Image)
{
  int i, j;
  for (j = 0; j < H_Image; j++) {
    for (i = 0; i < W_Image; i++) {
      if (xStart + i < LCD_WIDTH  &&  yStart + j < LCD_HEIGHT) //Exceeded part does not display
        Paint_SetPixel(xStart + i, yStart + j, (pgm_read_byte_far(image + j * W_Image * 2 + i * 2 + 1)) << 8 | (pgm_read_byte_far(image + j * W_Image * 2 + i * 2)));
      //Using arrays is a property of sequential storage, accessing the original array by algorithm
      //j*W_Image*2          Y offset
      //i*2                  X offset
      //pgm_read_byte()
     
    }
  }
}

image.h with the pointers and (I hope) the correct attribute for far program memory (been a while since I've done this, and I usually define the attribute as PROGMEM_FAR to make it easier to use).

#ifndef _IMAGE_H_
#define _IMAGE_H_
#include <avr/pgmspace.h>

extern uint_farptr_t gImage_3D_batt1percent_farptr;
extern uint_farptr_t gImage_3D_batt5percent_farptr;
extern uint_farptr_t gImage_3D_batt10percent_farptr;
extern uint_farptr_t gImage_3D_batt25percent_farptr;
extern uint_farptr_t gImage_3D_batt50percent_farptr;
extern uint_farptr_t gImage_3D_batt75percent_farptr;
extern uint_farptr_t gImage_3D_batt85percent_farptr;
extern uint_farptr_t gImage_3D_batt100percent_farptr;

extern const unsigned char gImage_3D_batt1percent[] __attribute__((section(".fini7")));
extern const unsigned char gImage_3D_batt5percent[] __attribute__((section(".fini7")));
extern const unsigned char gImage_3D_batt100percent[] __attribute__((section(".fini7")));
extern const unsigned char gImage_3D_batt85percent[] __attribute__((section(".fini7")));
extern const unsigned char gImage_3D_batt75percent[] __attribute__((section(".fini7")));
extern const unsigned char gImage_3D_batt50percent[] __attribute__((section(".fini7")));
extern const unsigned char gImage_3D_batt25percent[] __attribute__((section(".fini7")));
extern const unsigned char gImage_3D_batt10percent[] __attribute__((section(".fini7")));

void Image_Init();

#endif

And lastly the additional code for image.cpp:

uint_farptr_t gImage_3D_batt1percent_farptr;
uint_farptr_t gImage_3D_batt5percent_farptr;
uint_farptr_t gImage_3D_batt10percent_farptr;
uint_farptr_t gImage_3D_batt25percent_farptr;
uint_farptr_t gImage_3D_batt50percent_farptr;
uint_farptr_t gImage_3D_batt75percent_farptr;
uint_farptr_t gImage_3D_batt85percent_farptr;
uint_farptr_t gImage_3D_batt100percent_farptr;

void Image_Init(){
  gImage_3D_batt1percent_farptr = pgm_get_far_address(gImage_3D_batt1percent);
  gImage_3D_batt5percent_farptr = pgm_get_far_address(gImage_3D_batt5percent);
  gImage_3D_batt10percent_farptr = pgm_get_far_address(gImage_3D_batt10percent);
  gImage_3D_batt25percent_farptr = pgm_get_far_address(gImage_3D_batt25percent);
  gImage_3D_batt50percent_farptr = pgm_get_far_address(gImage_3D_batt50percent);
  gImage_3D_batt75percent_farptr = pgm_get_far_address(gImage_3D_batt75percent);
  gImage_3D_batt85percent_farptr = pgm_get_far_address(gImage_3D_batt85percent);
  gImage_3D_batt100percent_farptr = pgm_get_far_address(gImage_3D_batt100percent);
}

Hi David_2018

All are ok except it returned an error saying that array size too large.
But i thought your code is to make it able to handle large arrays?
Kindly advice. Thks

Fyi i added an asterisk * in front of "image," shown below that your code does not have, so i not sure if this is affecting it ? The original code does have an asterisk *image,

void Paint_DrawImage_FP(const uint_farptr_t *image, UWORD Startx, UWORD Starty, UWORD Endx, UWORD Endy);

Hi Johnwasser

i tried your suggestions and the compiler returned error:
""""exit status 1
Compilation error: size of array 'gImage_3D_batt10percent' is too large"""

i tried smaller arrays and it can compile up to 55% but images shown on Lcd are either mostly scrambled or cut off.
i tried to search internet for solution but still can resolve this. But why the codes that are supposed to handle larger array addresses turns out cant handle this same array?
Any idea whats wrong?. Thks

I did not change the name or type of the arrays themselves, and the changes I made to image.h are in addition to the lines you already had declaring the arrays.

The asterisk should not be there, uint_farptr_t is already a reference to a pointer.

Something else you can do to save a considerable amount of memory. It looks like you are using 12-bit color in the code, stored as two bytes per pixel, with only 41 different colors being used. The size of each image would be halved if you used an array to store the actual color codes, and the image file itself stored the index to the color.

Agreed.

Hi David
At setup() need remove _farptr. Then your code seems to compile ok now but its still the same as my original issue when sketch reach 28% it hangs. Any views on this?

Hi Mrburnette
i would like to use pi pico but there is no code for pi pico by supplier. They only give codes for big sized Pi.

PiPico, RP2040, is relatively new to the scene and example code is being created for Javascript, C++ (Arduino), and microPython and CircuitPython: the later being driven by Adafruit.

For Arduino, the "core" was developed independly and the documentation is here:
Arduino-Pico — Arduino-Pico 2.6.5 documentation

See also:
example "arduino" rp2040 display "jpg" - Google Search

Does it look like I've got the Wokwi simulation set up correctly? First time trying to use it.

https://wokwi.com/projects/351607101845406296

I can't really tell that there is any difference between the images, the arrays appear to be identical except for the name and the comment.

Yes, I think so. Should there be something else on the display ?
I checked the pins for the SPI bus and the ones in DEV_Config.h, I think they are right.

In the right-upper corner of the Wokwi page is a percentage that shows how fast your browser can run the simulation. If that is way below 100%, then try the Chrome browser.
When the simulation is not running, click on the display, then click on the question mark, then you go to the "Docs" and there are examples, so you can see what others made.

U using any PROGMEM in image.h? As u mentioned your codes are in addition of mine.

Any _farptr for LCD_mega2560 tab under setup() ? For example, do u use gImage_3D_batt100percent or gImage_3D_batt100percent_farptr ?

NICE Wokwi great setup with codes.
but Wokwi doesn't show the sketch's % flash memory consumption though.

Seems there is something wrong with your codes, as in some images are not displayed even when sketch is 10% where those not displayed were originally displaying well with different combination of pictures set to display at setup(). And when sketch is larger than 30% or 40% then all images will display fizzy images.

Does Wokwi show the total % of bytes after compile similar to Arduino IDE?