ESP32-cam frame buffer manipulation possible?

Projects Programming
1

Hi all,

I am using a AI-Thinker ESP32-cam module with Arduino IDE ver 1.8.2. and the ESP32/Camera/CameraWebServer example. The code is using 4 tabs, the main tab looks like this:

#include "esp_camera.h"
#include <WiFi.h>

//
// WARNING!!! PSRAM IC required for UXGA resolution and high JPEG quality
//            Ensure ESP32 Wrover Module or other board with PSRAM is selected
//            Partial images will be transmitted if image exceeds buffer size
//

// Select camera model
#define CAMERA_MODEL_WROVER_KIT // Has PSRAM
//#define CAMERA_MODEL_ESP_EYE // Has PSRAM
//#define CAMERA_MODEL_M5STACK_PSRAM // Has PSRAM
//#define CAMERA_MODEL_M5STACK_V2_PSRAM // M5Camera version B Has PSRAM
//#define CAMERA_MODEL_M5STACK_WIDE // Has PSRAM
//#define CAMERA_MODEL_M5STACK_ESP32CAM // No PSRAM
//#define CAMERA_MODEL_AI_THINKER // Has PSRAM
//#define CAMERA_MODEL_TTGO_T_JOURNAL // No PSRAM

#include "camera_pins.h"

const char* ssid = "*********";
const char* password = "*********";

void startCameraServer();

void setup() {
  Serial.begin(115200);
  Serial.setDebugOutput(true);
  Serial.println();

  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = Y2_GPIO_NUM;
  config.pin_d1 = Y3_GPIO_NUM;
  config.pin_d2 = Y4_GPIO_NUM;
  config.pin_d3 = Y5_GPIO_NUM;
  config.pin_d4 = Y6_GPIO_NUM;
  config.pin_d5 = Y7_GPIO_NUM;
  config.pin_d6 = Y8_GPIO_NUM;
  config.pin_d7 = Y9_GPIO_NUM;
  config.pin_xclk = XCLK_GPIO_NUM;
  config.pin_pclk = PCLK_GPIO_NUM;
  config.pin_vsync = VSYNC_GPIO_NUM;
  config.pin_href = HREF_GPIO_NUM;
  config.pin_sscb_sda = SIOD_GPIO_NUM;
  config.pin_sscb_scl = SIOC_GPIO_NUM;
  config.pin_pwdn = PWDN_GPIO_NUM;
  config.pin_reset = RESET_GPIO_NUM;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;
  
  // if PSRAM IC present, init with UXGA resolution and higher JPEG quality
  //                      for larger pre-allocated frame buffer.
  if(psramFound()){
    config.frame_size = FRAMESIZE_UXGA;
    config.jpeg_quality = 10;
    config.fb_count = 2;
  } else {
    config.frame_size = FRAMESIZE_SVGA;
    config.jpeg_quality = 12;
    config.fb_count = 1;
  }

#if defined(CAMERA_MODEL_ESP_EYE)
  pinMode(13, INPUT_PULLUP);
  pinMode(14, INPUT_PULLUP);
#endif

  // camera init
  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK) {
    Serial.printf("Camera init failed with error 0x%x", err);
    return;
  }

  sensor_t * s = esp_camera_sensor_get();
  // initial sensors are flipped vertically and colors are a bit saturated
  if (s->id.PID == OV3660_PID) {
    s->set_vflip(s, 1); // flip it back
    s->set_brightness(s, 1); // up the brightness just a bit
    s->set_saturation(s, -2); // lower the saturation
  }
  // drop down frame size for higher initial frame rate
  s->set_framesize(s, FRAMESIZE_QVGA);

#if defined(CAMERA_MODEL_M5STACK_WIDE) || defined(CAMERA_MODEL_M5STACK_ESP32CAM)
  s->set_vflip(s, 1);
  s->set_hmirror(s, 1);
#endif

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");

  startCameraServer();

  Serial.print("Camera Ready! Use 'http://");
  Serial.print(WiFi.localIP());
  Serial.println("' to connect");
}

void loop() {
  // put your main code here, to run repeatedly:
  delay(10000);
}`Preformatted text`

It runs without any issues, so now I am planning for my next step to make some use of this.
I have already experimented a bit and added sleep mode and wake it up via a "HC12" 433 Mhz RF module that also provides one byte of command code to the ESP. Commands can be 1) "Snap a picture w/o turning wifi on, and save it to Littlefs, then go back to sleep" or 2) "Turn on wifi and the web server", so I can use the CameraWebServer user interface. This also works fine without issues.

The second tab ("app_httpd.cpp") contains the command that executes the camera and loads the image data to the "fb" frame buffer: fb = esp_camera_fb_get(); and that is what I am using for the remote controlled "Snap a picture..." command.

I save the picture like this:

void LITTLEFS_save() {

  Serial.println("Saving to LITTLEFS: photo_0.jpg");
  File file = LITTLEFS.open("/photo_0.jpg", "w");  // only latest picture

  if (!file) {
    Serial.println("LITTLEFS.open failed");
    Serial.println("LITTLEFS.format");
    LITTLEFS.format();
    File file = LITTLEFS.open("/photo_0.jpg", "w");
  }

  bytesWritten = file.write(fb->buf, fb->len);
  if (bytesWritten == 0)  {
    Serial.println("file.write failed");
    Serial.println("LITTLEFS.format");
    LITTLEFS.format();
    bytesWritten = file.write(fb->buf, fb->len);
  }

  Serial.print("LITTLEFS bytesWritten: ");
  Serial.println(bytesWritten);

  file.close();
}

I only need help with one missing thing: When I use "command 2" (wifi & user interface), I would like to write the image data from Littlefs back to the "fb" frame buffer before running the UI, because then I would see the last picture taken instead of a blank picture frame.

Any ideas how to read one image from Littlefs back to the "fb" frame buffer ?

Thanks

2

why don't you read it from Littlefs if you know this is the right information?

3

That is exactly what I want to do. But I cannot figure out the code to do just that.

4

I just found the frame buffer ("fb") type structure:

typedef struct {
    uint8_t * buf;              /*!< Pointer to the pixel data */
    size_t len;                 /*!< Length of the buffer in bytes */
    size_t width;               /*!< Width of the buffer in pixels */
    size_t height;              /*!< Height of the buffer in pixels */
    pixformat_t format;         /*!< Format of the pixel data */
    struct timeval timestamp;   /*!< Timestamp since boot of the first DMA buffer of the frame */
} camera_fb_t;

This is beyond my programming skills. So how can I copy one jpg image file back, from Littlefs to fb ?
The other way goes like this:

file.write(fb->buf, fb->len);
5

well you have a file that has been created, so it's just a matter of reading that back (probably with readBytes())

6

The frame buffer was stored as a image. My understand is that the OP wants to put the image back into the frame buffer. Why? I ask because it may be possible to inject the image somewhere else.

The frame buffer is stored in himem which is accessible through the himem allocation API. By using the himem APi the image can be put back into the 'frame' buffer.

I have a WROVER running ML, where I store training models in the himem area. Just think of your allocated memory areas as large arrays of 32 bit memory pointers. Keep in mind that each memory is 32 bits wide. Even if a bool value is stored at the location.

7

The example sketch "CameraWebServer" works fine if I want to access the ESP32 and use the camera for taking stills or streaming video when I am connected to it via a browser. But there is no support in that sketch for showing an image that was taken before the last sleeping event. I have added Littlefs storage to the sketch, so now the last picture taken is always available after a sleep cycle.

If I could refresh the frame buffer from Littlefs right after the sleep cycle, I would be able to see the picture in "CameraWebServer" when I point my browser to it.

Thanks

8

writing back to the buffer is probably complicated, it would probably be easier to modify the code to serve the data from the file system instead of from the frame buffer.

is that the source code you use?

9

Yes, that is the sketch I am using.
That may be a good idea, to replace the FB with Littlefs for the still images.
But the FB would still be needed for the "live" streaming video function.

10

You need to be able to differentiate when you want to see the live stream and when you want to go for the archived stuff

11

Ok.
I have been testing a temporary solution, using a secondary web server running in the same ESP. I took parts of the code from "ESPAsyncWebServer.h", provided from ESP32-CAM Take Photo and Display in Web Server | Random Nerd Tutorials](ESP32-CAM Take Photo and Display in Web Server | Random Nerd Tutorials) .
So when I want to see the archived stuff I point my browser to this page, and when I want to use the video I go to the original page.
But if I could do the Littlefs-to-FB trick, I could clean up the code quite a bit and also end up with just one user interface.

12

you might have very little control over that frame buffer. it's might be allocated when you take the picture to match the bytes needed for the current encoding. seems risky business to go play with a memory region you don't own nor control... probably requires to go read deeper into the libraries and see what's this buffer...

13

Yes, I understand it´s not an easy task. I just want to reach out here if there is a Frame Buffer ninja out there, reading these lines...
Thanks for replying.

14

not me :wink:

15

I realize now that taking a jpg file and write correct data back to the frame buffer would be (almost) impossible.
But what about saving the different variables in camera_fb_t when I have access to them (before the sleep period) to Littlefs and then writing them back to the frame buffer after waking up ?
I think of it like "saving a zip file" with all the variables, then "unzipping" the file and update the buffer.

typedef struct {
    uint8_t * buf;              /*!< Pointer to the pixel data */
    size_t len;                 /*!< Length of the buffer in bytes */
    size_t width;               /*!< Width of the buffer in pixels */
    size_t height;              /*!< Height of the buffer in pixels */
    pixformat_t format;         /*!< Format of the pixel data */
    struct timeval timestamp;   /*!< Timestamp since boot of the first DMA buffer of the frame */
} camera_fb_t;

Would something like that be possible ?

16

Thank you so very much for the valuable insights dear.