Nicla Vision Manage Images On Arduino IDE

Hi everyone,

I recently bought a Nicla Vision for a university project, I was wondering if you are obligated to use the openMV to use the camera, or the Arduino IDE can handle the camera as well(didn't find any libraries).

The end goal is to send images to a web server when movement is detected or when an object is getting near the ToF sensor, and I believe that the arduino IDE is better than the OpenMV.

Best Regards, Ced

is the tool used to create code, compile and download to a microcontroller. The IDE is not involved in any data handling.
Arduinos are not suitable in handling pictures due to the insufficient memory.

1 Like

Thank you, I'll see with the OpenMV what can be done.

It depends on what you mean by "handling" and "pictures".

Machine vision doesn't necessarily require a lot of pixels or colors, so it can be possible to achieve your goal without a huge amount of image data. In fact, you might find that part of the work is downsampling the raw image enough so that a machine learning algorithm is able to process a video stream at a sufficient rate for the application.

I would say that this meets anyone's definition of "picture":

And that is using a Nano 33 BLE. Even more powerful Arduino boards are now available and Arduino has been putting a focus on machine vision in the most recent products. That includes the "Arduino Nicla Vision" @cedricave98 is using:

and the "Arduino Portenta Vision Shield" series:

Something like this?:

Arduino Access Point Server => PC Matlab Decoder and image show

Arduino INO:

#include <camera.h>

#include <WiFi.h>
#include <WiFiClient.h>
#include <WiFiSSLClient.h>
#include <WiFiServer.h>
#include <WiFiUdp.h>



// On Nano RP2040 Connect, RGB leds are connected to the wifi module
// The user APIs are the same, but we can't convert to int, so use defines
#if defined(ARDUINO_NANO_RP2040_CONNECT)

#include "WiFiNINA.h"
#define led1  LEDR
#define led2  LEDG
#define led3  LEDB

// On Nicla Sense ME, RGB leds are connected via an I2C module
// The user APIs are the same, but we can't convert to int, so use defines
#elif defined(ARDUINO_NICLA)

#include "Nicla_System.h"
#define led1  LEDR
#define led2  LEDG
#define led3  LEDB

#else

int led1 = LEDR;
int led2 = LEDG;
int led3 = LEDB;

#endif


//Camera Picture Values
FrameBuffer fb;
unsigned long lastUpdate = 0;
unsigned long StartTime = 0;
unsigned long StartTimeMeas = 0;
unsigned long PicTimer = 0;
unsigned long DeltaTime = 0;
bool PictureStatus_allowtakePicture = true;
bool PictureStatus_allowsendPicture = false;
bool toggleLEDR;


//Camera input
#ifdef ARDUINO_NICLA_VISION
#include "gc2145.h"
GC2145 galaxyCore;
Camera cam(galaxyCore);
#define IMAGE_MODE CAMERA_GRAYSCALE    //normal CAMERA_RGB565
#else
#include "himax.h"
HM01B0 himax;
Camera cam(himax);
#define IMAGE_MODE CAMERA_GRAYSCALE
#endif



//Server
///////please enter your sensitive data in the Secret tab/arduino_secrets.h
char ssid[] = "NiclaVision";    // your network SSID (name)
char pass[] = "123Qwerty";    // your network password (use for WPA, or use as key for WEP)
int keyIndex = 0;             // your network key Index number (needed only for WEP)

int status = WL_IDLE_STATUS;

WiFiServer server(80);

bool GET_Req_StBit_detected = false;
bool GET_Req_Picture_detected = false;
bool GET_Req_INFOS_detected = false;


void setup() {
  Serial.begin(1000000); //921600
  Serial.println("Access Point Web Server");

  // by default the local IP address of will be 192.168.3.1
  // you can override it with the following:
  // WiFi.config(IPAddress(10, 0, 0, 1));

  // The AP needs the password be at least 8 characters long
  if (strlen(pass) < 8) {
    Serial.println("Creating access point failed");
    Serial.println("The WiFi password must be at least 8 characters long");
    // don't continue
    while (true);
  }

  // print the network name (SSID);
  Serial.print("Creating access point named: ");
  Serial.println(ssid);

  //Create the Access point
  status = WiFi.beginAP(ssid, pass);
  if (status != WL_AP_LISTENING) {
    Serial.println("Creating access point failed");
    // don't continue
    while (true);
  }

  // wait 10 seconds for connection:
  Serial.println("Waiting 10s for Connection");
  delay(1000);

  // start the web server on port 80
  server.begin();

  // you're connected now, so print out the status
  printWiFiStatus();

  if (!cam.begin(CAMERA_R320x240, CAMERA_GRAYSCALE, 30)) {  //QVGA Resolution: CAMERA_R320x240  or R640x480| IMAGE_MODE CAMERA_RGB565  | framerate  by default begin(int32_t resolution=CAMERA_R320x240, int32_t pixformat=CAMERA_GRAYSCALE, int32_t framerate=30);
    Serial.println("Camera no Start");
  }      //CAMERA_R640x480
  else   Serial.println("Camera Start");


  //Take first Pichture
  // Time out after 40 and send new data
  //-------------------------------------------------------------------------------
  // Grab 1st frame and write to fb

  cam.grabFrame(fb, 40);                           //FrameBuffer &fb, uint32_t timeout

  // Setup the 3 pins as OUTPUT undschalte LEDs aus
  pinMode(LEDR, OUTPUT);
  pinMode(LEDG, OUTPUT);
  pinMode(LEDB, OUTPUT);
  digitalWrite(LEDR, HIGH);
  digitalWrite(LEDG, HIGH);
  digitalWrite(LEDB, HIGH);
}

void loop() {

  PictureStatus_allowsendPicture = false;
  PictureStatus_allowtakePicture = true;


  //Webside
  WiFiClient client = server.available();   // listen for incoming clients

  if (client) {                             // if you get a client,
    Serial.println("new client");           // print a message out the serial port
    String currentLine = "";                // make a String to hold incoming data from the client
    StartTime = millis();

    while (client.connected()) {            // loop while the client's connected

      if (millis() - StartTime >= 3000) {
        Serial.print("<<< Timeout >>>>");
        Serial.println(millis() - StartTime);
        break;
      }

      if (PictureStatus_allowtakePicture) {
        TakePicture_Function();
      }

      //Serial.println("Client Connected");
      if (client.available())              // if there's bytes to read from the client, &&PictureStatus_allowsendPicture
      {

        StartTime = millis();

        char c = client.read();             // read a byte  read bytes from the incoming client
        //Serial.print(c);
        if (c == '\n')                     // if the byte is a newline character\n //if HTTP request has ended
        { // if the current line is blank, you got two newline characters in a row.

          if (currentLine.length() == 0)
          { // Ende des GEt REquests erkannt
            // Answer depending on Status
            if (PictureStatus_allowsendPicture) {
              digitalWrite(LEDR, LOW);
              Serial.println("Send Picture");
              Respond_Picture(client);
              Serial.println("<<<<<Picture sended >>>>>>");
              PictureStatus_allowsendPicture = false;
              PictureStatus_allowtakePicture = true;
              digitalWrite(LEDR, HIGH);
            }
            else {
              digitalWrite(LEDG, LOW);
              Serial.println("Send Status");
              Respond_StBit(client);
              Serial.println("<<<<<Status Answer sended >>>>>>");
              digitalWrite(LEDG, HIGH);
            }
          }

          else {      // if you got a newline, then clear currentLine:
            currentLine = "";
          }
        }
        else if (c != '\r') {    // if you got anything else but a carriage return character,
          currentLine += c;      // add it to the end of the currentLine
        }
      }

    } //ende While
    // close the connection:
    delay(20);
    client.flush();
    client.stop();
    digitalWrite(LEDG, HIGH); // Client not avaliable
    digitalWrite(LEDR, HIGH);
    Serial.println("client disconnected");

  }
  digitalWrite(LEDG, HIGH);
  digitalWrite(LEDR, HIGH);
}

void printWiFiStatus() {
  // print the SSID of the network you're attached to:
  Serial.print("SSID: ");
  Serial.println(WiFi.SSID());

  // print your WiFi shield's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);

  // print where to go in a browser:
  Serial.print("To see this page in action, open a browser to http://");
  Serial.println(ip);
}

void TakePicture_Function() {
  //IMU and Pictures

  //HERE Picture
  Serial.println("InI Snapshot");
  if (PictureStatus_allowtakePicture)    //Picture is only allowed when get Picture is made
  { if (cam.grabFrame(fb, 60) == 0)          //FrameBuffer &fb, uint32_t timeout    ==0 wenn neuer FRame da ist => Framerate für VGA 200
    { //Serial.println("Camera Photoshot");
      PictureStatus_allowsendPicture = true;
      PictureStatus_allowtakePicture = false;
      Serial.println("<<<<< Snapshot >>>>>>");
    }
  }
}


void Respond_StBit( WiFiClient client) {


  client.println("HTTP/1.1 200 OK");
  client.println("Content-type:multipart/byteranges"); //multipart/byteranges
  client.println("Content-Length: 1");
  client.println("Connection: close");
  client.println();
  client.println("<html>");
  client.write(PictureStatus_allowsendPicture);
  client.println("</html>");
  //client.println("\r");
  client.println();

}

void Respond_Picture( WiFiClient client) {
  client.println("HTTP/1.1 200 OK");
  //https://mothereff.in/byte-counter
  client.println("Content-type:multipart/byteranges"); //multipart/byteranges
  client.println("Content-Length: 76800"); //multipart/byteranges bei QVGA RGB 153600 bei VGA  614400bytes + Headereoffset after OK 153673 ||| Grayscale QVGA 76800 = over all 76814 für VGA 307200
  client.println("Connection: close");
  client.println();
  client.println("<html>"); //<body>
  client.write(fb.getBuffer(), cam.frameSize()); //cam.frameSize()
  client.println("</html>"); //</body>
  client.println();
}

Matlab decoder:



clc
clear

import matlab.net.*
import matlab.net.http.*

i=true;

options_GET = weboptions('UserAgent','','Timeout',Inf); 
URI=matlab.net.URI('http://192.XXX.'); % see serial Monitor

while 1
     tic
    while i  
        answer=webread(URI, options_GET);

        if length(answer) >= 76700     
            i=false;
        end
      
     
    end

web_PIC=answer;
web_vector_withheader = web_PIC.';
web_vector = web_vector_withheader(1:end); %cut header away //27 -14
web_vector = circshift(web_vector, -14);  % Versatz Bild hoch und Runter vertikal aus <html><body> 
grey_img=reshape(web_vector,[320,240] ); %[320,240] [640,480]

imshow(flipdim(grey_img,2)); % title('grey IMG');

i=true; 
toc
    
end

Make shure you installed the Wifi Firmware Updater before, below examples/ Stm32..._System/Wifi FrimwareUpdater and installed it completely. Control Flash Status on serial Monitor (2x 100%)

Do you have a better way to format the above code. It looks really interesting.

Here the more readable version.

Bottleneck is the closing of Client connection between Matlab and Arduino and the use of cortex M7 only. Best would be a constant data stream without connecting and disconnecting. Im not able to use both cores M7 and M4 as in the examples shown (compared with code of Portenta H7 where you can select the core). Aim is to use one core for Sending via HTTP and one to take and process snapshots.

By the way @nan2 the Portenta with Vision shield can now run the camera code on the M4 core, or at least the next version after version 3.11 will be able to. If you want the updated camera.cpp file it is here. If it works thanks me for the PR, if it doesn't work blame Arduino for accepting my PR. :grinning:

As far as I know WiFi still only works on the PortentaH7 M7 core, but hopefully that will change, especially if we keep asking for it. I haven't tested it for a while. Reminder to like issues to increse the chance of it being worked on.

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.