Send Image over Serial to Arduino and Display on TFT (ILI9341 Driver)

Projects Interfacing w/ Software on the Computer
1

For a project I want to send an image generated in Matlab to an Arduino which has an external TFT Module (ILI9341-Driver with an 320x240 Display). I'm able to send the bytes over Serial, but the result is not really satisfying seen on the TFT. According to this tutorial: Fun With The Arduino Esplora: A Digital Picture Frame - miguelgrinberg.com

I'm pushing each pixel as a 2-Byte information containing the Colorcode over Serial (@115200 Baud). After Reading one row into a buffer on the arduino I display it. I'm having problems with synchronization and colors. I simply tried to adapt the code and ported it from python to matlab. Probably I made some bad mistackes. Any ideas?

Arduino code:

/*  Demo of draw circle's APP
    drawCircle(int poX, int poY, int r,INT16U color);
    fillCircle(int poX, int poY, int r,INT16U color);
*/

#include <stdint.h>
#include <TFTv2.h>
#include <SPI.h>

int ApertureDiameter = 12;
int delayMS = 400;

int x_size = 240;
int y_size = 320;

int x_centre = x_size/2;
int y_centre = y_size/2;

int NA_radius = 50;

unsigned short *line = 0;


void setup()
{
  Tft.TFTinit();                                      //init TFT library

  Serial.begin(115200);
  
  delay(1000);
  
  Tft.fillCircle(x_centre, y_centre, ApertureDiameter/2, WHITE);
  delay(1000);
  Tft.fillCircle(x_centre, y_centre, ApertureDiameter/2, BLACK);
  
  line = new unsigned short[x_size];


  Serial.println(-1.0);
}

void loop()
{

  int bytes;
    if (Serial.available() > 0) {
        char cmd = Serial.read();
        switch (cmd) {
        case 'I': // image
            for (int row = 0; row < x_size; row++) {
                bytes = x_size * 2; // one line has 240 pixel @ 2 byte per pixel (color)
                char* p = (char*)line;
                while (bytes > 0) { // serial port has a buffer of only 64 bytes, iterate until the entire line has been read
                    int n = Serial.readBytes(p, bytes);
                    if (n == 0) {
                        // timeout
                        return;
                    }
                    p += n;
                    bytes -= n;
                }
                for (int col = 0; col < y_size; col++)
                     Tft.setPixel(col,row,line[col]); 
            }
        }
    }
}

and Matlab-Code:

z = generateSquare(x, x, .4);
z = uint8(z);
z = (cat(3, z, z, z));
%Wait until Arduino is ready for communication
result = 0;
while(result~=-1.0)
    result=fscanf(serial_port,'%d')  
end 
%send image data over serial port
fprintf(serial_port,'I'); %sends string via serial
for j=1:size(z, 2) % cols     
    for i=1:size(z, 1) % rows
        red = z(i, j, 1);
        green = z(i, j, 2);
        blue = z(i, j, 3);
          red = circshift(red,-3,2);
          green = circshift(green,-2,2);
          blue = circshift(blue,-3,2);
          data_1 = dec2bin(bitor(circshift(bitand(green,3), 5, 2), red));
          data_2 = dec2bin(bitor(circshift(blue , 3, 2), circshift(green ,-3, 2)));
          data = sprintf('%c%c', data_1, data_2);
          %dataString = num2str(z(i, j)); %converts to string
          fprintf(serial_port, data); %sends string via serial
      end
      disp(strcat('Row: ', num2str(j) ))
  end

The image is a 240x320 px matrix, three channels with a square in the middle of it (Black/White).

2

You need to post ALL of your Arduino code if you really want help.

3

I updated the code :slight_smile: .. Another question. Is there any way to "sideload" an image (.bmp) to the SD of the TFT-Module and display this one instead? Could be faster :wink: Right now, pushing it via Serial takes ~1min

4

How is this

unsigned short *line = 0;

  line = new unsigned short[x_size];

better than

   unsigned short line[x_size];

Why are x_size and y_size not const? You don't really expect the image size to change while the Arduino is running, do you?

                bytes = x_size * 2; // one line has 240 pixel @ 2 byte per pixel (color)

But your array only has room for x_size values.

5

As a first solution I've ported the code mentioned above from python to matlab. Maybe there is a much better way to do it. But reading a whole row into a buffer and then display it makes sense though.

You're right, the size will never change. So what's the best thing to get it to work?

Matlab sends two bytes per pixel in BMP-Format. Could it be simply like that:

image = imread('test.jpg')
%iterate over cols/rows for each pixel
red = z(i, j, 1);
green = z(i, j, 2);
blue = z(i, j, 3);
        
red = circshift(red,-3,2);
green = circshift(green,-2,2);
blue = circshift(blue,-3,2);
        
data_1 = dec2bin(bitor(circshift(bitand(green,3), 5, 2), red));
data_2 = dec2bin(bitor(circshift(blue , 3, 2), circshift(green ,-3, 2)));
data = sprintf('%c%c', data_1, data_2);

fprintf(serial_port, data);

On Arduino side it has to be synchronized somehow. Read each Pxelcolor from Serial seperatly and display it.

uint8_t p[1]
//Store Pixelvalue in p
Serial.readBytes(p, 1);
Tft.setPixel(col,row,p);

To be honest, I'm a bit confused at the moment! .. Could you guide me in the right direction, what the best step could be to get it to work?

Thank you!:slight_smile:

6

It doesn't make sense to use readBytes() to read one byte. That is what read is for.

I don't understand why matlab is sending two bytes per pixel, when you call Tft.setPixel() with only one byte at a time. What type is the third argument?

7

Yep. Right. readBytes makes no sense in this constellation..

What would you suggest to be the best format sending a pixelvalue from Matlab to Arduino. The LCD-Library suggests in its BMP-Example to display colors like that:

/*********************************************/
// This procedure reads a bitmap and draws it to the screen
// its sped up by reading many pixels worth of data at a time
// instead of just one pixel at a time. increading the buffer takes
// more RAM but makes the drawing a little faster. 20 pixels' worth
// is probably a good place


#define BUFFPIXEL 20

void bmpdraw(File f, int x, int y) 
{
  bmpFile.seek(bmpImageoffset);

  uint32_t time = millis();
  uint16_t p;
  uint8_t g, b;
  int i, j;

  uint8_t sdbuffer[3 * BUFFPIXEL];  // 3 * pixels to buffer
  uint8_t buffidx = 3*BUFFPIXEL;


  for (i=0; i< bmpHeight; i++) 
  {

    for (j=bmpWidth; j>0; j--) 
    {
      // read more pixels
      if (buffidx >= 3*BUFFPIXEL) 
      {
        bmpFile.read(sdbuffer, 3*BUFFPIXEL);
        buffidx = 0;
      }

      // convert pixel from 888 to 565
      b = sdbuffer[buffidx++];     // blue
      g = sdbuffer[buffidx++];     // green
      p = sdbuffer[buffidx++];     // red

      p >>= 3;
      p <<= 6;

      g >>= 2;
      p |= g;
      p <<= 5;

      b >>= 3;
      p |= b;

      // write out the 16 bits of color
      Tft.setPixel(j,i,p);
    }
  }
  Serial.print(millis() - time, DEC);
  Serial.println(" ms");
}

So couldn't it be as simple as replacing the "bmpFile.read" with something like "Serial.readBytes()"? What datatype should it be? Simply sending a char over Serial? I'm always running into problems when it comes to the point of datatypes.. When I try this, the screen is turning grey.

8

That code says that the 3rd argument is an unsigned 16 bit value, consisting of 5 bits for red, 6 bits for green, and 5 bits for blue.

You would need to wait until there are two bytes to read. Read the first one, and store it in an unsigned int. Left shift the value by 8. Read the second byte, and add it to the int. Then, call TFT.setPixel() with the int as the 3rd argument.

9

Thank you very much! That was really helpful! I've changed the Matlab-Code. The 888->565 conversion is carried out on the PC, then the 2 Bytes are transfered to Arduino and displayed. Pixelwise though..
It is really slow. One picture takes roughly 2-3 minutes. Do you think there is any chance to optimize the code somehow? Baudrate is 115200 already, I think the Nano cannot go any faster without loosing data?!

Best
Bene

10

A 320 x 240 image, with two bytes per pixel, is 153,600 bytes. At 115200 baud, that's 11,520 bytes per second. The image should transfer in 13+ seconds. That yours takes roughly 10 times that long suggests that the code is not optimal.

It's also a mystery, as you haven't posted the latest code.

11

Ok. You're right, there are lots of things I could optimized though.. I'm not sure how fast matlab pushes its commands to the serial. The code below is the updated one:

Matlab - Code:

% Matlab + Arduino Serial Port communication
% Sending Zernike Polynomials to Arduino

close all;
clear all;
clc;

%Declaration of Serial Port
delete(instrfind({'Port'},{'COM7'}));
serial_port=serial('COM7');
serial_port.BaudRate=115200;
warning('off','MATLAB:serial:fscanf:unsuccessfulRead');

%Open Serial Port
fopen(serial_port); 

%Generate Image with pixel-number according to Arduino
x = linspace(-1, 1, 240);
[X,Y] = meshgrid(x,x);
[theta,r] = cart2pol(X,Y);
idx = r<=1;
z = nan(size(X));
z(idx) = zernfun(5,1,r(idx),theta(idx));
z = padarray(z, [0, (320-size(z, 1))/2], z(1,1));
z = imadjust(z)*(2^8-1);
 
figure
imagesc(z), shading interp
axis image, colorbar
title('Zernike function Z_5^1(r,\theta)')
draw now 

z = uint8(z);
z = (cat(3, z, z, z));

%Wait until Arduino is ready for communication
result = 0;
while(result~=-1.0)
    result=fscanf(serial_port,'%d')  
end 

%send image data over serial port
fprintf(serial_port,'I'); %sends string via serial

for j=1:size(z, 2) % rows     
    for i=1:size(z, 1) % cols
        red = z(i, j, 1);
        green = z(i, j, 2);
        blue = z(i, j, 3);
             
        % Convert Pixel from Grayvalue to Binary representation
        red_array = fliplr(de2bi(double(red)));
        red_bin = zeros(1,16);
        red_bin(1, (1+size(red_bin,2)-size(red_array,2)):end)=red_array;
        
        green_array = fliplr(de2bi(double(green)));
        green_bin = zeros(1,8);
        green_bin(1, (1+size(green_bin,2)-size(green_array,2)):end)=green_array;       
        
        blue_array = fliplr(de2bi(double(blue)));
        blue_bin = zeros(1,8);
        blue_bin(1, (1+size(blue_bin,2)-size(blue_array,2)):end)=blue_array;       
        
        % p >>= 3;
        red_bin = circshift(red_bin,3,2);
        % p <<= 6;
        red_bin = circshift(red_bin,-6,2); 
        
        % g >>= 2;
        green_bin = circshift(green_bin,2,2); 
        
        % p |= g;
        red_bin = bitor(red_bin, padarray(green_bin, [0, 8], 'pre'));
        % p <<= 5;
        red_bin = circshift(red_bin,-5,2);
        
        % b >>= 3;        
        blue_bin = circshift(blue_bin,3,2);

        % p |= b;
        data = bitor(red_bin, padarray(blue_bin, [0, 8], 'pre'));
        
        upper_byte = data(1:8);
        lower_byte = data(9:end);

        data_matrix(i, j, 1) = bi2de(upper_byte);
        data_matrix(i, j, 2) = bi2de(lower_byte);
        
        data = sprintf('%c%c', bi2de(upper_byte), bi2de(lower_byte));
        fprintf(serial_port, data); %sends string via serial
        
    end
    disp(strcat('Row: ', num2str(j) ))
end

% Close connection and reset
% fclose(serial_port); 
% delete(serial_port);

Arduino Code:

#include <stdint.h>
#include <TFTv2.h>
#include <SPI.h>

const int x_size = 240;
const int y_size = 320;

const int x_centre = x_size / 2;
const int y_centre = y_size / 2;

void setup()
{
  Tft.TFTinit();                                      //init TFT library

  Serial.begin(115200);

  delay(100);

  Tft.fillCircle(x_centre, y_centre, ApertureDiameter / 2, WHITE);
  delay(1000);
  Tft.fillCircle(x_centre, y_centre, ApertureDiameter / 2, BLACK);

  Serial.println(-1.0);
}

void loop()
{
  char data[2];
  uint8_t lower_byte;
  uint16_t upper_byte;

  int bytes;
  if (Serial.available() > 0) {
    char cmd = Serial.read();
    switch (cmd) {
      case 'I': // image
        for (int row = 0; row < y_size; row++) {
          for (int col = 0; col < x_size; col++) {

            int n = Serial.readBytes(data,3);
            if (n == 0) {
              // timeout
              return;
            }

            upper_byte = data[0];
            upper_byte <<= 8;
            lower_byte = data[1];

            upper_byte |= lower_byte;
            
            Tft.setPixel(col, row, upper_byte);
          }
        }
    }
  }
}
12 
            int n = Serial.readBytes(data,3);
            if (n == 0) {
              // timeout
              return;
            }

Why are you trying to read 3 bytes?

            upper_byte = data[0];
            upper_byte <<= 8;
            lower_byte = data[1];

            upper_byte |= lower_byte;

or

upper_byte = (int)data[0] << 8 + data[1];

Of course, upper_byte is a silly name for a multi-byte variable.

Try this, instead:

    switch (cmd) {
      case 'I': // image
        for (int row = 0; row < y_size; row++)
        {
          for (int col = 0; col < x_size; col++) 
          {
            while(Serial.available() < 2) {} // Do nothing until there are two bytes to read
            byte a = Serial.read();
            byte b = Serial.read();
            int color = (int)a << 8 + b;

            Tft.setPixel(col, row, color);
13

Thank you very much for your reply again! I've tried to implement your changes, but some weird effects happened. The pattern was moved, I guess half a line. I moved back to the old code again.
What I did now, was changing the Baud to 250000 and incredibly the small Nano catches the commands without errors. Another thing was to decrease the size of the ROI. I just need a fraction of the scren. Now the update rate is about 30 sec for a 80x80 area.

I think the bottle-neck is Matlab. The calculation might take some time and the command-sending process is not that fast I think. Anyway. With your help I was able to get it to run! Thank you very much! :slight_smile: