Go Down

Topic: what's needed to make the camera module to work with FPGA? (Read 2280 times) previous topic - next topic

eljuligallego

Hi,

I'm trying to make my own image processing project with a simplest design that takes the MIPI data and convert it to RGB, to send it to HDMI with no NIOS and peripherals to have the most space possible for my logic.

I'm using the IPs arduino provide andused the MKRVIDOR4000_graphics project to understand what to do and build the project like you see in RTL attached.

Just take the MIPIcsi2 block to convert from MIPI serial to RGB, then use the FBST block to generate the Sync signals and finally use the DVIout block to stream it to the HDMI port.

I use the USBblaster emulator from Arduino to program and debug in signaltap but for now I just was able to generate a black HDMI signal than the screen recognize correctly but no video data since there is no video data in MIPI interface. at least I can't adquire data in signaltap and if I use the MIPI clk as driver the signal tap does not adquire (waiting for clock so maybe not clock signal is present).

This could be due there no I2C configuration in my design?.

I tried to workaround this issue preloading the VidorEnableCamera sketch to enable the camera and after that tried to load my design but when loading the Usb_Blaster.ino to program my gateware the FPGA seems to reconfigure since the camera stops working and the arduino logo appears. then the sensor is not responding again if I load my design like in previous test.

I tried to capture signaltap data on power up but nothing no data valid from MIPI neither Start bit rising and don't know why.

when load the Usb_Blaster.ino, Is loading a new design to the FPGA or re-configuring something inside?.

I you want to try I just use the MKRVIDOR4000_template project and modify the code of Top design:

Code: [Select]
/*
* Copyright 2018 ARDUINO SA (http://www.arduino.cc/)
* This file is part of Vidor IP.
* Copyright (c) 2018
* Authors: Dario Pennisi
*
* This software is released under:
* The GNU General Public License, which covers the main part of
* Vidor IP
* The terms of this license can be found at:
* https://www.gnu.org/licenses/gpl-3.0.en.html
*
* You can be released from the requirements of the above licenses by purchasing
* a commercial license. Buying such a license is mandatory if you want to modify or
* otherwise use the software for commercial activities involving the Arduino
* software without disclosing the source code of your own applications. To purchase
* a commercial license, send an email to license@arduino.cc.
*
*/

module MKRVIDOR4000_top
(
  // system signals
  input         iCLK,
  input         iRESETn,
  input         iSAM_INT,
  output        oSAM_INT,
 
  // SDRAM
  output        oSDRAM_CLK,
  output [11:0] oSDRAM_ADDR,
  output [1:0]  oSDRAM_BA,
  output        oSDRAM_CASn,
  output        oSDRAM_CKE,
  output        oSDRAM_CSn,
  inout  [15:0] bSDRAM_DQ,
  output [1:0]  oSDRAM_DQM,
  output        oSDRAM_RASn,
  output        oSDRAM_WEn,

  // SAM D21 PINS
  inout         bMKR_AREF,
  inout  [6:0]  bMKR_A,
  inout  [14:0] bMKR_D,
 
  // Mini PCIe
  inout         bPEX_RST,
  inout         bPEX_PIN6,
  inout         bPEX_PIN8,
  inout         bPEX_PIN10,
  input         iPEX_PIN11,
  inout         bPEX_PIN12,
  input         iPEX_PIN13,
  inout         bPEX_PIN14,
  inout         bPEX_PIN16,
  inout         bPEX_PIN20,
  input         iPEX_PIN23,
  input         iPEX_PIN25,
  inout         bPEX_PIN28,
  inout         bPEX_PIN30,
  input         iPEX_PIN31,
  inout         bPEX_PIN32,
  input         iPEX_PIN33,
  inout         bPEX_PIN42,
  inout         bPEX_PIN44,
  inout         bPEX_PIN45,
  inout         bPEX_PIN46,
  inout         bPEX_PIN47,
  inout         bPEX_PIN48,
  inout         bPEX_PIN49,
  inout         bPEX_PIN51,

  // NINA interface
  inout         bWM_PIO1,
  inout         bWM_PIO2,
  inout         bWM_PIO3,
  inout         bWM_PIO4,
  inout         bWM_PIO5,
  inout         bWM_PIO7,
  inout         bWM_PIO8,
  inout         bWM_PIO18,
  inout         bWM_PIO20,
  inout         bWM_PIO21,
  inout         bWM_PIO27,
  inout         bWM_PIO28,
  inout         bWM_PIO29,
  inout         bWM_PIO31,
  input         iWM_PIO32,
  inout         bWM_PIO34,
  inout         bWM_PIO35,
  inout         bWM_PIO36,
  input         iWM_TX,
  inout         oWM_RX,
  inout         oWM_RESET,

  // HDMI output
  output [2:0]  oHDMI_TX,
  output        oHDMI_CLK,

  inout         bHDMI_SDA,
  inout         bHDMI_SCL,
 
  input         iHDMI_HPD,
 
  // MIPI input
  input  [1:0]  iMIPI_D,
  input         iMIPI_CLK,
  inout         bMIPI_SDA,
  inout         bMIPI_SCL,
  inout  [1:0]  bMIPI_GP,

  // Q-SPI Flash interface
  output        oFLASH_SCK,
  output        oFLASH_CS,
  inout         oFLASH_MOSI,
  inout         iFLASH_MISO,
  inout         oFLASH_HOLD,
  inout         oFLASH_WP

);

// signal declaration

wire        wOSC_CLK;

wire        wCLK8,wCLK24, wCLK64, wCLK120;

wire [31:0] wJTAG_ADDRESS, wJTAG_READ_DATA, wJTAG_WRITE_DATA, wDPRAM_READ_DATA;
wire        wJTAG_READ, wJTAG_WRITE, wJTAG_WAIT_REQUEST, wJTAG_READ_DATAVALID;
wire [4:0]  wJTAG_BURST_COUNT;
wire        wDPRAM_CS;

wire [7:0]  wDVI_RED,wDVI_GRN,wDVI_BLU;
wire        wDVI_HS, wDVI_VS, wDVI_DE;

wire        wVID_CLK, wVID_CLKx5;
wire        wMEM_CLK;

wire [24:0] oMIPI_DATAFF;

// MIPI input
assign bMIPI_GP[0]=1'b1;
assign bMIPI_GP[1]=1'bz;

assign wVID_CLK   = wCLK24;
assign wVID_CLKx5 = wCLK120;
assign wCLK8      = iCLK;

// internal oscillator
cyclone10lp_oscillator   osc
  (
  .clkout(wOSC_CLK),
  .oscena(1'b1));

// system PLL
SYSTEM_PLL PLL_inst(
  .areset(1'b0),
  .inclk0(wCLK8),
  .c0(wCLK24),
  .c1(wCLK120),
  .c2(wMEM_CLK),
   .c3(oSDRAM_CLK),
  .c4(wFLASH_CLK),
   
  .locked());


MIPICSI2 u0 (
.iMIPI_CLK       (iMIPI_CLK),       //    mipi.clk
.iMIPI_D         (iMIPI_D),         //        .data
.oMIPI_DATA      (oMIPI_DATA),      // mipi_st.data
.oMIPI_START     (oMIPI_START),     //        .start
.oMIPI_DATAVALID (oMIPI_DATAVALID)  //        .dv
);

FIFO u1(
.wrclk (iMIPI_CLK),
.rdclk (wVID_CLK),
.data ({oMIPI_START,oMIPI_DATA}),
.rdreq (oMIPI_DATARDYFF),
.wrreq (oMIPI_DATAVALID),
.rdempty (~oMIPI_DATAVLDFF),
.q(oMIPI_DATAFF)
);

FBST #(
.pHRES   (640),//   pHRES=1280 640
.pVRES   (480),//   pVRES=720    480
.pHTOTAL (762),// pHTOTAL=1354   762
.pVTOTAL (525),// pVTOTAL=910    525
.pHSS    (656),//    pHSS=1300   656
.pHSE    (752),//    pHSE=1340   752
.pVSS    (490),//    pVSS=778    490
.pVSE    (492) //    pVSE=782   492
)FBST_u0(
.oBLU          (wDVI_BLU), //   vport.blu
.oDE           (wDVI_DE),  //        .de
.oGRN          (wDVI_GRN), //        .grn
.oHS           (wDVI_HS),  //        .hs
.oVS           (wDVI_VS),  //        .vs
.oRED          (wDVI_RED), //        .red
.iCLK          (wVID_CLK),      // vid_clk.clk
.iFB_START     (oMIPI_DATAFF[24]),  //  stream.start
.iFB_DATA      (oMIPI_DATAFF[23:0]),   //        .data
.iFB_DATAVALID (oMIPI_DATAVLDFF),     //        .dv
.oFB_READY     (oMIPI_DATARDYFF)   //        .ready
);

DVI_OUT
(
  .iPCLK(wVID_CLK),
  .iSCLK(wVID_CLKx5),

  .iRED(wDVI_RED),
  .iGRN(wDVI_GRN),
  .iBLU(wDVI_BLU),
  .iHS (wDVI_HS),
  .iVS (wDVI_VS),
  .iDE (wDVI_DE),

  .oDVI_DATA(oHDMI_TX),
  .oDVI_CLK(oHDMI_CLK)
  );

reg [5:0] rRESETCNT;

always @(posedge wMEM_CLK)
begin
  if (!rRESETCNT[5])
  begin
  rRESETCNT<=rRESETCNT+1;
  end
end

endmodule


Also attach some signaltaps capture to see what I'm talking about.

Thanks in advance,
regards.


purisame

Hi,

Preloading the sketch will not work because the camera may lose power when you upload a new sketch. You need to have I2C on the FPGA somehow.


Here's generally how it's supposed to work:
  • Set camera resolution and streaming frequency over i2c
  • Tell camera over i2c to start streaming video
  • Receive stream via MIPI D-PHY
  • Put the stream in the SDRAM
  • Retrieve the stream from the SDRAM
  • Output retrieved stream to HDMI


There is not enough RAM in the FPGA to directly connect the camera to HDMI, which is why it is put it in the SDRAM.


I have implemented all the necessary components including I2C from scratch:



I've tested all the components individually (except the arbiter) and they work.


I'm combining them and running into clock domain crossing issues with Intel's DC FIFO, which hopefully I can resolve this week or next week.

eljuligallego

#2
May 01, 2020, 06:34 pm Last Edit: May 01, 2020, 07:54 pm by eljuligallego
thanks a lot for the reply, I have a very important help with your code to start with.

I didn't understand why the sdram was needed but my knowledge about the MIPI/CSI interface is limited.

I just discover the links to the specifications, I guess the info is there.

I was hoping the pixel throughput was simple and just had to synchronize the sync signals to have
a valid hdmi signal but seems not the case.

Also I was looking a way to passthrough the I2C interface between the camera and the samd21 since would be easier to configure the camera with the Arduino but seems hard to implement in the FPGA an I2C passthrough interface due meta stability in I2C signal.

I'll try to make it works and see what can I do for helping you but not sure if I can but, I'll try!.

Thanks again!.

purisame

I was hoping the pixel throughput was simple and just had to synchronize the sync signals to have
a valid hdmi signal but seems not the case.
That's what I thought too, I tried it and it just displayed garbled noise.

Also I was looking a way to passthrough the I2C interface between the camera and the samd21 since would be easier to configure the camera with the Arduino but seems hard to implement in the FPGA an I2C passthrough interface due meta stability in I2C signal.
That is not easy either, which is why the official repos have the "mailbox" RPC protocol for telling the FPGA to do things.

i.e. control NINA SPI, I2C with MIPI, HDM output control, etc.

eljuligallego

Hi @purisame,

any updates on code?.

Now I'm able to acquire data thanks to your I2C and a OV5647 sensor I have but the sdram controller is not working properly....

I have to debug it to understand what's happening but when I figure out I'll make let you know..

thanks for your help.

purisame

Hi @purisame,

any updates on code?.
I tried a bunch of ways to sync the clocks with DCFIFO but I still don't know what's going on. I am 95% sure the MIPI stuff is right though. I did this dummy test where the current sensor values are output to a VGA character console and I could tell that it was getting brightness values by keeping the camera steady and flashing a light on and off over it. It goes wrong when I try to move the data with a FIFO (Intel's IP for now).

Haven't had the time to debug it much yet, maybe this weekend.

Now I'm able to acquire data thanks to your I2C and a OV5647 sensor I have but the sdram controller is not working properly....

I have to debug it to understand what's happening but when I figure out I'll make let you know..

thanks for your help.

Are you transferring data to the SDRAM via a different clock-domain? I know it works in the same clock domain, but if you do clock domain crossing you should have a sync chain and a FIFO.

By the way, if you see hold violations but no setup violations in the timing analysis, try disabling the fast output registers setting for the SDRAM pins.

eljuligallego

Hi,

after some debug I realize my problem also is in the DCFIFO...

I'm curious about the MIPI clock signal and how is it's configuration. In the datasheet:

https://cdn.sparkfun.com/datasheets/Dev/RaspberryPi/ov5647_full.pdf

you can see there are lot of registers for configuring the pll but still no idea about how is the MIPI clk configuration now, I'll have read carefully all the registers values and understand what's doing.My idea was to play with this registers to find the right configuration that works for us with our FPGA but not sure if this will work.

in other code I found the configuration differs a little bit:

https://github.com/torvalds/linux/blob/master/drivers/media/i2c/ov5647.c

Also looking at the MKRVIDOR4000_graphics project to have some inspiration in how works the mipi interface you can see the mipi clock domain change is done with a dual clock Ram inside the QRCode finder block. the curious thing is lot of logic is done inside the FPGA before put the data inside the RAM block so in our case should be better to implement.

what do you think?.

 

purisame

Just thought I should share an update. It has taken me some time, but I got things to work yesterday:

https://github.com/hdl-util/mipi-demo




eljuligallego

Now it's working!. I get the same image as you. Thanks a lot.


looking forward for the debayer/demosaic to have RGB.


P.D. I have seen you finally implements the FIFO by your own instead of the quartus IP core.

Go Up