Troubles getting the MIPI-demo to work

I have been trying to get the mipi-demo to work on my Arduino MKR VIDOR 4000.

I am trying to follow the lead on this discussion forum regarding the same. I have successfully added all the required files to my Quartus project and successfully compiled and uploaded the Bitstream to the VIDOR.

However, the program does not seem to be working as expected. It look like the either the camera is switching off immediately or the HDMI signal is not working after a few second. I did make sure that my camera and HDMI hardware are functioning properly by running an inbuild camera module with Arduino IDE.

I see that once the code is written to the flash of the FPGA, the camera briefly turns on and then shuts off. I deduce this because my HDMI screen shows a light color for a second before it says there is no HDMI signal.

I do not know what I am missing here. Kindly help me with the same.

This link will have the Quartus setup and the files that I used.

Looking over your schematic as you have wired it, I cannot see any problems. It may have disappeared from this post along with the links to the technical information on your system.

void setup()
{
    none;
}
void loop ()
{
    Serial.print("You are missing: ");
    Serial.print("a drawing, ");
    Serial.print("your components, ");
    Serial.print("the code running on your micro controller, ");
    Serial.print("the results you expect, and ");
    Serial.print("the results you are experiencing.");
    IF (your_project_is_set_up_as_the_links_you_shared)
        your_project_would_work = TRUE;
    THEREFORE
        your_project_is_not_set_up_as_the_links_you_shared = 1;
}

Sorry for the incomplete information.

From the forum discussion I send earlier, this was the rough design the form is going for

As I followed with the discussion and compiled what was in mipi-demo repository, this was the final design.

As for the Microcontroller, I am using a basic Arduino IDE EmptySketch, with which I'll add an app.h file which will contain the bitstream. To the best of my knowledge, this empty sketch will command the microcontroller to instruct the FPGA to burn itself with the bitstream that will be written to its flash.

My desired result is a continuous video begin streamed from the MIPI camera through the HDMI and on the screen.

However, that is not what I am getting. As I said, I am having a brief HDMI signal along with the camera turning on briefly after which nothing seems to work.

I am very new to this and every step is a big learning. Thank you very much for your patience!

I was looking deeper into the added files themselves, hoping to find something that I might have to change, and I found something called module address.

module ov5647 #(
    parameter int INPUT_CLK_RATE,
    parameter int TARGET_SCL_RATE = 400000,
    // Some ov5647 modules have a different address, change this if yours does
    parameter bit [7:0] ADDRESS = 8'h6c

Where can I find this address if it is specific to me? Is it camera related or board-related address?

Thank you for your time and patience.

Hello,

from what i see, you used the excellent work of Sameer Puri.

(EDIT) My mistake : "He succeeded to display on HDMI camera output".. Could you perhaps share your code to see what could be the problem ?

In any case, you have to go through external memory because the synchronization of MIPI and HDMI frequencies seems complicated to me.
As for the I2C address in 0x6C (specific to the camera board), if the example of the ARDUINO library works, it is good.
For my part, I am not yet able to publish the VHDL code of my kernel, but you can try if you wish the VIDOR_SE library which is still in ALPHA version.

Hey,

Thank you so much for looking into it. Yes, I can absolutely share my code. This is the top file that I am using. I have not changed much from the source at all.

module mkrvidor4000_top
(
    // system signals
    input CLK_48MHZ,
    input RESETn,
    input SAM_INT_IN,
    output SAM_INT_OUT,

    // SDRAM
    output SDRAM_CLK,
    output [11:0] SDRAM_ADDR,
    output [1:0] SDRAM_BA,
    output SDRAM_CASn,
    output SDRAM_CKE,
    output SDRAM_CSn,
    inout [15:0] SDRAM_DQ,
    output [1:0] SDRAM_DQM,
    output SDRAM_RASn,
    output SDRAM_WEn,

    // SAM D21 PINS
    inout MKR_AREF,
    inout [6:0] MKR_A,
    inout [14:0] MKR_D,

    // HDMI output
    output [2:0] HDMI_TX,
    output HDMI_CLK,
    inout HDMI_SDA,
    inout HDMI_SCL,

    input HDMI_HPD,

    // MIPI input
    input [1:0] MIPI_D,
    input MIPI_CLK,
    inout MIPI_SDA,
    inout MIPI_SCL,
    inout [1:0] MIPI_GP

);

// internal oscillator
wire OSC_CLK;
cyclone10lp_oscillator osc ( 
    .clkout(OSC_CLK),
    .oscena(1'b1)
);

mem_pll mem_pll(.inclk0(CLK_48MHZ), .c0(SDRAM_CLK));

wire clk_pixel_x5;
wire clk_pixel;
wire clk_audio;
hdmi_pll hdmi_pll(.inclk0(CLK_48MHZ), .c0(clk_pixel), .c1(clk_pixel_x5), .c2(clk_audio));

localparam AUDIO_BIT_WIDTH = 16;
localparam AUDIO_RATE = 48000;
localparam WAVE_RATE = 240;

logic [AUDIO_BIT_WIDTH-1:0] audio_sample_word;
sawtooth #(.BIT_WIDTH(AUDIO_BIT_WIDTH), .SAMPLE_RATE(AUDIO_RATE), .WAVE_RATE(WAVE_RATE)) sawtooth (.clk_audio(clk_audio), .level(audio_sample_word));

logic [23:0] rgb;
logic [9:0] cx, cy, screen_width, screen_height;
hdmi #(.VIDEO_ID_CODE(1), .AUDIO_RATE(AUDIO_RATE), .AUDIO_BIT_WIDTH(AUDIO_BIT_WIDTH)) hdmi(
    .clk_pixel_x5(clk_pixel_x5),
    .clk_pixel(clk_pixel),
    .clk_audio(clk_audio),
    .rgb(rgb),
    .audio_sample_word('{audio_sample_word >> 9, audio_sample_word >> 9}),
    .tmds(HDMI_TX),
    .tmds_clock(HDMI_CLK),
    .cx(cx),
    .cy(cy),
    .screen_width(screen_width),
    .screen_height(screen_height)
);

logic [1:0] mode = 2'd0;
logic [1:0] resolution = 2'd3; // 640x480 @ 30FPS
logic format = 1'd0; // RAW8
logic ready;
logic model_err;
logic nack_err;

ov5647 #(.INPUT_CLK_RATE(48_000_000), .TARGET_SCL_RATE(100_000)) ov5647 (
    .clk_in(CLK_48MHZ),
    .scl(MIPI_SCL),
    .sda(MIPI_SDA),
    .mode(mode),
    .resolution(resolution),
    .format(format),
    .ready(ready),
    .power_enable(MIPI_GP[0]),
    .model_err(model_err),
    .nack_err(nack_err)
);

logic [7:0] image_data [3:0];
logic [5:0] image_data_type;
logic image_data_enable;
logic [15:0] word_count;
logic frame_start, line_start, interrupt, frame_end;


camera #(.NUM_LANES(2)) camera (
    .clock_p(MIPI_CLK),
    .data_p(MIPI_D),
    .image_data(image_data),
    .image_data_type(image_data_type),
    .image_data_enable(image_data_enable),
    .word_count(word_count),
    .frame_start(frame_start),
    .frame_end(frame_end),
    .line_start(line_start),
    .interrupt(interrupt)
);


 logic [7:0] raw [3:0];
 logic raw_enable;
 raw8 raw8 (.image_data(image_data), .image_data_enable(image_data_enable), .raw(raw), .raw_enable(raw_enable));

logic [25:0] camera_counter = 26'd0;
always @(posedge CLK_48MHZ)
    camera_counter <= camera_counter + 1'd1;

always @(posedge CLK_48MHZ)
    if (ready && mode == 2'd0)
        mode <= 2'd1;
    else if (ready && mode == 2'd1 && camera_counter + 1'd1 == 26'd0)
        mode <= 2'd2;

logic pixel_enable;
assign pixel_enable = cx < screen_width && cy < screen_height;
logic [7:0] pixel;

arbiter arbiter (
    .pixel_clk(clk_pixel),
    .pixel_enable(pixel_enable),
    .pixel(pixel),
    .mipi_clk(MIPI_CLK),
    .mipi_data_enable(image_data_enable),
    .mipi_data(image_data),
    .frame_start(frame_start),
    .line_start(line_start),
    .interrupt(interrupt),
    .sdram_clk(SDRAM_CLK),
    .clock_enable(SDRAM_CKE),
    .bank_activate(SDRAM_BA),
    .address(SDRAM_ADDR),
    .chip_select(SDRAM_CSn),
    .row_address_strobe(SDRAM_RASn),
    .column_address_strobe(SDRAM_CASn),
    .write_enable(SDRAM_WEn),
    .dqm(SDRAM_DQM),
    .dq(SDRAM_DQ)
);

always @(posedge clk_pixel)
    rgb <= {pixel, pixel, pixel};

endmodule

Additionally, you can find the compiled Quartus prime life "mkrvidor4000_top.qpf" in this link. Kindly let me know if there is anything else you'd like me to share.

I did a little digging myself on this issue and with some experienced advice, I was told that perhaps my camera is having power issues. This is a typical issue with the newer Raspberry pi V2 cameras. However, I do not know if that is the real issue here because my camera is V1, which should work.

Also, where can I find the VIDOR_SE library, could you please guide me on that?

Thank you for your time and consideration.

Hello,

The power supply is set with MIPI_GP[0]. You have to wait a bit before you start sending the I2C configuration commands.
I just adapted Sameer's HDMI module, so I'm not too familiar with the rest of his code, but I'll try to look in the next few days.
For the VIDOR_SE library, it is available under GITHUB:

VIDOR_SE library

The VERILOG source code is not yet available because I still have a lot of problem to solve, and code to clean up.

1 Like

Hello,

Hope you are doing well. I was looking into the State machine set up in Sameer's work after you spoke about the power supply and I2C configuration command.

Although I am not entirely sure, just to brainstorm possible issues, probably what is happening with my code here is that the camera is on standby mode, and then due to some Mode_ID error issues the camera skips all the streaming mode and goes to shut down.

My suspicion for the same came from Sameer's post regarding the progress of this project in here.

Here is a glimp of the OV5647.sv file from my project

module ov5647 #(
    parameter int INPUT_CLK_RATE,
    parameter int TARGET_SCL_RATE = 400000,
    // Some ov5647 modules have a different address, change this if yours does
    parameter bit [7:0] ADDRESS = 8'h6c
) (
    input logic clk_in,
    inout wire scl,
    inout wire sda,
    // 0 = Power off
    // 1 = Software standby
    // 2 = Streaming
    input logic [1:0] mode,

    // 0 = 3280x2464
    // 1 = 1920x1080
    // 2 = 1640x1232
    // 3 = 640x480
    input logic [1:0] resolution,
    // 0 = RAW8
    // 1 = RAW10
    input logic format,
    // input logic horizontal_flip,
    // input logic vertical_flip,
    // input logic [7:0] analog_gain,
    // input logic [15:0] digital_gain,
    // input logic [15:0] exposure, // aka integration time

    // Goes high when inputs match sensor state
    // Changing inputs when the sensor isn't ready could put the sensor into an unexpected state
    output logic ready,
    output logic power_enable,
    // ov5647 Model ID did not match
    output logic model_err = 1'b0,
    output logic nack_err = 1'b0
);

logic bus_clear;

logic transfer_start = 1'b0;
logic transfer_continues = 1'b0;
logic [7:0] address;
logic [7:0] data_tx = 8'd0;

logic transfer_ready;
logic interrupt;
logic transaction_complete;
logic nack;
logic [7:0] data_rx;
logic address_err;

i2c_master #(.INPUT_CLK_RATE(INPUT_CLK_RATE), .TARGET_SCL_RATE(TARGET_SCL_RATE)) i2c_master (
    .scl(scl),
    .clk_in(clk_in),
    .bus_clear(bus_clear),
    .sda(sda),
    .address(address),
    .transfer_start(transfer_start),
    .transfer_continues(transfer_continues),
    .data_tx(data_tx),
    .transfer_ready(transfer_ready),
    .interrupt(interrupt),
    .transaction_complete(transaction_complete),
    .nack(nack),
    .data_rx(data_rx),
    .address_err(address_err)
);

logic [15:0] MODEL_ID = 16'h5647;

logic [24:0] PRE_STANDBY [0:2];
assign PRE_STANDBY = '{
    {1'b1, 16'h300a, MODEL_ID[15:8]},   // Read module_model_id high
	{1'b1, 16'h300b, MODEL_ID[7:0]},    // Read module_model_id low
    {1'b0, 16'h0100, 8'd0}              // mode_select <= standby
};

logic [24:0] PRE_STREAM [0:88];
assign PRE_STREAM = '{
  {1'b0, 16'h3034, 8'h08}, // PLL ctrl0: mipi 10 bit mode
  {1'b0, 16'h3035, 8'h41}, // SC common PLL ctrl1:  system_clk_div by 4, scale_divider_mipi by 1
  {1'b0, 16'h3036, 8'h46}, // PLL multiplier: times 70
  {1'b0, 16'h303c, 8'h11}, // PLLS ctrl2: plls_cp 1, plls_sys_div by 1
  {1'b0, 16'h3106, 8'hf5}, // SRB ctrl: pll_sclk / 4, enable sclk to arbiter
  {1'b0, 16'h3821, 8'h07}, // Timing TC: r_mirror_isp, r_mirror_snr, r_hbin
  {1'b0, 16'h3820, 8'h41}, // Timing TC: r_vbin, 1 unknown setting
  {1'b0, 16'h3827, 8'hec}, // Debug mode
  {1'b0, 16'h370c, 8'h0f}, // ???
  {1'b0, 16'h3612, 8'h59}, // ???
  {1'b0, 16'h3618, 8'h00}, // ???
  {1'b0, 16'h5000, 8'h06}, // Black/white pixel cancellation
  {1'b0, 16'h5001, 8'h01}, // Auto-white balance
  {1'b0, 16'h5002, 8'h41}, // Auto-white balance gain, Win enable
  {1'b0, 16'h5003, 8'h08}, // Buffer enable
  {1'b0, 16'h5a00, 8'h08}, // Unused bit set, not sure why
  {1'b0, 16'h3000, 8'h00}, // ???
  {1'b0, 16'h3001, 8'h00}, // ???
  {1'b0, 16'h3002, 8'h00}, // ???
  {1'b0, 16'h3016, 8'h08}, // Mipi enable
  {1'b0, 16'h3017, 8'he0}, // pgm_vcm = 11, pgm_lptx = 10
  {1'b0, 16'h3018, 8'h44}, // Mipi two lane, mipi enable
  {1'b0, 16'h301c, 8'hf8}, // ???
  {1'b0, 16'h301d, 8'hf0}, // ???
  {1'b0, 16'h3a18, 8'h00}, // aec gain ceiling = 248
  {1'b0, 16'h3a19, 8'hf8}, // ctd.
  {1'b0, 16'h3c01, 8'h80}, // 50/60 Hz detection
  {1'b0, 16'h3b07, 8'h0c}, // exposure time
  {1'b0, 16'h380c, 8'h07}, // total horizontal size = 1896
  {1'b0, 16'h380d, 8'h68}, // ctd.
  {1'b0, 16'h380e, 8'h03}, // total vertical size = 984
  {1'b0, 16'h380f, 8'hd8}, // ctd.
  {1'b0, 16'h3814, 8'h31}, // horizontal subsample odd increase number = 1, horizontal subsample even increase number = 3
  {1'b0, 16'h3815, 8'h31}, // vertical subsample odd increase number = 1, vertical subsample even increase number = 3
  {1'b0, 16'h3708, 8'h64}, // ???
  {1'b0, 16'h3709, 8'h52}, // ???
  {1'b0, 16'h3808, 8'h02}, // x output size = 640
  {1'b0, 16'h3809, 8'h80}, // ctd.
  {1'b0, 16'h380a, 8'h01}, // y output size = 480
  {1'b0, 16'h380b, 8'he0}, // ctd.
  {1'b0, 16'h3800, 8'h00}, // x addr start = 0
  {1'b0, 16'h3801, 8'h00}, // ctd.
  {1'b0, 16'h3802, 8'h00}, // y addr start = 0
  {1'b0, 16'h3803, 8'h00}, // ctd.
  {1'b0, 16'h3804, 8'h0a}, // x addr end = 2623
  {1'b0, 16'h3805, 8'h3f}, // ctd.
  {1'b0, 16'h3806, 8'h07}, // y addr end = 1953
  {1'b0, 16'h3807, 8'ha1}, // ctd.
  {1'b0, 16'h3811, 8'h08}, // ISP horizontal offset = 8
  {1'b0, 16'h3813, 8'h02}, // ISP vertical offset = 2
  {1'b0, 16'h3630, 8'h2e},
  {1'b0, 16'h3632, 8'he2},
  {1'b0, 16'h3633, 8'h23},
  {1'b0, 16'h3634, 8'h44},
  {1'b0, 16'h3636, 8'h06},
  {1'b0, 16'h3620, 8'h64},
  {1'b0, 16'h3621, 8'he0},
  {1'b0, 16'h3600, 8'h37},
  {1'b0, 16'h3704, 8'ha0},
  {1'b0, 16'h3703, 8'h5a},
  {1'b0, 16'h3715, 8'h78},
  {1'b0, 16'h3717, 8'h01},
  {1'b0, 16'h3731, 8'h02},
  {1'b0, 16'h370b, 8'h60},
  {1'b0, 16'h3705, 8'h1a},
  {1'b0, 16'h3f05, 8'h02},
  {1'b0, 16'h3f06, 8'h10},
  {1'b0, 16'h3f01, 8'h0a},
  {1'b0, 16'h3a08, 8'h01}, // b50_step = 295
  {1'b0, 16'h3a09, 8'h27}, // ctd.
  {1'b0, 16'h3a0a, 8'h00}, // b60_step = 246
  {1'b0, 16'h3a0b, 8'hf6}, // ctd.
  {1'b0, 16'h3a0d, 8'h04}, // b60_max = 4
  {1'b0, 16'h3a0e, 8'h03}, // b50_max = 3
  {1'b0, 16'h3a0f, 8'h58}, // WPT stable range high limit
  {1'b0, 16'h3a10, 8'h50}, // BPT stable range low limit
  {1'b0, 16'h3a1b, 8'h58}, // WPT2 stable range high limit
  {1'b0, 16'h3a1e, 8'h50}, // BPT2 stable range low limit
  {1'b0, 16'h3a11, 8'h60}, // High VPT
  {1'b0, 16'h3a1f, 8'h28}, // Low VPT
  {1'b0, 16'h4001, 8'h02}, // Start line = 2
  {1'b0, 16'h4004, 8'h02}, // blc line num = 2
  {1'b0, 16'h4000, 8'h09}, // adc11bit mode, blc enable
  {1'b0, 16'h4837, 8'h24}, // PCLK_PERIOD
  {1'b0, 16'h4050, 8'h6e}, // BLC max
  {1'b0, 16'h4051, 8'h8f}, // BLC stable range
  {1'b0, 16'h503d, 8'b00000000}, // test pattern control
  {1'b0, 16'h4800, 8'b00000100}, // MIPI ctrl
  {1'b0, 16'h0100, 8'h01}
//   {1'b0, 16'h4202, 8'h00},
//   {1'b0, 16'h300d, 8'h00}
};

logic [24:0] POST_STREAM [0:0];
assign POST_STREAM = '{
	{1'b0, 16'h0100, 8'h00} // Send to standby
	// TODO: standby spinlock
};


// 0 = Off
// 1 = Pre-Standby
// 2 = Standby
// 3 = Pre-Stream
// 4 = Stream
// 5 = Modify Stream
// 6 = Post Stream (shutting down)
// 7 = Error
logic [2:0] sensor_state = 3'd0;

logic [7:0] rom_counter = 8'd0;
logic [1:0] byte_counter = 2'd0;

// Uninit, Standby, or Stream
assign ready = sensor_state == 3'd0 || sensor_state == 3'd2 || sensor_state == 3'd4;

assign power_enable = sensor_state != 3'd0;

logic [7:0] rom_end;
assign rom_end = sensor_state == 3'd1 ? 8'd2 : sensor_state == 3'd3 ? 8'd88 : sensor_state == 3'd6 ? 8'd0 : 8'd0;

logic [24:0] current_rom;
assign current_rom = sensor_state == 3'd1 ? PRE_STANDBY[rom_counter] : sensor_state == 3'd3 ? PRE_STREAM[rom_counter] : sensor_state == 3'd6 ? POST_STREAM[rom_counter] : 25'd0;

always @(posedge clk_in)
begin
    case (sensor_state)
        3'd0: begin 
			if (mode != 2'd0)
				sensor_state <= 3'd1;
        end
        3'd1, 3'd3, 3'd6: begin
			if (interrupt || transfer_ready)
			begin
				if (interrupt && (address_err || (!address[0] && nack))) // Catch write nacks
				begin
					transfer_start <= 1'b0;
					transfer_continues <= 1'b0;
					byte_counter <= 2'd0;
					rom_counter <= 8'd0;
					nack_err <= 1'd1;
					sensor_state <= 3'd7;
				end
				else if (transfer_ready && byte_counter == 2'd0) // Write address MSB
				begin
					transfer_start <= 1'd1;
					transfer_continues <= 1'd1;
					address <= {ADDRESS[7:1], 1'b0};
					data_tx <= current_rom[23:16];
					byte_counter <= 2'd1;
				end
				else if (interrupt && byte_counter == 2'd1) // Write address LSB
				begin
					transfer_start <= 1'd0;
					transfer_continues <= !current_rom[24];
					data_tx <= current_rom[15:8];
					byte_counter <= 2'd2;
				end
				else if (interrupt && byte_counter == 2'd2) // Write/Read register
				begin
					transfer_start <= current_rom[24];
					transfer_continues <= 1'd0;
					if (current_rom[24])
						address <= {ADDRESS[7:1], 1'b1};
					data_tx <= current_rom[7:0];
					byte_counter <= 2'd3;
				end
				else if (interrupt && byte_counter == 2'd3) // Readback
				begin
					transfer_start <= 1'd0;
					transfer_continues <= 1'd0;
					byte_counter <= 2'd0;

					if (current_rom[24] && current_rom[7:0] != data_rx) // Read did not match expected
					begin
						rom_counter <= 8'd0;
						if (sensor_state == 3'd1) // was a model error
							model_err <= 1'd1;
						sensor_state <= 3'd7;
					end
					else if (rom_counter == rom_end) // This was the last operation
					begin
						rom_counter <= 8'd0;
						if (sensor_state == 3'd5)
							sensor_state <= 3'd4; // Modifications complete
						else if (sensor_state == 3'd6)
							sensor_state <= mode == 2'd1 ? 3'd2 : 3'd0; // Either go to standby or power off
						else
							sensor_state <= sensor_state + 1'd1; // Pre-standby and Pre-stream
					end
					else
						rom_counter <= rom_counter + 1'd1;
				end
			end
        end
        3'd2: begin
            if (mode == 2'd0)
                sensor_state <= 3'd0;
            else if (mode == 2'd2)
                sensor_state <= 3'd3;
            else
                sensor_state <= 3'd2;
        end
		3'd4: begin
			if (mode != 2'd2)
				sensor_state <= 3'd6;
			else
				sensor_state <= 3'd4;
		end
		3'd5: begin // Not entered, modify support still WIP
		end
        3'd7: begin
            if (mode == 2'd0)
            begin
                model_err <= 1'd0;
                nack_err <= 1'd0;
                sensor_state <= 3'd0;
            end
        end
    endcase
end

endmodule

I do not know if that's the issue or if it's even worth looking at. I asked my friends to run the same code on their hardware and it seems to be working fine. I really wanna get to the depth of this issue.
What is your take on that?

Thank you in advance

I tried to assemble the code from Sameer but without success.
Unfortunately, I don't have much time to continue on this path.
I will add a register reading function in my library. This will allow among other things to read addresses h300A and h300B (camera type which should be 6547).
It can still be useful to test the I2C address and the correct response of the camera.

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.