Recommended camera module for fast capture of single pixel row?

[plengqui]

What would be the best camera module or image sensor for this project?

I want to detect distinct monochrome intensity changes at points along a wide (1m) and very narrow (1cm) area. I think about 1x400 pixel resolution is ok. End-to-end latency needs to be very low, preferrably <5 ms. Note that both frame rate and latency is important.

The project idea is to create a MIDI keyboard out of an old traditional piano, by a camera visually detecting the movement of the felted "hammers" that hit the strings in the piano when a key is pressed. Acceptable latency end-to-end including sending the "key down" MIDI message is <10 ms. The camera is to be placed inside the piano about 40cm above the row of hammers facing down. To improve visual contrast, the hammers could be spray painted black and equipped with reflex tape on top, and a bright LED can be placed beside the camera. 

[Koepel]

The Raspberry Pi has nice cameras with a good Sony 8 Megapixel sensor.
Most Raspberry Pi boards have already a connector for a camera. However, there is one board that has a mini-connector.
https://www.raspberrypi.org/products/camera-module-v2/

There is also a ESP32-CAM. This is a nice tutorial: https://randomnerdtutorials.com/esp32-cam-video-streaming-face-recognition-arduino-ide/.

When using an Arduino board, then a camera might not be possible. You could install 88 sensors. One for each key. I think that is doable.

[wvmarle]

If you don't care about the force the hammers hit the strings with (you're basically turning your piano into a harpsichord), many other options become available, such as indeed small switches.

88 switches need 19 I/O pins in matrix configuration or just 10 when charlieplexed. A 10 ms latency is no issue for an Arduino, which I expect to be able to scan that matrix a few times over within that time, even if charlieplexed.

A camera placed 40 cm above a row of hammers in a piano requires a VERY wide angle view - the row of hammers is about 3 times that wide. The difference in distance to the far end vs. the middle makes for a highly distorted image, and you need a high aperture and corresponding bright lighting to get a focussed image.

[plengqui]

Thank you for the reply @Koepel! Sadly I'm afraid none of the suggested solutions will work.
A raspberry will not reach below 50ms latency due to the driver. Since only a single pixel row needs to be monitored, this should be possible with an arduino, provided I can find the right camera module.
The ESP32-CAM (which uses the OV2640 camera module) maxes out at 60fps, and defining a narrow window (Region Of Interest) will not result in higher framerates, according to the datasheet.
@wmarle: I do care about velocity too, the idea is to monitor two or more rows. But I'm starting with one row as a first proof-of-concept  :-)


While the project is possible to do with mechanical or electromagnetic sensors, a visual approach is less invasive and easier to install.

[wvmarle]

As you also want to measure velocity you're likely going to need way higher frame rates than 100 fps (which is what you need just to reach the 10 ms maximum delay), probably something to the tune of 500-1,000 fps. That's high speed camera territory (and of course they also demand serious lighting to work). Your resolution also must be high enough for accurate measurement of the speed (you will have only a few frames for that at best!).

You're going to need a lot of computing power to be able to do real time image recognition for that kind of process. I hope you have a very healthy budget for this project.

The more realistic solution is probably an FSR built into each key, sensing when the player's finger presses down on it and with how much force. That just needs an array of 88 ADCs...

[jremington]

the hammers could be spray painted black and equipped with reflex tape on top

Mount an array of 88 optical reflectance sensors on a strip above the hammers.

[plengqui]

Thank you for great input :-)

To clarify the ide: the camera is above the hammers. It is a standing piano so the camera sees the edge of the hammers. So the movement is in the picture plane. I intend to see the movement of the hammers from simple hammer seen or not seen, essentially on/off, logic. The next iteration of the project would be to monitor a couple of such 1x400-ish pixel rows and detect when a hammer "enters" and "leaves" each row, register the exact timing and from that determine the velocity. But in the first iteration I just want to register state as "moving" or "resting" per hammer, with a single pixel row view.

I think a perfect component would be a mainstream (=cost efficient) monochrome image sensor like the OV7175, set to view a thin 1x640 window (Region of Interest = ROI). If I find a sensor with the right scan control logic, it should be possible for a module that is capable of 30pfs at 640x480 resolution to deliver a 1x480 window at 30*640 = 19.200 fps. Right?


The problem is that all image sensors I have found so far don't increase the frame rate when set to a more narrow ROI. Do you know of any image sensor that can increase framerate whan ROI decreases?

[MarkT]

They must exist, otherwise the 1000fps setting on my digital camera wouldn't work - but I've
no idea how to efficiently search for relevant parts.  Be imaginitive with your search terms, look
for similar projects too, not just sensors, as this may already be solved somewhere.

[wvmarle]

If I find a sensor with the right scan control logic, it should be possible for a module that is capable of 30pfs at 640x480 resolution to deliver a 1x480 window at 30*640 = 19.200 fps. Right?

Only if you completely ignore the physics of light detection in the CCD.

[plengqui]

Only if you completely ignore the physics of light detection in the CCD.

@wvmarle: i certainly wouldn't want to ignore the physics. Any further insight or links you could share on this issue would be greatly appreciated. Is this limitation similar for both CCD and CMOS sensor technologies? 

This page on imaging basics says: " The maximum frame rate for a system depends on the sensor readout speed, the data transfer rate of the interface including cabling, and the number of pixels (amount of data transferred per frame) ", but also " However, there is a finite minimum time between exposures (on the order of hundreds of microseconds) due to the process of resetting pixels and reading out ". Since hundreds of microseconds translates to thousands of fps, it still sounds workable? But please share any experience or deeper knowledge.

[plengqui]

I just found out there are framerate calculator tools online. This one for example says a PYTHON480 sensor could do 1600 fps using a 4x600 pixel Region of Interest. Neat!

[wvmarle]

Individual pixels in the sensor need to receive a certain amount of photons per time unit to record an image, that's why super high speed cameras need super bright lights (the light of a few kW of white LEDs per square meter kind of bright). That is simply how fast a pixel can react, you can't get frame rates faster than the time it takes for the sensor to sense the image. That's also why you have to make sure there's a lot of light inside your piano to have your CCD or CMOS or whatever type sensors pick up the image.

Besides there is indeed also the data transfer rate from the image sensing chip to the outside world, limiting how many images per second you can actually transfer from your sensor to your image capture device. That can be helped greatly by reducing the number of pixels read.