Cameras...not exactly as advertised?

Right.

I am a bit of a photographer (I get paid for it apparently).

A question has always bugged me.

A Canon 5DIII takes 14 bit per channel images (so I assume RGB with 2^14 resolution in each channel).

The camera is 22.3MegaPixels...it produces roughly 4000x6000px images.

So my working was:

Size of RAW file (they format the camera exports) = 14bits * 3 channels (RGB) * 22.3E6 (MP)

Should be around 120MB per file...

But they are not. They are like 25MB per RAW file as shown on the camera (and in the Manufacturer Specs). This equates more to around 1 byte (8bits) per pixel. What is going on? Is the manufacturer taking 14 bit images on the sensor and then squidging them down to 8bit colour when saving each pixel on the CF card?

Seems a bit of a cop-out to be honest.

How is this an Arduino question?

There are several methods for loss-less data compression.

...R

Robin2:
How is this an Arduino question?

There are several methods for loss-less data compression.

...R

Am I mistaken:

General Discussion - 
Feel free to talk about anything and everything in this board.

Johnny010:
Am I mistaken:

IMHO Yes. I believe it means "General Discussion about Arduino stuff"

Use Bar Sport for things that have nothing to do with an Arduino.

However your particular question looks like it would benefit from the expertise on a Camera forum.

...R

I see.

I avoided camera forums because to be honest, although the technical photography ranges from bad to pretty good, their actual knowledge of how a camera works (especially digital level / the computing bit) they are terrible to non-existent.

Felt a place with more like minded techies would have been a better shout.

Johnny010:
their actual knowledge of how a camera works (especially digital level / the computing bit) they are terrible to non-existent.

Interesting.

Did my comment about data compression help?

I'm afraid I'm a lazy phillistine and never use RAW on my camera.

...R

It may do...I thought maybe once they get the 14bits per channel (RGB) they can then use some fancy maths to pack the file size down without losing data (lossless) using some sort of fancy data categorising algorithms...

When looking for what is in a RAW file, there is the usual expected camera data and then the vast vast majority is the pciture data (as you'd expect) but I googled for a while and can't seem to find the actual "protocol" if you like of how the data is stored byte wise.

Is it a bit like a BMP with each pixel value.
Is it more of a LUT with co-ordinate systems or something?
Is it far more complex?

Couldn't really find these answers :(.

It would have explained why a 23MP image with 42 bits per pixel comes out at about 1/5th of what youd expect! Just when it is almost exactly a byte per pixel...it begs my curiosity.

RAW format image files are not adjusted to a color space. That may or may not mean they are compressed.

Johnny010:
Seems a bit of a cop-out to be honest.

Seems like you need to spend a bit more time reading about Canon's RAW file format. Or any other manufacturer's RAW file format.

I never said they were...and that is just a bit like the map() in arduino. It just shifts the upper and lower bounds (except in photography it will keep the resolution and not "bin" bits).

I assumed simply, for example, here is the page for the Canon 5DmkIII:
Canon 5DIII specs

Specification Details
Still Image Type 
JPEG: Fine, Normal (Exif 2.3 [Exif Print] compliant) / Design rule for Camera File system (2.0),
RAW: RAW, sRAW1, sRAW2 (14bit, Canon original RAW 2nd edition),
Digital Print Order Format [DPOF] Version 1.1 compliant

So it says it takes 14bit images. I firstly assumed that it means you get 14bits PER pixel with each having an RGB component...so 3 x 14 = 42bits per pixel.

Now on further reading, I believe there is a filter of the pixels. A Bayer filter...which sits over pixels allowing light of only one colour (R G or B) through. So on say a 20MP sensor...an RGGB Bayer filter means 1/4 are red filttered, 1/4 blue and 1/2 green.

Then maths occurs to "calculate" the actual colour of the pixels by looking at neighbours values in the RGB channels and each pixel has 14 bit depth after the maths/"demosaicing". So each pixel is 14bits as a whole when recorded on the RAW file. Not 14 bits PER channel...but 14 bits for all channels. Maybe 4 x 4 x 4 x 2 (alpha) or something...

Is this more "correct" thinking?

It leads closer to the 30MB I expected...

The 14-bits is per colour. i.e. 14-bits for each of R, G and B. It would not make sense for such an expensive camera to store RAW images with only 14-bits per pixel. Pretty well any camera these days, including much cheaper ones than the 5DIII, have 24-bits per pixel (i.e. 8-bits for each of R, G and B).
The RAW files usually use proprietary lossless compression algorithms to bring the size of the file down to something more manageable than if there were no compression at all.

Pete

el_supremo:
The 14-bits is per colour. i.e. 14-bits for each of R, G and B. It would not make sense for such an expensive camera to store RAW images with only 14-bits per pixel. Pretty well any camera these days, including much cheaper ones than the 5DIII, have 24-bits per pixel (i.e. 8-bits for each of R, G and B).
The RAW files usually use proprietary lossless compression algorithms to bring the size of the file down to something more manageable than if there were no compression at all.

Pete

Thank you! This is more the answer I was looking for but could not seem to find it.
I assume these "lossless compression" algorithms are pretty well secret...or are there resources you could point me towards (keywords would be nice) about highly efficient image RGB channel lossless compression?

I assume these "lossless compression" algorithms are pretty well secret

secret a.k.a proprietary.
There is (or at least, was) a program called dcraw which, IIRC, can convert some of the early Canon RAW files (e.g. those from the 10D). If you can find the source code, you might be able to figure out how the compression works from figuring out how dcraw does the decompression.

Pete

Why would the compression system be secret?

Your camera comes with a PC program to de-compress the data and from that point on you have full access to every element of every pixel.

The key requirement of the compression system is that it does not take so much computation that it slows down the camera.

I suggest some Googling and study of lossless compression algorithms.

...R

I would assume that the .raw format for a 14bit camera would have 14 bits per pixel of whatever color was in the "Bayer filter" for that particular pixel. No "RGB" for each pixel, just R, G, OR B. The converter for raw images has to know which pixels are which (that COULD be recorded in the raw file too. It would only take a couple of bytes, total. "RGBG, repeating." or whatever.
There are some multiple-sensor video cameras, but I don't think I've seen it in a still camera (the physical containers aren't very compatible, and still camera sensors tend to be much larger and more expensive. Yahoo | Mail, Weather, Search, Politics, News, Finance, Sports & Videos)

westfw:
There are some multiple-sensor video cameras, but I don't think I've seen it in a still camera (the physical containers aren't very compatible, and still camera sensors tend to be much larger and more expensive. Yahoo | Mail, Weather, Search, Politics, News, Finance, Sports & Videos)

The sensor measures light intensity and the ADC converts to 14bits (or whatever). Light being a mixture of colours, white is 1/3G + 1/3R + 1/3B. The Bayer filter, (based on how the retina interprets light,) prioritises green over blue and red (50%, 25%, 25% respectively). For every pixel intensity value, the green element is complete (1/3), while red and blue are half what they actually are (1/2 * 1/3) but the true values can be interpolated from the intensity of neighbouring pixels. The downside of Bayer filtering are artefacts and colour bleeding appearing on high contrast boundaries.

The exception, and as far as I know it is unique, is the Sigma Foven x3 'direct imaging' sensor. Light waves of different frequencies, penetrate silicon to different depths, much as they do in water. Sigma take advantage of this by stacking sensor material, each 'plane' producing discrete R and G and B intensities at the location of each and every pixel. The result is a (relatively) compact APS-C sensor which does not require a Bayer filter.

RAW formats are a recording of an intermediate step in a one way, image processing chain; between the ADC sampling the rather noisy electrical output of the sensor, and the post processed image. Typically, at the RAW stage, noise has been removed but white balance has not been introduced. So in a RAW editor you can adjust the white balance but can not reintroduce the noise that came off the sensor.

I would presume the RAW formats are kept secret to slow down the competitors between model generations. Knowledge of the format would make it much easier to reverse engineer the processing engine, which, apart from lens investment, manufacturers depend on to differentiate products made from largely similar components. Compression is the least interesting component, I would have thought.

There are only so many ways to compress digital data losslessly; zip, lhz, arj etc.) Take your pick.

Your camera comes with a PC program to de-compress the data

Yes, but for conversion of RAW images, that program contains proprietary code which does the decompression. If you want to write your own program to convert a Canon RAW file to BMP, JPG or whatever, there is no publicly available documentation which explains the compression algorithms or other info you would need to be able to convert a RAW image to, say, JPG.
You can get a software development kit from Canon which converts a RAW file to (IIRC) BMP or TIFF and also gives you control over various things like the white balance used during the conversion. I haven't used their SDK for several years but they still have a website describing the two SDKs - one for EOS cameras and one for Powershot.

Pete

I believe that the picture is compressed as a TIFF file, but the format of the data is different. In fact, there are many RAW formats, and nearly none of them are interchangeable. DNG was proposed as a universal RAW format, but aside from Leica, it was largely ignored. Canon and Nikon have both put a lot of work into the extras provided by their own CRW and NEF formats.

el_supremo:
Yes, but for conversion of RAW images, that program contains proprietary code which does the decompression.

It's a long time since I tried the RAW feature on my Fujitsu camera. I assumed that once the file was decompressed it could be edited with any image editing software - such as GIMP. I must experiment again.

...R

I assumed that once the file was decompressed it could be edited with any image editing software

Yes, that is true of any camera if the RAW image is converted to any standard image format such as BMP, JPG, TIFF etc. In Canon, and I believe Nikon and other cameras, the conversion from RAW (which involves more than just decompression) to another format is the bit that is usually proprietary.

Pete

el_supremo:
Yes, that is true of any camera if the RAW image is converted to any standard image format such as BMP, JPG, TIFF etc. In Canon, and I believe Nikon and other cameras, the conversion from RAW (which involves more than just decompression) to another format is the bit that is usually proprietary.

Pete

Is that conversion “on the fly” as AcDSee opens raw images directly and does not seem to convert that I can see ?
Gimp also seems to just open them too

Nikon raw formats here.