I don't know how you organize your video buffer array, 128 x 96 as 12 k-pixels doesn't fit in arduino memory, only if you do bit-staffing keeping memory 1 bit per pixel - 1536 bytes.
Anyway, this is how I'll calculate if the shape is round. If you noticed, I choose 1d video array in my project, to overcome memory limits the same time not sacrifice resolution ( 832 x 512 ). First of all, I'd run array in "for" loop to find upper and lower line of detected object. (Power measuring subrouting does it in the linked code) It provides X1:Y1 and X11:Y11 (look at the pictures below).
Radius = (Y11 - Y1) / 2;
Yc = Y1 + Radius;
Than , for any points,
Rn = sqrt( (Xn - X1)^2 + (Yn - Yc)^2)).
All Rn must be equals, with specified accuracy. Simply running check, algorithm would make a decision if its round or not. Could be some kind of optimization - filtering/noise reduction implemented. I think arduino is good at sqrt, about 50-100 per 1 msec, so math should be optimized as well for 8-bit. One more things, correction coefficient has to be precalculated, defining number of pixel per line (Y expressed as lines numbers, but X - is 16 MHz "ticks"). For example, for camera with 4:3 vision field, ( 832 / 4 ) x 3 = 624, that isn't 512 ( 0r 492 ?)
