If analogread() returns 2bytes this corresponds to 16bit of information, it states that the mcu has max 10bit adc, how is this possible ?
Because there is no ten bit native data type on the AVR microcontroller, the next largest size must be used to store the ten bit result.
when I connect analogpin to 3.3V on board it fluctuates around 172-174.
You're missing precision here - you're only taking the least significant eight bits
(3.3 / 5) * 1023 = 675 (or thereabouts), which is more than you can store in a single byte.
The 172 you see is the result of the truncation of the most significant bits.