I am trying to develop a board with three processing units on it i.e 2 ATmega328 and 1 ARM cortex M3. For that i was going through the schematic files of different arduino products and found out that in some cases (like nano) a FT232 IC is used while in some (like UNO and DUE) a ATmega 168 controller is used instead. So my question was that how do you select and decide that which IC should be used?
Probably you mean ATmega16U2 (not 168) vs FT232 on the USB/serial converter place.
ATmega168 is same family as 328. It has just half of 328's memory but can be used instead 328 as main microcontroller. Older boards used ATmega168.
FT232 is USB/serial converter and was used in older boards. Later was replaced by ATmega8U2 an then ATmega16U2. ATmega16U2 requires firmware to be loaded because it is general purpose MCU while the FT232 is supplied ready to use.
I think better choice for you would be FT232 not only more simple use but has serial transfer with less errors throughout serial speeds as the ATmega8/16U2.
You select the ICs to use according to your needs.
Also you consider if you can handle surface mount devices or do you need everything to be through hole.