is there anyway to make smaller antenna for FPV system ?

if you have a piece of decent 75 ohm coax, why would you want to attach antenna to it?
just use cable's core as the antenna. strip it to needed lenght making shure that the screen doesn't touch it, then solder other end to the antenna output, solder screen to the ground and you should be fine.

tv coax will be a pain to work with though. those are rather thick and not very flexible. not exactly something good for RC imo