or take the Arduino out of the equation and build the thing again using just a chip and the components I require?
Yes!
If so can I use one of the smaller chips such as on the mini or nano?
Not very easily. The code would have to be recompiled for the smaller chip and then you would have the problem of actually programming the chip.
Don