I think you're confusing people by saying DTR pin on the board. The DTR pin would be on your FTDI cable/board. The DTR pin generates the reset signal which is connected to the reset pin on the Atmega board. If you, in fact, do have a DTR pin on your custom board then you need to tell us what it's connected too. Posting a schematic would make it easier for people to help you.
What I would do, if I were in your shoes, is try and solder a very small wire directly to the Atmega reset pin (it's not that difficult) so you have the reset pin broken out to continue troubleshooting. Also since your Arduino is a 2009 Duem. it's fairly easy to pull the chip out of it's socket. Then simply connect the Rx's, Tx's and RESET'S (along with ground) between the two boards and upload your sketch as you normally would. This would at least tell you if the problem lies within your custom board.