Yargh - trouble bootloading arduino-based board

Hey all,
I've just begun my first attempt at designing an arduino-based board for a project I'm working on. Actually it's my first time trying to design a board at all really. Essentially I wanted it to have a small footprint, and an on-board motor controller. I went with the L293 for running the motors because it didn't have to be high performance, it's a pretty common device, and it was available in an SMD version (L293DD).

Here's where I'm at. I designed the board, and had it sent off for manufacturing. I've soldered it all together and am trying to use my AVRISP Mk II to burn the arduino bootloader onto it. I keep running into the same problem though.
I've tried using AVRStudio, AVRdude and the adaptation of avrdude built into the Arduino IDE. All of them seem to fail with the same results.
The problem seems to be that my ATmega168 isn't communicating back any information about itself (signature, info about fuse bits, etc). Avrdude tells me "initialization failed, rc=-1". Avrstudio isn't able to obtain any information either, with similar results.

I know the programmer is connected properly and I'm pretty sure my pinouts are all correct, because the programmer is lit up all green when I connect it to the ICSP headers. When using the same sort of configuration on a Diecimila, it works perfectly.

Now I'm stumped as to whether it's a hardware problem or if there's some sort of lock bits on the virgin atmegas that are keeping me from working with it. I'm not even entirely sure what the best way would be to begin debugging it, as I've been staring at these things for so long I'm going cross-eyed.

Does anybody have any suggestions on what the problem might be or how I might diagnose the problem? I'd really appreciate any help I can get on this - the time I've got to work on it is starting to run short.
Thanks again!

Apparently you can't make external links on your first post in the forum, so here are the eagle files:
http://www.groundhogmedia.ca/rich/file/board1.brd
http://www.groundhogmedia.ca/rich/file/board1.sch

It sounds like a hardware problem. Unfortunately, "how do I debug my hardware?" never has an easy answer.

Check the RST line to make sure it's not being forced low when not programming. Check to make sure it does go low when the programmer tries to communicate with the AVR.

Check to see if the SCK line is oscillating during communication. Check it right at the AVR pin, not at the connector.

Then check MOSI...same thing: see if it wiggles when you try to program.

Finally, check MISO. Does it ever change?

BTW, your schematic has an error: because you overlapped a net with the pin on the AVCC pin, the AVCC pin is floating and is not connected to +5V.

Thanks Rugged! You saved me! The schematic does indeed have an error with the avcc pin floating and just overlapped, not connected. Turns out the AVR reset pin was in the same situation...no connection in the schematic and somehow I missed it on the board layout. I jumped it to the next pin over (one of the ADCs), and connected that to the rest of the reset circuit and it worked!

Now, at the risk of sounding needy...I have one more problem. I got the bootloader on successfully (yay, lights blinking!), but I'm trying to do the standard arduino sketch upload via FT232RL. The tx line jumps around on the scope for a bit, but then stops and the software tells me "programmer is not responding". Anybody have any idea what might cause it to not respond? The reset line is being pulled low correctly once (from what I can tell, the Diecimila does this twice - once before loading the code and once after to restart the sketch). Not sure what's going on here. Any pointers to places where I could learn a little more about the procedure the bootloader goes through would be greatly appreciated too. Thanks again!

It could be a timing mismatch. What crystal clock frequency are you using? Did you program the fuses when you programmed the bootloader to use an external crystal?

Any errors in the above will mean that the baud rate the bootloader is setting up will be incorrect.

Hmm, good point. I'm using a 16mhz external crystal, and the scope shows it's putting out the correct speed there. For the fuses, I used the Arduino IDE's function with fuse burning + bootloader uploading in one, so I imagine it should be set to the stock fuses for a diecimila. I'll keep that in mind though, thanks.

http://www.groundhogmedia.ca/rich/file/board1.sch

I ran the error check on your schematic and came up with 17 warnings that need to be fixed. They all have either a "missing junction" or "overlaps pin" problem. I am surprised the board worked.

You might also want to look at the DRC errors on your BRD as well.