Well I don't know for sure what to try next. I got it working so in principle it should work. In your shoes I would be starting to swap things around. Do you have a Uno or similar you can hook the wires up to, to see if it works there? Or another programmer like a USBtinyISP? Or another chip?
I have other atmega328 ... i tray swap it but no success
i swap crystal and cap
i swap the breadboard
i only have an arduino mega 2560 ... i have plan to make an arduino on breadboard and upload sketch with FTDI Basic from sparkfun but without the bootloader i can do nothing... maybe i will buy another atmega238 with the bootloader preloaded to upload bootloader to my other atmega328 (ArduinoISP)... or just a UNO