I've only skimmed your code, but it looks like you're handling the SPI communication correctly. Do you have access to an oscilloscope? It would be helpful if you could see what your MOSI, MISO, and SCK lines look like. Have you verified that you're using an SPI clock speed that your sensor can handle? Is there anything in the datasheet about a required delay between when you send the register address and when you initiate the junk-byte transmission that will retrieve data from the sensor?
I made a post recently to help someone with an SPI problem here:
http://forum.pololu.com/viewtopic.php?f=1&t=986
My post links to some sample mega644 code for performing SPI communication, but this should be almost identical (if not exactly identical) to what you'd do on the mega168.
- Ben