Which pins are you using for your connections? It would appear that Adafruit and your library are doing bitbang SPI rather than hardware SPI.
I have found that a fair amount of bitbang code seemed to be tested only for the frequencies of the AVR Arduinos. The high clock rate of the Due would make that tough. The official SPI calls have a frequency but it's different from the older functions, it wants an integer 1-255 but that is not explained but appears to be a divisor of the 84 MHz clock. Simulating slower clocks with delay would appear very hit or miss but could work.
The gLCD library has been ported to work on the Due but it appears to differ from the code Adafruit publishes by quite a bit.
http://arduino.cc/forum/index.php/topic,144446.msg1111046.html#msg1111046Having worked on another SPI library for the Adafruit 1.8" TFT LCD the past days has reaffirmed my opinion that this area is not well documented at all and could use some community fleshing out, both hardware and bitbanged.