storage on a microSD

I have a microSD and a mSD module hooked up to my Arduino and I wanted to know what I can do in terms of data storage.
At the moment my sketch is taking up around 21k of the 32k byte size and I've still got much to add.
So I was wondering if there was a way for the Arduino to read bits of code/data from the SD card directly instead of using up the Arduino's memory.

Looking at the guide on the site, I noticed this bit of code:

  File dataFile = SD.open("datalog.txt");
  if (dataFile) {
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
 }
dataFile.close();

Is this how it works basically? Could I have the SD open a cpp file for example, and have it read and process the data?

No.
The uC runs on compiled code that is accessed from flash.
Data on SD/uSD is just data, not code that can be executed.
To run from SD, you can run a program called bitlash that interprets data on the SD card and executes them as instructions.

If you want to keep up the same performance, by executing from flash, go to a chip with more memory. I am in favor of the 1284P. 128k flash, 16K SRAM (twice that of a mega).
I offer Duemilanove style boards (FTDI interface vs 16U2 -unless you can find a 16U2 module) as kits or assembled boards:
http://www.crossroadsfencing.com/BobuinoRev17/

bitlash sounds interesting and thanks for recommending the board, I might purchase one soon.

I have TFT also wired up to my Arduino.
Could I get bitlash to print text on the screen for example?

Don't know. I've not tried it. My biggest project used a 1284 with a 14000 byte array to drive shift registers on other card.