Are you running out of program memory (of which there is 8 KB on the ATmega8) or RAM (1 KB available)? If the former, and you have a hardware programmer (e.g. AVR-ISP), you could program the Arduino board without a bootloader to get an extra 1 KB of program space. Otherwise, you might be able to comment out some parts of the core Arduino libraries (in ARDUINO/lib/targets/arduino) that you aren't using. What does your code need to be able to do?