This sounds like a challenge...
Let me know how converting a 32-bit codebase to 8-bits works out (I am not saying it is impossible, just that you aren't appreciating the scale of what you are proposing!).
Is there any way you can increase code memory for the arduino?
AFAIK, no.
There is also the tiny matter of having to deal with the Harvard architechture of the device, which wouldn't be a big deal if SRAM was much, much larger. You could potentially use an SPI serial eeprom device, or serial RAM - to make up for the lacking space, but you would have to write your own drivers (or hack the standard C libraries to support them?) to utilize the extra room for data. This would be just to gain extra area for variables, arrays, and such. You would want to add a filesystem (or something similar) on top of that too.
Your best bet, if you are serious about such a thing, would be to look into how old 8-bit operating systems were created; popular ones of the period were CP/M, OS-9, and Flex. I am sure there are open-source or freeware source for such older systems you could study (unfortunately they are probably all written in assembler, targetting a particular CPU, bus, disk subsystem(s), etc).
One that is fairly "detailed", used by many enthusiasts still today, though, is OS-9 for the Motorola 6809 8-bit cpu (TRS-80 Color Computer); the open-source project NitrOS-9 can be found here:
http://www.nitros9.org/This is essentially the "modern" version of OS-9 for the Color Computer.
There are probably other resources, too, for "minimalist" 8-bit operating systems. I also know that others here on this forum have created something akin to an OS for the Arduino (some more complex than others) - so this kind of thing is possible to an extent. There are benefits and downsides to using such a system, of course - the biggest one being speed. But you could potentially gain real-time control over the i/o pins and other functionality of the Arduino, plus (if you have a scripting language included - unless there is a way to segment the program flash RAM?) you could have multiple "programs/processes" running at one time (not true multitasking, but something close - more similar to Windows 3.1 than anything).
Ultimately, though, at least on the 168/328 - it would be nothing more than a curiosity and/or learning experience to attempt such an exercise. It reminds me in a great way of when people were hacking QBASIC/QuickBasic to create GUI "operating systems" for DOS, back in the late 1990s. A lot of interesting things came out of that experimentation; maybe the same would be possible for the Arduino.
Perhaps this is one area where a "multi-cpu" Arduino (using SPI or I2C - or even a dedicated parallel bus using the digital i/o lines) could be used as well...?
