I'm not sure what you're trying to compile on the Yun, but I've compiled a bunch of stuff and performance was OK. The only things that are going to take anywhere near 10 hours are something like gcc, or the OS distro.
Opencv, It took almost same time as gcc at cross compile, at cross compile Opencv could be compiled on some hardware target but not all. Native GCC in high speed MIPS CPU could help. The list could go on and on.
For compiling huge packages it obviously makes sense to cross compile but for example when I needed a "real" grep it took only a few minutes to compile and install gnugrep.
agree
Then again I'm old enough to remember when a 400MHz Mips WAS a high end machine.
My first computer ( in kits format and I soldered it together when I was kids)
Sinclair ZX81
CPU Z80 at 3.25 MHz
Memory 1 kB (64 kB max. 56 kB usable)

but now there are multiple cores MIPS based computer could run a couple of GHz with a lot of memory.

very environmental green, could run without big heat sink or fan.
http://en.wikipedia.org/wiki/Loongson
For a typical user, cross compiling is a near non-starter: just setting a Debian VM is enough work! Building the openWRT distro on that is a pain (and it regularly fails!).
I start with openvz since it performance near type 0 hypervisor, but back to bear metal + ghost for 10% gain. Now plan to migrate to Xen for Amazon Web Services.
sonnyyu:
...
The Atheros AR9331 (Hornet) is a Wi-Fi System-On-Chip (WiSOC), typically used in AccessPoints and router platforms. It is based on MIPS @400MHz 32 bits CPU with no floating point support, has 64 MB DDR2 RAM and 16 MB Flash Memory.
...
One of sample Web Services: Amazon Web Services (cr1.8xlarge) has 32 vCPU (32 of 1.7 GHz Xeon processor 64 bits CPU) and 244 GB RAM plus 2 x 120 SSD driver.
...
VirtualBox is type 2 hypervisor will be suffer more than type 1 hypervisor.
http://forum.arduino.cc/index.php?topic=235360.msg1726450#msg1726450
As you noted you spent hours getting to the stage of STARTING to compile your project.
4~5 hours to compile complete OS on bear metal 8 cores/16GB RAM/120GB SSD, then 30 mins for gcc and 28 mins for opencv. both could failed and offen will be failed. hit enter key and pray.
Also a lot of packages aren't set up to cross compile nicely.
True, but after careful tune up, I manage keep it at single digit. I do get ~4000 packages from trunk.