lewisd:
Is there something code or hardware related that you think would make this impractical? I'm just curious if you have any specific reasons, or if it just fills you with a general sense of unease.
My biggest worry is how long programming might take. Using a YouTube video on a phone or something, I think it would be tough to get more than 30 bits/second (or whatever the video framerate is) without a much more complicated mechanism. Similarly, even a dedicated app on a desktop PC would be limited by the 60Hz refresh rate of a monitor. The only way to do fast programming would be to use an LED, I would imagine.
There's actually a technology out there that uses LED lamp sources (like in an office environment, that replace overhead fluorescent bulbs) for high-speed networking, termed "li-fi":
Also, there is (was?) the IRDA standard (Infrared Data Association - Wikipedia) for optical data transfer. It used to be something installed on laptops and printers, mainly - but you don't see it much (if at all?) any more.
I can see what the OP wants being possible, using high-brightness IR LEDs (or clusters), and IR receiver modules (or maybe a photocell/phototransistor as proposed - but I think a proper module would be better); it seems like it would basically just be a hardware issue. The best way to experiment would be to hook up an FTDI cable (to get TTL serial from the PC) to a circuit doing IR TX/RX, and on the other end having the same thing, but connected to pins 0 and 1 on the Arduino.
In fact, a quick google of "IR serial" turned this up:
http://iguanaworks.net/products/series-transceiver/
/once again, something the OP could've done...