Thanks for the response guys. I will defo check out that link now thanks! The openGL code will most probably be written in C or C++, of which I have not heard anything about communicating with the serial port. I was thinking reading/writing files but getting the three softwares to do this in real-time, and have similar run-times will be hard.
I'm not exactly going to be using mouse co-ordinates in the arduino, was just an example. It's more a case of creating some GUI in openGL, signal processing some input data using matlab and some hardware interface based on the arduino. The arduino will be simply commanded using a binary word about the status of it's ports and how to process data.
I'm not going to be using an arduino for this project as the bit rate is too low, but I recon it is probably a good starting point as I have not tried interfacing a micro-controller with several environments before.