Well, there's this:
http://cds.linear.com/docs/en/datasheet/1328f.pdf
"The LT®1328 is a photodiode receiver that supports IrDA® data rates up to 4Mbps. The LT1328 plus five external components is all that is required to make an IrDA compatible receiver."
That proves it's possible.
Another choice which offloads the IrDA encoding/decoding from the Arduino is this:
http://www.nxp.com/documents/leaflet/75015676.pdf
It can handle the IrDA 1.15 Mbps mode.