Yes, each of the Teensy models has a good USB stack which includes USB MIDI (class compliant) as well as USB Keyboard, mouse, etc. The USB mode is chosen with a menu option.
In terms of compatibility the Teensy 2 and Teensy ++2 are both AVR-based so code that works on an Arduino Uno will likely work there; a major difference is that Serial (over USB to the serial monitor) and Serial1 (using Tx and Rx) are independent 9so you can use both at once, unlike Arduino).
Teensy 3.0 uses a fast, 96MHz ARM chip so it is more like the Arduino Due. Except you can use the Arduino 1.0.5 with it, its not restricteed to 1.5.x.
There are a lot of libraries available
, which either work as-is or have been ported to add Teensy support or have been specially written for Teensy.
Its a good choice for a USB Midi device.