I don't have a list of libraries that don't work.
Any library that calls delay() or has a wait loop for something to complete will block all lower priority threads.
The ideal way to handle devices with an RTOS is to start an operation in a thread then wait on a semaphore. When the operation completes, an interrupt occurs and the device ISR signals with the semaphore to wake the waiting thread. This means many Arduino libraries need some mods for optimum performance.
The malloc() included with new versions of the Arduino IDE will not work in threads. This is O.K. since dynamic memory is not a good idea in real-time systems after setup.
FreeRTOS uses the most RAM and flash, ChibiOS/RT is smaller, faster, and has more features. Nil RTOS is the smallest but has few features and probably is not appropriate for your system.
Certainly Due is the minimum but I can't guess what you will need.