Yes it is the library code you have to hack as well. Unfortunately the 8X8 monome is a different arrangement and so the code won't help you. I have never done more than a 4:1 multiplex on the TLC9540.
What you have to do is to increase the count before you wrap round. At the moment this is done with an increment and an AND with 3 instruction. That will only work for multiplex lines that are powers of two. You need to do something like:-
index++;
if(index >= 7) index=0;
Okay, I'm going to try to take it slow and do as much research on my own as to not drive you crazy while I figure this out.

A couple of questions to help get me going though..
1) Are there any resources/documentation on multiplexing in general or this driver that I can reference?
2) I'm trying to wrap my head around the LED buffer code. If I read the comment correctly, each LED is 12 bits (0-4095) and each hex number is 1 byte (8 bits). Does this mean that each led 'shares' a byte with its neighbor? Also, what does 'bb', 'bg', 'gg' etc refer to? I figure this is some simple math thing
* Each value is 12 bits (0-4095). 24 bytes is 192 bits, which is 16 led's * 12 bits each. Each of the numbers in the array below is 1 byte (0-255).
* For example,
*
* | 1st byte | 2nd byte | 3rd byte | 4th byte | 5th byte | 6th byte | ...
* | LED 16 value | LED 15 value | LED 14 value | LED 13 value | ...
*/
// Change this array initilisation to get a diffrent switch on pattern
static uint8_t ledBuffer[] = {
// row 1 (botom)
// lower LEDs not used bb bg gg rr rb bb gg gr rr bb bg gg rr rb bb gg gr rr
// row 2
0x0, 0x0, 0x0, 0x0, 0x0, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x04, 0x00, // red row
Also, this doesn't work yet, but do you mean something like this?
scanindex++;
if (scanindex >=7) {
scanindex=0;
}
scan = (scanindex) & 3; // Increment row count
bRow = pBuffer[scan]; // point at next row in the bufer precalculated to save time