For a mux, this is cheapest: https://www.sparkfun.com/products/9056 at $5, but this is probably easiest (well not really, it's really the same basic thing but with 3 of them): https://www.sparkfun.com/products/9832
That one for $5 will give you 16 inputs, should be enough, right?
You can set up a photo-diode to work with a digital pin so you don't have to worry about an analog mux at all. But, maybe kill two birds with one stone and just get these: https://www.sparkfun.com/products/9454
These are IR reflectance sensors. You would install them just on one side and they will reflect off a close object and you can detect the book. These will need to be fairly close though and you would get better results if you were able to put white (tape) on the spines of the book. But at $3 a pop, they are worth investigating. They are just plug and play. I forget how many books you said were total. 16 books would be $48 in sensors, but you don't need the mux.
Did you need a display for the count?