I have a license, although it's not required here.
So 5 wires to the 8 - 16 switches is enough to detect which switch is being pressed? What happens if you press 2 buttons simultaneously?
This is not multiplexing, this is shift registers. Each and every switch separately corresponds to one bit in the shift register, when you clock (send a clock pulse to) the shift register, it moves the state of the bits along one step and they appear one by one at the output. You then "chain" them so that the output of one register is fed into the next one and so on. If you have eight such "stations" of 8 switches in a chain, then by clocking, you read all 256 switches one after the other, and similarly for whatever other combinations you want. The Arduino uses only three pins, a "latch" output to cause all the switch positions to be loaded into the shift registers, a "clock" pulse output to cause each shift by one bit at a time, and a data input to receive the bits, one by one.
So if I am understanding this correctly, I can use a shift register (in my wall boxes or Arduino box) that have a 74HC14 chip connected to the switches and one to the Arduino to amplify the signal?
Switch panel 1 --- 74HC14 --- Cat 5 --- 74HC14 --- Arduino?
Well, the shift register goes between the actual switches and the gates in the 74HC14 which is simply to "clean up" at each end the signals going along the Cat 5, but that's pretty much how it is. Of course the gates in the 74HC14 are inverters, but with one at each end of the wire, the signal is inverted twice and ends up the right way up.
Note that if you really want robust (and faster) communication, instead of the 74HC14, you use RS-485 driver (and receiver) chips to send each signal to a pair of wires in the Cat 5; for three signals you would use three such pairs plus the fourth pair for power.
Just out of curiosity, how would you make the Pro Mini talk to the main board? I looked up the price and Sparkfun is carrying them for around 9,95$.
Well, the eBay suppliers are a quarter of that price, tricky to figure out whether the quality differs.
That is where the real RS-485 comes in, if you have an actual MCU at each location, it encodes the switch combinations into a "packet" which is sent serially down a single channel (two wires). This means that all the stations are connected to the same two wire bus, and use a packet sending protocol (like the Internet) which includes such features as checking that the data received was that which was sent, making sure that it was received at all, handling "collisions" where two stations attempt to send data at the exact same time and so on.
There are three basic solutions or approaches to this multi-station problem. The packet network using a single communication channel is one; a "chain" as previously described where each station passes data on to the next which simply passes it on until it reaches the central station, is another and the third is what you originally envisaged, each station connects to the central one which then has to have a lot of connections to handle them.