Don't forget Charlieplexing.
will make the circuit more complex and wiring even more difficult.. and if 1 led or so might fail it could be very hard debug... Anyways nice thought though... never thought about it myself 
I've thought from the beginning that using one pin for input and output would not work.
So I guess this a dead end then... 
Really hoped to get it working this way... Back to the usual method now... Thank you everyone for your valuable inputs