A bit of context: I am making a sort of digital modular toy, where the player can connect modules freely. Each module does a function such as storing a variable or making a math operation over the value that receives. A bit like a physical programming language. These modules have many inputs, and one output. They can be connected in any way, so what happens with the system is defined by how these things are connected together. I expect it to work like a modular synthesizer, but using digital signals. So each module doesn't care of the ID or function of the parent, but only about what numbers he sends. They will be probably fixed 3 or 4 byte bessages.
The approach I was thinking, is that each input is a register in a multiplexor. The multiplexor common input is the AVR's RX pin. While in idle mode, each module is scanning his multiplexor inputs, waiting for an incoming message flag.
When a module has a message to communicate it's 'child' module, it sets a flag (raises or lowers the voltage) in it's own TX. This child AVR is scanning the multiplexor, and when it finds that in a port, the level is raised, it raises it's TX pin aswell, letting it's parent know that he is listening, and so this parent AVR can start sending his serial.
For this, I need that the AVR will be switching the use of a same pin for both, digital reading of the flag, and serial reading.
- Will this work, or will it get hairy?
- is this a proper way of doing this kind of custom communication protocols?
- Where does one learn the proper way of making MCU's commuinicate?
(Please don't suggest using I2C, because the data is very short, the bus is not shared, I don't want ACK and units are not aware of other unit's Id's as they are dynamically connected and disconnected)
Thanks for reading, regardless of wether you have an answer 
they are dynamically connected and disconnected
This means they are "hot plugged". Do you know that for correct hot plugging you have to connect the ground & power first followed by the signals, otherwise the chips could latch. Look at a USB plug from something like a memory stick, you will see the power and ground connectors are slightly longer meaning they connect first.
I am not sure I understand your proposed solution but using the RX pin will also interfere with serial communications and the bootloader on power up, so I would put it in the "hairy" category. Using an Arduino with an additional serial port would help. Like the Mega, Leonardo or Micro.
And there are two things I completely ignored and seemed important: "hot plugging" will be a usefull term to search and learn, and I had no idea whatsoever that you needed to connect ground first. This kind of things are what make me waste several days each time I want to communicate two devices.
Thanks!
Joaquin:
The approach I was thinking, is that each input is a register in a multiplexor. The multiplexor common input is the AVR's RX pin. While in idle mode, each module is scanning his multiplexor inputs, waiting for an incoming message flag.
When a module has a message to communicate it's 'child' module, it sets a flag (raises or lowers the voltage) in it's own TX. This child AVR is scanning the multiplexor, and when it finds that in a port, the level is raised, it raises it's TX pin aswell, letting it's parent know that he is listening, and so this parent AVR can start sending his serial.
Two thoughts come to my mind ...
{A} Use 4 pins - two for the recognition that you mention and two for Tx and Rx. Plus power and GND of course.
{B} Use the sending of messages as the indicator of connection. For example the master is listening all the time and the slave sends a "hello" message every (say) 500 millisecs when it is connected. If the master receives a "hello" message it responds and starts a proper conversation. If the master does not receive a "hello" message (or anything else) after a 500msec (or maybe 1000msec) interval it knows there is nothing connected.
...R
Thanks, Robin2. It is really helpful to have ideas from others
Perhaps I should think that the units are powered by a common power bus before anything, like the eurorack modules. In this way I remove the potential latch problem. That way I get to use simpler patching cables.
It shouldn't be necessary for the master to send constant keepalives, as it could be listening for potential signals anyway. I agree with you that any connection needs both directions regardless of the intended real direction, in order to send permission to communicate and so on.