is possible? heterogeneous common bus input, common bus output protocol

Sorry that every now and then I these strange questions about communication.

TLDR:

  • I need to make a network of arduinos that will receive data, process it and send it to other output arduinos.
  • The destination and origin of the messages is given by wether they are plugged.
  • They must allow many inputs and outputs

I have been studying and struggling on how to make a kind of distributed processing environment using arduinos, and of course, the communication protocol is the hardest part. I tried to make my own protocol with a simpler scheme, but I learned that I would need a lot of low level knowledge on AVR and C programming that I don’t have. For the sake of prototype I am thinking to use ready made libraries, but it is not so easy yet:

I describe below what I am trying to do in terms of communication. But an image will make it way easier:

[see attached image]
Capture.PNG

I mainly need each arduino to have n possible data inputs, and n possible data outputs. The best would be that n is ‘virtually unlimited’, but it will be ok for now if n is just more than 1, in which case I could do a module that forks the stream, effectively making a virtually unlimited amount of outputs (at cost of speed), but that is not well desired.

*The data outputs can all be the same, as the output value will always be the same to all the “children”. However, some “children” may be busy, meaning that perhaps it is easier to keep outputs independent.

The devices are dinamically plugged and unplugged, but they can be already powered then, avoiding hot plugging problems (thanks, grumpy_mike http://forum.arduino.cc/index.php?topic=476989.msg3258658#msg3258658 )

I also need the “network” to be “heterogeneous”, meaning that hopefully it doesn’t need one single master, specially because there are many isolated communication nodes. If I am a node, I know what I am sending to my children node, but I have no idea of what that “children” node is sending to my “grandchildren” node.

If interested, an example application is the one in this video (https://www.youtube.com/watch?v=YdsDUqan77s), only that in hardware, with all the changes that it implies to the idea.

So, a thought solution was that each device (arduino) would use the I2C protocol twice. Each device is both a slave and a master: it is a master to all the devices down, and a slave to all the devices up. Would this work? or will I be wasting my time, as I have been quite some days already?
I know that mega readily does that, but I think it is on the expensive side. Whilst I will try to avoid it, if it is too much of a hassle, I can go for it.
if so, what is the best solution?
Am I dreaming of making a rocket to mars?

[note: I posted the same question in the AVR freaks forum http://www.avrfreaks.net/forum/heterogeneous-common-bus-input-common-bus-output-protocol]

If you want to send message from and to arbitrary nodes of your network, every node must have a unique address. Then you can connect all nodes to a singe I2C, Ethernet or any other bus.

Otherwise you need hardware that can listen to a number of channels at the same time, like the four Serial channels of a Mega. Or each node must supply signals indicating ready-to-send/receive, so that a next-level controller can serve the individual channels.

Did you have some particular form of 'data' in mind? On/Off signal? Fixed-width pulses? Variable width pulses? Analog voltage? Asynchronous serial data? Synchronous serial data?

In my personal experience, your biggest challenge will be in "error control and recovery". If you have determined a message from any device is corrupted and needs to be resent, how does that work. In same vein, if you send a message to some anonymous device, how will you know it actually exists and received the message as you sent it?

Paul

Designing a network from scratch is absurdly hard. The number of things that you have to think about is almost unimaginable. It's as hard to do as outer space is big.

Say the master calls A. A responds. But there's a glitch on the line and the master doesn't hear all of A's response. The master is expecting a response of a certain size so it waits for A to finish. A thinks it has already finished and doesn't have anything more to send. The master isn't calling A because it already got some of the response. Meanwhile B has a really urgent message to send but it knows the master is talking to A and it isn't allowed to say anything while A has the control of the line. But it saw the complete response from A and knows that the line is now vacant so it is allowed to send its urgent message. The master hears this but it thinks it's valid data coming from A, not B's emergency signal so then it has a long message that it thinks is from A and it must have some method of checking that this is really what A meant to say and maybe it has to ask A to repeat the response, except B is still trying to get a response to its emergency message...

And that's not even the most complex thing that can go wrong!

But all this hard work has been done for you, many times even. There's a lot of different network protocols out there which have been thoroughly planned to cover every possible sequence of events. How do you choose?

Start with the distance. How far do you need to communicate? These days, radio transmitters cost less than 20 feet of wire, so radio is a very cheap way to do it. (Look at WiFi, nobody plugs their laptops into the network any more.) I2C is great but distances are very limited.

Then look at the expected interference. Are you communicating between devices in a forest or in an industrial plant? (Animal interference is a nasty one. Rats eat cables.)

At some point, you will need to consider practical concerns like how do you power these devices and if they need waterproof plugs or something. That can affect which network you choose.

Do you need true peer-to-peer networking or can you have a master which addresses the slaves individually? For Arduino-type projects, master-slave is significantly easier.

Do you need the slaves to be "smart" where they know what kind of sensor they are and can describe that to the master or does the master know that slave 1 is always the humidity sensor? (Smart is very hard to do. Do that later as a secondary layer on the network. It shouldn't be a service provided by the basic communication layer.)

johnwasser: Did you have some particular form of 'data' in mind? On/Off signal? Fixed-width pulses? Variable width pulses? Analog voltage? Asynchronous serial data? Synchronous serial data?

Sorry that I missed that important detail. I plan to send short words. Perhaps three or four bytes, at a high rate.

MorganS: all this hard work has been done for you, many times even. There's a lot of different network protocols out there which have been thoroughly planned to cover every possible sequence of events. How do you choose?

Paul_KD7HB: In my personal experience, your biggest challenge will be in "error control and recovery". If you have determined a message from any device is corrupted and needs to be resent, how does that work.

So yes, my constrain is on not having too many lines, and being able to unplug. I don't plan to cover distances, and I don't need it to be 100% error proof, but just fast. I don't even need acknowledgement. I totally realize that I should use some other protocol. So that's why I thought that I2C was a good idea; (except that it has ack) but in this case, I would need to make each node a master downstream, and a slave upstream. Do you think it is possible?

joaquin0:
Sorry that I missed that important detail. I plan to send short words. Perhaps three or four bytes, at a high rate.

And you also missed the important detail of what you mean by ‘high rate’

srnet: And you also missed the important detail of what you mean by 'high rate'

I have worked a lot with MIDI. Midi tends to get a bit too much latency/ clogging too easily, but is about to be fast enough. I don't know enough as to tell a number confidently.

I would not be quite as pessimistic as @MorganS in Reply #4. Yes, all those things need to be dealt with but I reckon they could be with a fairly simple system that uses a fixed message length.

What I am struggling with, when I look at the diagram in the Original Post, is how the devices can be connected together. It is all very well to join them with squiggly lines in a diagram on a page - but the real world needs something a bit more substantial.

Are you aware of the Mesh library for nRF24L01+ wireless transceivers. I have not used it but it seems to have the characteristics you are looking for.

...R

I2C can do what you want. It allows for multiple masters and collision handling (bus arbitration). The achievable speed depends on the extent (cable length) of the network and number of collisions, and the number of nodes is restricted to 120 (7 bit addresses). These restrictions can be reduced by hardware and firmware, that is not included in the Arduino hardware (multiplexers, line drivers...).

All you need for an exploration are multiple Arduino boards (or other I2C devices), which you'll need anyhow. Find out more yourself...

Robin2: What I am struggling with, when I look at the diagram in the Original Post, is how the devices can be connected together. It is all very well to join them with squiggly lines in a diagram on a page - but the real world needs something a bit more substantial.

Are you aware of the Mesh library for nRF24L01+ wireless transceivers. I have not used it but it seems to have the characteristics you are looking for.

...R

Yep. Let's imagine for now that they will be RCA cables. I have not designed that yet, because the protocol may require different wiring and so. So the idea is that I can provide the object with many data outputs in the simplest possible way (i.e. just adding RCA sockets). And yes, data length can be fixed.

I was not aware of the RF24 protocol or the library whatsoever. I like that it is designed to connect and disconnect automatically from the devices. The problem is that it works with a single network, whereas I need each network to be isolated; meaning that one module needs not to be aware of what other modules are sending, only if they are physically plugged. If I were to use RF24, I would need to change the concept of what I am doing, in which case, the I2C library would also be useful.

DrDiettrich:
I2C can do what you want. It allows for multiple masters and collision handling (bus arbitration).

Yes, but I2C leaves me with two questions:

  • can an arduino handle two I2C networks? because in such a case, it would need to have one input network in which he is a slave, and one output network, in which he is a master.

  • can a node be removed without blocking the net or causing problems? I understand that I2C needs ACK once a message has been sent. Removing an element would result in a node going to timeout repeatedly, unless there is a broadcast function that would not require ACK from the slave nodes.

perhaps if all modules were to share one same I2C network, then what you would plug is just a cable that lets one module to “poke” the other module and tell it that the message is for him, as a way for elements to let know that they are plugged together. The cable patching becomes some sort of illusion trick, where the user expects data to pass through the cable, but actually data goes through the common bus. I find this solution a bit unelegant though.

Only few controllers (Due?) have two I2C buses.

Every single byte is acknowledged, in hardware. If not, the transmission terminates with an error. A newly plugged node can broadcast a message, telling its address to all other nodes.

It would sort me out if there was a cheap serial buffer chip. Something that I can have one per input. And once any serial message arrives, it stores it until the arduino has time to read it. If so, it would be even possible to put a multiplexor, and read these one by one. Does such a thing exist? I have been searching, but so far have found nothing.

DrDiettrich

DrDiettrich: Only few controllers (Due?) have two I2C buses.

Every single byte is acknowledged, in hardware. If not, the transmission terminates with an error. A newly plugged node can broadcast a message, telling its address to all other nodes.

I thought that you could make a serial bus out from a couple of digital pins, in a SoftwareSerial fashion. Is it not?

joaquin0: Yep. Let's imagine for now that they will be RCA cables. I have not designed that yet, because the protocol may require different wiring and so. So the idea is that I can provide the object with many data outputs in the simplest possible way (i.e. just adding RCA sockets).

Sorry, but I can't imagine what you have in mind - maybe you can make a simple drawing.

  • By "RCA cable" do you mean a cable with a single core and a co-ax braid?
  • What does the cable connect to on an Arduino?
  • Assuming there is a cable connected to an Arduino what does the other end of the cable connect to?

I was not aware of the RF24 protocol or the library whatsoever. I like that it is designed to connect and disconnect automatically from the devices. The problem is that it works with a single network, whereas I need each network to be isolated;

I don't understand this business of "each network to be isolated" Do you mean that you want (say) three separate systems with a few Arduinos on each and the Arduinos on one network are prevented from communicating with the Arduinos on a different network? If so I can't see why you could not do that with nRF24 transceivers - just operate each network on a different channel (frequency)

...R

joaquin0: I thought that you could make a serial bus out from a couple of digital pins, in a SoftwareSerial fashion. Is it not?

A software solution can not normally serve (even not monitor) multiple channels at the same time.

Robin2:
I don’t understand this business of “each network to be isolated”

Thaks for the questions. Sometimes I am not that good contextualizing. I gladly made a drawing of an example application. The picture shows an absurd application. The real application is to make more complex operations on midi-like messages.

[picture]
IMG_20170527_224528__.jpg

Imagine that you generate an initial number by pressing a button in the element that is drawn with a finger by it. This number will propagate down until it reaches a dead end, or it feeds back. As you can see, each module is only aware of his inputs and his outputs, in other words; each module is connected to two isolated networks.

I imagine this physically like an eurorack setup, or like a bunch of small tabletop devices. Both cases imply that they relate to each other by wiring. Wiring is one of the user interfaces for this thing.

I have two non-technical questions for you.

  1. Do you have space and finances to have an individual PC connected to each Arduino while you are developing and testing your system?

  2. Do you have time to develop your system? I did two different projects with similar technical complications in my life and each one tool 18 months to complete. That was working 8 hours+ per day, 5 or more days per week.

Paul

joaquin0: propagate down until it reaches a dead end, or it feeds back. As you can see, each module is only aware of his inputs and his outputs, in other words; each module is connected to two isolated networks.

Which direction is "down"? Do the messages only move in one direction, that is, are the arrows fixed to the modules? Or, does a message coming in on any port get forwarded to all other ports?

You have drawn several modules there with 3 arrows in or out. That is a conflict with your "two networks" statement.

If it's truly unidirectional and you only want to plug in RCA cables, then that can work quite nicely. The thing is, the RCA cable isn't usually set up as a unidirectional cable. It has the same plug on both ends. So there's no physical prevention to stop you plugging an input into an input and wondering why it doesn't work. If you can get chassis-mount RCA plugs and plug-to-socket "extension" cables, then you will have a good system. Or choose another type of connector, like outputs are always 6.5mm sockets and inputs are always RCA sockets.