Easy, simultaneous development on several Arduinos (8+, 30+?)

Projects Discussion and Showcase Device Hacking
1

Writing here to discuss a project idea. Wondering:

  • does it already exist?
  • is it feasible?
  • would it be useful (and/or what would make it more useful)

The idea

A device with microcontroller (maybe mega2560) connected to a support circuit which allows some kind of switching between a large number of heterogous arduino dev boards (each serving heterogenous purposes).

Use case example

Example project 1: a fancy robot having many microcontroller subsystems

This project might have separate microcontrollers for each leg and for each arm, separate controllers for various sensor banks, and other controllers for coordination

Example 2: a boat automation system

This project might have separate microcontrollers associated with engines, anchor, plumbing, lighting, situational awareness sensors, etc

In both of these examples, I can imagine wishing to have a reliable and simple way to do things like: identify or update the firmware running on a particular subsystem, view recent Serial output (serving as an inexpensive and simple troubleshooting log), possibly even arbitrarily connect a debugger to an individual controller

Design principle

The device would act as a proxy for development operations, as a logger, and would help with the organizing and identification of subsystem names/roles and allow reliable connection to each. The device would keep track of each connected subsystem microcontroller and call that a "node". The 'supersystem' consisting of "the device" plus all of its connected nodes would be relatively static, but it should not be terribly difficult to add a new node (e.g. if you've just decided to add a new subsystem into your robot).

  • nodes are associated with specific microcontrollers via some reliable mechanism, either mechanical (each node is consistently plugged into 'device' using the same port) or via auto-detection of some unique property (???) or possibly using some kind of manual association procedure (e.g. when a a new node is connected, you immediately configure that node's params, such as name, role, etc.). The association mechanism must be able to survive power resets on subsystems (obviously)

  • "the device" can keep track of a library of firmware images (stored on external mass storage) and automatically orchestrate these being accurately flashed onto nodes as appropriate. Firmwares can be addressed individually to nodes or perhaps to node type (role). Multiple versions can exist at once and it should be easy to instruct the device to rollback to previous versions (e.g. "leg software rev 54 isn't working, switch back to rev 53")

  • Serial output from each node is continuously and reliably recorded to mass storage. Each output item is (obviously) associated correctly with its node/role and as a bonus, gets a timestamp prepended to each line. Live access can be gained at any time, allowing the user to easily peek at the current output from any particular node (e.g. "right now, the left leg controller is reporting joint angles: 53, 39, and 62")

Design details

The software

Depending on the feature set (do we try to allow remote access? provide some kind of FTP or web server interface?) I am confident that the high level software should not prove too great a task. It will consist of the orchestration of a few things: mass storage, a few buttons, a simple display and communication 'mux' system.

The hardware

I've brainstormed a few different approaches to take here:

  1. nodes will connect via USB. My device will be a USB host connected to a large USB hub which will in turn connect individually to each node device. Total price: one strong microcontroller + one good hub + one USB cable per node

  2. nodes connected via "a couple of wires". Wires are "ICSP"? Is this some kind of rs232 connector maybe? In this version, (fairly complicated) support circuitry is built which provides for: UART, rx and tx buffering, appropriate signaling for invoking dev board reset (ie RTS) and this circuitry is repeated and 'mux'd together somehow as many times as needed (eg 8 times, 30 times, etc). Via an external SRAM or something and some kind of DMA-type process, the external circuitry pushes RX data into large buffers and possibly even straight to mass storage. This takes burden off of "the device" controller, allowing it to keep up with a large number of nodes. I've done some research into ICs and chips on digikey - this approach looks fun but crazy, but probably doable....
    Total cost: one fast microcontroller, one rs232 cable per node, support circuitry UART,etc (maybe $2-10) per node

  3. use mega2560, which provides 4 uarts. Chain multiple devices together to get desired total number of nodes. e.g. to support 30 nodes, I'd implement "the device" through the coordinated efforts of 30 / 4 = 8 mega2560s.. Total cost: number of nodes / 4 * $25

EDIT, adding #4.:

  1. give each 'node' board some kind of inexpensive bus-oriented adapter other than USB such as CAN bus or RS485. Then, put my "magic device" on that bus too and handle messages from there. Similar to approach #1 but favoring a less-complicated and less-expensive bus protocol instead of USB.

If you've read this far, thank you for your time and interest!

I'm new to the forum (and pretty new to Arduino in general). I apologize in advance for any ideas that might be naive or stupid. :slight_smile:

2

Welcome to the forum

The whole thing sounds over complicated. 8 slaves and a master means that there are 9 different sketches to write, maintain and keep compatible for a start

When a single microcontroller is is not enough for a project then the best answer is rarely 2 microcontrollers, let alone more

3

Thanks for the feedback, Heli Bob. I still think the idea of using multiple arduinos is sometimes appropriate:

  • home automation - don't you often need the controller to be relatively close to the sensor? doesn't multiple displays, multiple keypads all throughout the house imply that there will be a large number of microcontrollers?

  • boat automation - again, lots of sensors and lots of subsystems serving wildly different purposes (engine, anchor, nav, galley, etc)

Also, I have watched youtube project videos featuring even fairly simple robots using multiple microcontrollers. In one video, they do describe the concept as being a bit controversial.

I think it is noteworthy that there is a similar debate in the world of software systems. Some people prefer implementations using 'microservices', which is really the software analog of the 'lots and lots of different microcontrollers' idea

Lastly, I agree that the idea of coordinating development of, as you said in your comment, "9 different sketches" is cumbersome. But, if you have a magic device that coordinates it all (as described in my post), I think it becomes manageable and, in the context of modern software development practices such as Continuous Integration, etc pretty normal and possibly even preferable.

Those are just some of my system-level thoughts on how and why the proposed "magic device" might be appropriate in some situations.

4

Nothing like that currently exists for "Arduino", which isn't even well defined.

New boards are coming out all the time, with different processors operating at different speeds and voltages, with different compilers, linkers and Arduino "cores" for each one.

Think of Arduino as a learning platform, not something for which to implement any sort of interconnectivity standard .

5

Thanks jremington

I think you make a good point about the loose definition of Arduino (vs the broader field of embedded programming) and about the dynamic nature of the hardware landscape.

I'm now heavily favoring taking a new approach comprised of:

  • design and build an adapter module, which can be attached to arduino nodes (and other types of nodes I guess). The adapter could be based on the AT90CAN32 AVR chip. It would speak CANBus on one side and on the other side it could speak serial (actually, could do two serials per AT90CAN32 since they have two uarts).

  • implement my central controller concept separately and as effectively just another station on the CANBus

The end result would be a simple system for providing sort of "Meta Control" over some set of (8, 30+, etc) arduinos. Each Arduino needing to be attached to the CANBus (forming sort of a control backplane) via an adapter module. The adapter module being some compact build involving some chip (such as the AT90CAN32). That puts me at about $10-15 per node, which is not too bad. On the plus side, it provides bomb-proof connectivity thanks to the well-established CANbus standard.

Thoughts?

6

CAN-bus is already used with Arduino, and libraries and adapter modules are available. From time to time posts crop up on this forum, but the interest does not seem overwhelming.

You might take a look at what others have done and see if that can be improved. One example: Arduino CAN Examples | Adafruit Feather M4 CAN Express | Adafruit Learning System

7

Thanks for the references.

But, does the following make sense?:

Keep in mind, though, that my main goal is to enable reset and reprogram of a target MCU (in addition to relaying serial IO for monitoring purposes). To me, this means I need to speak CANBus (or whatever system this lands on) in a domain outside the MCU being targeted. This means CANBus must be implemented in a secondary MCU or in some dedicated driver IC. This means, I must pay for a second Arduino (or other MCU) which would then run the CAN-bus libraries you linked to.

If I'm already getting a dedicated MCU, my new question is whether it is worth it to go for an MCU with a tried and true, on chip implemented, and "Certified" (per AVR's datasheet for that special chip) implementation...

I suppose my other option is to run CANBUS on the target MCU Arduino and implement some fancy scheme for self-reprogramming, but I find that idea less appealing for a few reasons:

  • processing CANbus messaging deprives MCU of instruction cycles it could use for real work
  • suffers from chicken and egg problem where I can only upgrade the running firmware if the firmware that is running is working (ie the whole system falls over and is useless if the chip's software has 'crashed' in any way
  • I think self-reprogramming schemes will necessarily also cut my total maximum firmware size roughly in half
8

Usually "all things to all people" architectures are not practical. The type of interface depends on various things like distance, physical connection. There is always some overhead with a generic solution, so implementing it on small devices becomes a problem. Also I would not use "the way Arduinos" work as a starting point.

Typical modern cars have dozens of computing devices, reaching into the 100's. They use several types of bus, depending on requirements. Remote programming is probably straightforward, of course you will need a bootloader that supports the bus. So most things are possible, it is just a question of writing code. I would suggest restricting the goals though to make it achievable.

I doubt whatever you come up with be useful for other people though.

9

I skimmed this (it's a lot of walls of text!) and I still don't understand one fundamental thing.
What problem are you trying to solve for which there isn't already multiple commonly-available solutions?

1 Like
10

I'll try again:

Suppose you're integrating several MCUs together on a robot. Many controllers are buried deep within. You want to troubleshoot all the different MCUs while the robot is being tested as a system. Specifically, you want to: see troubleshooting info they've written to serial AND you want to be able to easily flash in new code to try adjusting things. But you don't want to have to take everything apart.

My proposal:

Put all those MCUs onto a bus which is capable of resetting and reflashing any one of those controllers at any time as needed. This bus is dedicated to this type of "meta control" and is just for

My question:

how is this done at present? do other people find this to be a pain-point?

Does that clarify at all, @cedarlakeinstruments?

11

I've been following this out of interest but not replying.

I've not built a robot. I think if I did and had the problem you describe I'd make sure that either: The MCUs were accessible, or if they were not then make sure they had a USB cable brought out to somewhere accessible where I could connect to them easily.

12

Arduino uses USB serial for debug and programming, but that is really not typical. Commercial systems would use JTAG for programming and debug. For development, the JTAG connection would be exposed somewhere. Otherwise, in systems where you have multiple MCUs, they are probably already connected on a bus. So we just use that bus for management functions.

Adding an extra management plane is usually unnecessary, adding cost and complexity.

13

I notice two important areas you fail to address. The first is feedback to the central controller. The second is any safety consideration to is any subsystem fails it does not destroy the whole project or injure someone.

14

Thanks for the feedback, Paul!

I was thinking about a fuse on the power line (if any). diodes on power and ground and optoisolators on the signal lines. maybe a fuse on ground too (to eliminate the possibility of any big discharge there). Would that help? I'm brand new to all this, so please let me know what kinds of things might be wrong with those solutions.

15

I think what you are saying makes sense, @PerryBebbington . Thanks for the perspective from the pro-arena, @bobcousins.

I think I'm looking to add the concept of the small development-oriented helper MCU on top of those solutions. This is the "magic device" I referenced earlier, which brings features I think would be useful: 1) continuous logging from all the MCUs and 2) the ability to easily reflash anything at will without the need to bring over the laptop (thanks to local storage of multiple versions of firmware and the ability to remotely load new ones in from your laptop via the onboard web server.

In the software world recently, microservices are usually combined with precisely those services: remote, persistent, continuous logging, and automatic deployment of new versions. I've found that development paradigm is tremendously helpful.

I'm wondering whether having those capabilities would also be helpful in the DIY microcontroller world. I think they would be (if I ever get my boat project started...).

16

Fuses are there to stop fires in the wiring.

17

Given the notoriously portable nature of laptops, that seems like an unlikely benefit.

It really seems like you are trying to address an issue that no one really has.

18

For sure! And I don't like fires. Especially in my boat :slight_smile:

But, yeah, I think I know where you're coming from. Some people think fuses are to protect people. And, far too many people, like to put fuses on the wrong end of a long wire! :slight_smile:

19

Laptops are indeed notoriously portable. You've got me there!

However, I know for sure that I (at least) will not want to need to carry my laptop down into the engine compartment, the head, the chain locker or out on the slippery foredeck to observe logs or change firmware.

There is at least one person with this issue (me), but perhaps you are right, Bob - this might not have broader usefulness.

20

Yes it clarifies it.

Not a robot, but I spent many years developing for a complex medical device that used approximately 50 stepper motors and about a dozen other serially-connected devices on various controller cards. For that system, having to update firmware was very rare because the controllers were well debugged and most of the complexity was on the host which was an x86 64-bit backplane PC. I suspect most robotic distributed CPU systems follow a similar pattern.

Next, I've come across something closer to the problem you describe in what were essentially distributed data aggregators spread around a building. Each "aggregator" module had 4 CPUs and they were connected together by an Ethernet network of generally a handful to a few dozen modules. For that case we did software updates over Ethernet. In some cases the software updates being done from a different country altogether.

Cars typically do their firmware updates over CAN, but that's not something I can claim to have worked on.

I don't know that there is a "standard" framework for doing distributed firmware updates, but many of the tool vendors offer their own solutions already. Can't speak to how useful or performant they are since I haven't used them.