Multi-node CAN bus: a progress report

Using Arduino Project Guidance
1

SUMMARY

This is a progress report on an "open" CAN bus wheelchair controller system that I've been working on. Luckily I am now retired so I am able to devote a lot of uninterrupted time to a project that is much more involved than anything I have done before, with the exception of my daughter Rachele's head switch and gaze operated multilingual communications/computer control system which only reached its current state after years of work.

Having been involved with wheelchair design for over 20 years, I am well aware of the safety issues involved; after all, a wheelchair user can not "bail out" if something goes awry. Indeed, creating a "bullet proof" system is a major part of the work involved. For a geneticist working on a project that really needs the skills of an ME, an EE and a programmer this is decidedly non-trivial.

One of the satisfying, and at the same time worrisome, aspects of this is that I learn something new just about every day. For example, common practice is to protect a semiconductor switching circuit from relay coil kickback by putting a reverse-biased diode across the coil. This indeed will protect the driver, and it is a good way to do things when dealing with signal-level relays (or AC relays in general), it is, however, a relay-killer if used on high current DC relays. I only discovered this when a footnote on a spec sheet led me to these Tyco application notes. If you are dealing with controlling more than mA currents, they are a must read.

http://relays.te.com/appnotes/app_pdfs/13c3311.pdf

http://relays.te.com/appnotes/app_pdfs/13c3264.pdf

So what's worrisome about learning all these new things? What's worrisome is the question: What other equally critical factors exist that I should know about that, because of my ignorance, I don't even realize that I should study?

In the end, I expect to need a lot of testing at high power in a high noise environment before the design is really "final". And that's a problem for me: pushing everything to its limits requires equipment, money and time I don't (and won't ever) have, and I've yet to decide what level of risk I'm willing to tolerate in a system that's not been "tested-to-destruction". Hence, in both the hardware and software I'm trying to use a "belt-and-suspenders", fail-safe and safe-fail, approach and that's why the more people who are willing to invest some time in double-checking and questioning what I've done the more comfortable I will feel.

At this point, I have a three-node net running at 1M bits per second without dropouts (compared to the usual WC controller rate of 100k bps or so). Transmission rate will be set lower on the chair after testing at 1M to give a nice safety margin. During testing, a PC will also be connected to the motor controller to start its script each time it's powered up. Once I'm sure that the script doesn't have any bad bugs, it can be set to autorun on startup. The PC will have NO role in actual operation - no PC operating system is to be trusted EVER with driving a wheelchair!

The full description of the project is far longer than the forum allows, so I've attached it as a pdf. There is also a zip file of the hardware (WC CAN circuits.zip) with schematic and pcb designs as pdf and DesignSpark files, and a zip file of the program files (WC CAN programs.zip).

PROJECT DESCRIPTION: HARDWARE
CAN nodes:

The system will have four CAN bus nodes, though more could be added if other functionality is needed:

(1) the MOTOR CONTROLLER node that handles the high current needs of the motors and some medium-current digital outputs. I am designing around the characteristics of the Roboteq HDC2450, but relatively little work would be needed to adapt this to other controllers. Some external hardware is also needed: high current safety disconnect relay, SPDT reed relay for power to the Roboteq's microprocessor, pre-charge resistor, regeneration bypass diode (in case the safety relay opens), voltage clamping on the motors (if not built in to the motors themselves), relays or semiconductor H-bridges for actuators etc.

(2) the MASTER node that collects information from an analog joystick and/or switches, and does the arithmetic and logic to turn that into CAN messages to the Roboteq for controlling the motors, brakes etc. Because the Roboteq does not treat CAN the same way as it treats analog, pulse or serial input, all of the calculations for scaling the joystick, setting dead band, implementing a speed pot (if desired), applying different degrees of exponential curving and so on must be done here. This is actually similar to Dynamic DX/DX2 systems in which the power module only has motor programming, and the master computer is in the remote. Of all the driving calculations needed, only mixing and motor compensation are handled in the Roboteq node. MASTER also traps errors, collects information from the Roboteq and Aux nodes, and sends this information on to the DISPLAY node. A toggle switch is used to go between joystick and switch input, and momentary contact switches are read to give forward, reverse, left, right, mode (driving, seat, lights), power On (or Wake)/Off, and STOP.

(3) the AUX node. Although the Roboteq has a number of 1A digital outputs, there are not actually enough to do everything one might want to do: separate left and right brakes, seat actuators (1, 2 or 3), lights. Moreover, I don't want to have separate On/Off switches for different modules so we need some way to control power to the Roboteq via the CAN bus. These functions will be handled by AUX node. AUX can also handle a shock sensor &/or other stability sensor (such as I now have on Rachi's chair), and speed slow-down &/or inhibit sensors on the seat actuators. (This is where one could add gyros, compass, even a full inertial navigation system if one wanted too - I don't, but a Nano has enough guts for all of this.)

(4) the DISPLAY node. This node receives messages from MASTER to display them on a small TFT screen and to log them on a micro SD card. Graphics will show which way to move the joystick for different seat movements or light settings, an indicator of whether in joystick or switch-driving mode, present voltage and Amp Hours consumed since last charge.

A jpg of my breadboard is attached; for design details see "WC CAN description a.pdf" and "WC CAN circuits.zip".

SOFTWARE

CAN protocol:

A standard CAN frame consists of an identifier (11 bits) and up to 8 bytes of data. The CAN standard includes a number of features, both in how things are coded and in the physical layer (CRC check, bit-stuffing timing check, repeated transmission until receipt is acknowledged) to make sure that messages arrive intact. Because a wheelchair is a life-critical device for a user who may not be able to "bail out" if things go awry, my programming adds a couple further layers of protection. My messages contain at most 4 bytes of data and the other space is used for the bit-wise complements of these. A receiving node first does the built-in CAN checks and filters for proper ID before putting it in a message buffer (the MC 2151 has two buffers with separate masking/filtering). My (AUX, DISPLAY or Roboteq script) program then compares the data and their complements to make sure that the information is correct and then sends a confirmation message back to MASTER. If the original message was an information request (e.g. battery amps), the confirmation is the requested information itself. If the original message was a control or configuration command (such as a motor command), the confirmation message is the same data (and complements thereof) with the ID changed to show that it is the confirmation. MASTER again does the data/complement check, traps any errors and does not proceed on unless a proper confirmation is received.

For more complete description of the protocol's design and the functions of each node, see "WC CAN description a.pdf" and for implementation see "WC CAN programs.zip".

If you want to see the details of how these things have been done, I'm afraid you'll have to study the schematics, pcbs and read through the code. I hope that at least some of you will. This is my first "deep" foray into embedded programming and I'm sure there are many improvements that can be made.

WC CAN breadboard.jpg
WC CAN breadboard.jpg1024×773 146 KB

WC CAN description a.pdf (342 KB)

WC CAN circuits.zip (550 KB)

WC CAN programs.zip (37.9 KB)

2

Dear LROBBINS,

For the moment, here in Brest, we are working on assisted sail-boat steering for people dependent on electric wheel-chairs. Open-source electronics for wheel chairs is technically speaking very close and is one of my long-term ideas/projects, so I'm very interested. And the way you are doing it, seems not far from what we do here.

Your CAN hardware, for example, is very similar to our Arduino compatible CANinterfacer.

And just now, we are searching for an appropriate protocol to work just above the raw CAN protocol. We have difficulties to find an open-source protocol for this robotics-like application, and one that is simple enough to work with Arduino-IDE programmed 8-bit AVR processors. And as you, we are just about to abandon search for an existing protocol and to create our own, open-source one.

  • As it looks now, our protocol will have provisions for synchronization between nodes, a sort of global system clock, which will permit things like respecting real-time constraints of distributed control loops, time-stamping, and collecting distributed sensor data measured at the same moment. (For example : if you want to know if the wheel chair steer left or right while braking, you must be sure to make a speed measurement on each wheel at the same moment.)
  • We think introducing a protocol mechanisms to create CAN-bus wide variables. These variables would contain sensor data reflecting the state of the system (movement, speed, actuator-positions) and can be used too for transmitting control commands (motor speed, actuator-position, brake-state...)
    (Both mechanisms used together can be used to detect node failure, think "timestamped heartbeat" for each node.)

And, you did it already and we did not think enough about it, for our application special measures about data integrity have to be include in the protocol. You could take a look at Cyclic Redundancy Check, which very broadly used because it is simpler to implement, takes less bandwidth and detects far more errors than a simple double transmission of complement data. One step further would be usage of Error correcting codes, which can detect and correct errors in certain (very impressive) limits, but implementation of these correcting codes is perhaps too processor intensive for 8-bit micro-controllers. Anyway, the CAN frame contains allready a CRC, transmission limited to single CAN frames,should be sufficiently protected.

We can probably do some stuff together.
Thank you for having shared your work,
Bernt

3

So this is where that on/off circuit was bound.

Looks like a worthwhile and interesting project.

Rob

4

Dear Bernt,

For the moment, here in Brest, we are working on assisted sail-boat steering for people dependent on electric wheel-chairs. Open-source electronics for wheel chairs is technically speaking very close and is one of my long-term ideas/projects, so I'm very interested. And the way you are doing it, seems not far from what we do here.

That was my impression when I read your post about FestivalCAN, but wanted to post a description of what I'm doing before contacting you.

Your CAN hardware, for example, is very similar to our Arduino compatible CANinterfacer.

Thank you for your link. I would not have been up to making an "all in one" board, but your hardware is indeed similar. One of the things mentioned there is something I wish I knew how to do: "reprogramming of the board through the CAN bus when using a previously installed special bootloader ". It surely would be nice to have access to the MCUs in all of the nodes by connecting to just one (even if separate programming of the undisclosed-type 32-bit MCU of the Roboteq motor controller would probably be unavoidable.)

And just now, we are searching for an appropriate protocol to work just above the raw CAN protocol. We have difficulties to find an open-source protocol for this robotics-like application, and one that is simple enough to work with Arduino-IDE programmed 8-bit AVR processors. And as you, we are just about to abandon search for an existing protocol and to create our own, open-source one.

I imagine that our thinking evolved similarly. The Dynamic DX control system on my daughter's chair serves her well, at least after an almost complete mechanical re-design done last year. A number of people who want more than current commercial chairs have to offer, however, have started experimenting with more potent motor controllers (mostly the Roboteq HDC2450, but at least one with a Roboteq brushless-motor controller). When Roboteq added CAN capability, I thought "Ah, now we can add the things that Roboteq lacks - a noise-immune joystick, non volatile memory, a display, single-cable wiring, and some added safety checks" and I thought it would be simple. After all, CAN output joysticks are now available. Then I discovered two things: (1) almost none of the signal shaping that the Roboteq offers for analog, serial and pulse inputs can be done for incoming CAN messages, (2) the CAN joysticks are OEM items and have to be "configured at the factory" to your CANbus - hardly possible for someone who wants to buy just 1 or 2. Hence my decision to see if I could scratch design a CAN network that would meet the high safety standards needed for wheelchair use. I'm still not sure that I can, but so far I haven't hit any insurmountable stumbling blocks.

The Roboteq offers CANopen and raw CAN capability. So, I thought, ahaa I'll go CANOpen, until I discovered that Roboteq's information is by no means sufficient for using it: they don't even tell me what "flavor" of CANopen they use, nor have they published the "dictionary" for their messages in a usable form. BTW - there are quite a few variants of CANopen, only some of which are really suitable for safety-critical systems. Next I looked at the cost of joining the CANopen consortium - thousands of Euros/year. There's nothing "open" about CANopen except the name. Lastly, I looked very briefly at open-source attempts at implementing CANopen, and decided that things are in so primitive a state that it would be years before something usable would come along. Hence my decision to develop my own basic CAN scheme that I could implement on an Arduino and that would be compatible with Roboteq's raw CAN.

  • As it looks now, our protocol will have provisions for synchronization between nodes, a sort of global system clock, which will permit things like respecting real-time constraints of distributed control loops, time-stamping, and collecting distributed sensor data measured at the same moment.

I would not know how to do this, nor would I have though of doing it!

(For example : if you want to know if the wheel chair steer left or right while braking, you must be sure to make a speed measurement on each wheel at the same moment.)

There are two kinds of wheelchair braking: dynamic (regenerative) braking via the motor controller, and solenoid-actuated parking brakes that engage only after the motors have come to a stop. Feedback such as you're talking about would only apply to the former, and the Roboteq already has that built in to its software. However, there are very few situations in which a closed-loop control is necessary, or even desirable, on a wheelchair. Tracking is available on most commercial systems (at a big cost) for those, like my daughter, who must drive with a switch array rather than a proportional joystick. Before the mechanical re-design/re-build of her chair, I had actually added optical caster angle sensors as a feedback element; it worked and was, at that time - 12 years ago, more easily done than adding encoders or gyros. When most needed, however, it really pooped out (just as the commercial ones do) because the motors didn't have the torque to do the correction without slowing the motor that was too fast. That's one of the problems of closed-loop control on present power chairs, and one of the reasons it's so mal di mare uncomfortable driving them. When I put in motors with 3-times the power output of the originals I found that simple open-loop IR motor compensation, with no sensors, worked so well that there was no longer any need at all for the feedback and I took the sensors off! (IR compensation just reads motor amps to estimate motor load; multiplied by (motor+wiring) milliohms this gives an estimate of voltage drop - add that back in, and the lugging motor speeds up - as long as it has the torque capability to do so. Actually,this is an open loop positive feedback situation so one adds back a little less than I x R to avoid twitchiness if there's just a bit too much, or runaway if you really exagerate.)

If I were to want closed-loop steering, however, I wouldn't do it, in this day and age, with motor encoders. Look at what hobbyists have done with stability-assisted, or even autonomously piloted, helicopters and think about the minimal cost of even a 9 DOF gyro/accelerometer/magnetometer board. Then think about what's involved in buying encoders and finding room for mechanically coupling them to motors that don't have them. I'd go with a mini "inertial navigation unit" rather than the encoders, and an 8-bit Atmel can certainly handle that.

to be continued ...

5

... continued

(Both mechanisms used together can be used to detect node failure, think "timestamped heartbeat" for each node.)

I think that I achieve the same detection of node failure by having the receiving node send a confirmation back to the commanding node. I suppose that what you're doing is possible, but I do wonder how you'll pull it off while retaining the robust arbitration of CAN.

Anyway, the CAN frame contains allready a CRC, transmission limited to single CAN frames,should be sufficiently protected.

CRC is used in CAN only so that a receiving node can decide whether to send an ack. If there's no ack the message is re-transmitted until an ack is returned or until an error flag is set after multiple failures. Unfortunately, you can't tell whether the first ack was sent by your intended recipient or some other node. Hence my addition of confirmation messaging. I also don't think that the CRC field is passed into the receive buffer, so my content check is the less-comprehensive, but fast, data ^ complement in "void RECEIVE_MESSAGE ()". I agree that error-correction would be hard to implement in an 8-bit MCU, but given the relatively slow message rates actually needed I think that discarding bad messages and re-sending is safer than trying to fix the bad ones. So far, I actually haven't seen any re-sends because messages were actually corrupted. Sometimes, however, when sending the first message of a given ID class, a re-send or even two has happened because MASTER's message buffers were already full with an un-processed message with a different ID before the confirm message arrived. For this, some of the MCU's with built-in CAN and more receive buffers might be better, but for an amateur like me, building a system around an MCU not already in the Arduino-recognized group is, I think, out of the question.

Thank you for having shared your work,

Your welcome.

We can probably do some stuff together.

I hope so, but first I have a plea. I know that reading someone else's code, especially code written by an amateur more used to Pascal and PerfectScript (I'll bet you don't know what that is!), can be a painful experience. Nevertheless, I hope that you, and others will make a stab at it.

Ciao,
Lenny

P.S. If you would like to see just how well-engineered a wheelchair can be, spend some time at http://www.wheelchairdriver.com/ (and perhaps join in on the forum). John Williamson (a.k.a. Burgerman) is a largely self-taught, widely experienced, highly opinionated, and, in my opinion, brilliant engineer who has built a series of ever-better wheelchairs matched to his needs and skills. He is obsessive-compulsive about detail, yet contends that he is constitutionally incapable of learning programming. It was seeing what he has done that gave me the courage to re-do Rachele's chair so that instead of being a gutless wonder that couldn't climb some of Siena's medieval city streets it is now an (almost) go-anywhere ATV that still works well inside the house. And it was that experience that has given me the courage to embark on the current project, though I may need some more injections of courage to carry it to completion.

6

Dear Rob (alias Graynomad),

So this is where that on/off circuit was bound.

Yep.

Looks like a worthwhile and interesting project.

Thank you. I want to, however, address the last paragraph and P.S. of my post to Brent to you as well. From looking at your web site, I have the impression that you are the micro-computer equivalent of what Burgerman is for the wheelchair world. I am, as I keep saying, a rank amateur and I am VERY NERVOUS about this project. I think that I am doing a good job, but a wheelchair is a special situation - I REALLY don't want to hurt anyone and I may well drop this if I don't get some critical feedback (or unless someone with a lot more money than I have materializes who wants to, from the goodness of his/her heart, make it FDA/CE compliant - fat chance).

Ciao,
Lenny

7

Hi Lenny,

I would hate to see you drop this project as it does seem worthwhile, I have many things on at present but would like to help a bit if I can.

You are right in that CAN has error detection at the low level but you need to implement a higher-level protocol with a lot of fault and general sanity testing of data.

I think it's fair to say that the Arduino core code (and maybe most libraries) is not robust enough for a critical system, I've seen all sorts of examples of code that for example takes an input at face value and uses that to index into an array. Case in point

int analogRead(uint8_t pin)
    {
	uint8_t low, high;
        if (pin >= 54) pin -= 54; // allow for channel or pin numbers

This is the start of the analogRead() function, there is no testing of the parameter "pin", Lord knows what happens of the value passed is 255.

I've only had a quick look at your code and you do similar, for example

 for (byte I = 0 ; I < Damping ; I++) {
    sumRawThrottle = sumRawThrottle + analogRead(ThrottlePin);
  }

The results of analogRead() are use without being tested first. Admittedly you are integrating the result but still, if the current average is 20 and you read a value of 500 it's fair to say something has gone amiss, maybe a wire broke on the sensor. In which case more drastic action is required because if you get 50 of them in a row you are going full speed before you know it.

Here's a code snippet from a string function I wrote for my LPC Arduino port.

uint8_t stringGetCharAt(string * s, uint32_t offset) {

	VERIFY_OBJECT(s, OBJID_STRING);

	if (offset >= s->max_len) {
		SYS_ERROR (ERR_STRING_BAD_OFFSET);
	}
	return (*(s->str + offset));
}

Note there is only a single line of code that does the main job, the rest is error checking. Now I may have gone overboard for a simple "return the Nth char from a string" function but probably not, what if the char returned was then used for a critical purpose?

Note that the "string" is not just an array of char, it's a structure with guard bytes at the beginning and the end and a length variable. The VERIFY_OBJECT macro tests for these byte to be the correct value AND the inverse of each other. If either test fails we have memory corruption or have been passed an invalid pointer. Then I test the offset into the string to be within expected bounds.

If either test fails it's serious and I drop into SYS_ERROR.

/* could be #define to reduce RAM use */
// for joystick driving
byte AttendantFwdSpeed = 100; // percent, reduce if headroom needed for turning or to get lower top speed

Yes you should use define or const for that lot.

Some other things that are largely stylistic...

UPPER_CASE is generally reserved for macros and constants. Using that for function names is slightly confusing.

The files are huge, it's time to break the code up into manageable sections in separate files. This makes it easier to review because you can set the cursor in appropriate places in multiple files and "^TAB" between them, as it is if I want to quickly see what xxx() does I have to scroll down 500 lines and by the time I find it I've lost the context and can't remember why I was looking in the first place :slight_smile:

Next I looked at the cost of joining the CANopen consortium - thousands of Euros/year. There's nothing "open" about CANopen except the name.

This is common for many so-called open systems and it's a joke.

Re the protocol, I too have looked at various offerings and I find them all to be too complex even if they are genuinely free. I have some experience with similar projects (albeit many years ago), for example I designed much of the building monitoring system for the ASIO headquarters (Australia's answer to the CIA) and a fire brigade turn out system.

Those systems were master/slave and to be fair that does work well. But it's 100% dependant on the master and these days I prefer a multi-master publish/subscribe protocol. CAN is multi-master but I think you are proposing having a master that processes all info and then tells nodes what to do. While there will always be a nominal master to generally control stuff I think it's more robust to distribute some of the processing to nodes. Thus some functions can continue of the master craps itself or the cable is cut. This is harder to implement of course and may not be appropriate, it depends on the system and you have 4 nodes which may be an appropriate level of distribution for this application.

Another thing is a dead man's handle, for example a motor-control node should only drive the motor while it's getting commands to do so, if no command arrives for say 100mS it shuts the motor down or takes other appropriate action. If the Roboteq doesn't implement something similar you should add that function before it.

My apologises if you have already covered these issues above, there's a lot to read and I may have missed stuff.

As for the hardware if this is a one-off I would be inclined to look at Bernt's board as it seems well designed. If you plan to take this further I think designing your own WC-specific boards might be a better way to go.

That's a long enough post for now.

Rob

8

Rob, How can someone who is retired have many things on at the moment !!! XD

Probably busy doing some more thinking about what to do tomorrow !!!

Craig

9

:), hmmm good point.

While it's true I retired 14 years ago at the tender age of 45 I have since learned to appreciate what people mean when they say "I don't know how I found the time to work".

I have 25 acres of land to clear of grass that in places is as tall as me. I'm cutting a series of walking tracks around my land which involves a lot of digging and track hardening with logs, I reckon I have 3-4ks of track to make and I've done maybe 400m. Yesterday I chain-sawed 3 fallen trees and moved the logs to my log pile.

I'm designing a house made of shipping containers plus landscaping around our existing containers, installing an eco loo in one container and building an office/workshop in the other plus a roof over the two and installing solar panels on top of one.

I'm doing some contract circuit and PCB design work for a Brazilian company and likewise for a Russian company, plus designing a Due-like CAN controller board and a monitoring-and-control network for personal use but also hopefully for release as an open source project. I also do a lot of nature photography and until recently wrote photography articles for magazines.

Then of course I have to see to my home brew (where it's entirely my responsibility to control production, distribution and consumption), maintain my 6x6 army truck/home and occasionally drive said truck to a new location when we get bored with the view in our current location.

After all that I need time to chill out, wind down, watch the wallabies feeding outside my window, listen to music and generally contemplate my life and/or navel.

I tell you, retirement ain't what it's cracked up to be.

Rob

10

Dear LROBBINS,

LROBBINS:
One of the things mentioned there is something I wish I knew how to do: "reprogramming of the board through the CAN bus when using a previously installed special bootloader ".

To do that you need at least 3 "parts" :

  • A script able to send a .hex sketch through USB to a board
  • A program which will transform the board into a USB2CAN adapter
  • A CAN-bootloader into each nodes on the network

Initially, the idea was developped by Fabian Greif, we just adapted his code to make it works on Arduino boards and create an Arduino sketch to do the USB2CAN stuff.
The programming operation is made by a Python script that initiates the communication process between the PC and the Arduino. This board will receive the new program by USB or serial port and sends it encapsulated in CAN messages to reprogram the specified node. This node must be into its bootloader section (by a reset or a software reset) to start the reprogramming.
Bernt actually working to make this code publicly available

11

Dear Rob,

I would hate to see you drop this project as it does seem worthwhile, I have many things on at present but would like to help a bit if I can.

And your comments are exactly the kind of critique I'm asking (begging) for.

You are right in that CAN has error detection at the low level but you need to implement a higher-level protocol with a lot of fault and general sanity testing of data.
I think it's fair to say that the Arduino core code (and maybe most libraries) is not robust enough for a critical system, I've seen all sorts of examples of code that for example takes an input at face value and uses that to index into an array. Case in point ...
I've only had a quick look at your code and you do similar, for example

for (byte I = 0 ; I < Damping ; I++) {

sumRawThrottle = sumRawThrottle + analogRead(ThrottlePin);
  }


The results of analogRead() are use without being tested first. Admittedly you are integrating the result but still, if the current average is 20 and you read a value of 500 it's fair to say something has gone amiss, maybe a wire broke on the sensor. In which case more drastic action is required because if you get 50 of them in a row you are going full speed before you know it.

Good point. Although I don't do a sanity check on every analog read, there is one before the average gets used:

  if ((*RawThrottle < 500) || (*RawThrottle > 4500)) {
    *RawThrottle = 2500; // centered joystick
    *RawSteering = 2500; // centered joystick
    if (lastThrottle != 0){
      Serial.println (F("Throttle disconnect - send Safety Stop message"));
    // send safety stop message to Roboteq and "throttle axis joystick error" message to display
    }
  }
  if ((*RawSteering < 500) || (*RawSteering > 4500)) {
    *RawSteering = 2500; // centered joystick
    *RawThrottle = 2500; // centered joystick
    if (lastSteering != 0){
      Serial.println (F("Steering disconnect - send Safety Stop message"));
    // send safety stop message to Roboteq and "steering axis joystick error" message to display
    }
  }

(BTW: the Roboteq safety stop command will shut the motors down even if my Throttle=0, Steering=0 message fails.)
You note later for a snippet of yours:

I may have gone overboard for a simple "return the Nth char from a string" function but probably not, what if the char returned was then used for a critical purpose?

Do you think that my pre-use check is not enough, that I should actually range check every analog read? Or even break out of the averaging as soon as one unreasonable value is read? Will that really afford more protection or will it only raise the probability of noise-provoked false positives?

/* could be #define to reduce RAM use */

// for joystick driving
byte AttendantFwdSpeed = 100; // percent, reduce if headroom needed for turning or to get lower top speed



Yes you should use define or const for that lot.

Agreed. In fact they should be moved into a UserParameters.h as they are the only things that 99.9% of users might want to change. For now I've left them as variables because they give me a quick way of spotting if I'm running out of SRAM without my having to actually track RAM use.

Some other things that are largely stylistic...
UPPER_CASE is generally reserved for macros and constants. Using that for function names is slightly confusing.

I wasn't aware of this convention. Among PerfectScript writers, preferred form is to write vVariableName and cConstantName (or even cConstantName, gGlobalVariableName, vLocalVariableName, pPersistantVariableName, and so on).

The files are huge, it's time to break the code up into manageable sections in separate files.

Absolutely, it is probably past time to do so. The way I work (at least in Pascal and PerfectScript) is to add functionality first directly at point of use until I know that it's working, them move it out into a procedure (or a nested set of procedures), then move those procedures out into one or more libraries. Until that last step, I make a lot of use of the Find dialog, and usually put a // TO HERE where I am working so that I can hop back and forth. It starts out as real spaghetti code (which unlike what we eat here, is not good), but eventually gets cleaned up. There are a few procedures shared among all nodes, and those will be moved off into a central library, or moved into the CAN library if directly related to CAN manipulations. Others will be moved into separate files in the same directory as sketches for individual nodes so that I can just flip among tabs. I've kept Constants.h in the same folder as MASTER for that reason, even though it means that I have to compile MASTER first in any session before I can compile the other sketches that use Constants.h.

I have some experience with similar projects (albeit many years ago), for example I designed much of the building monitoring system for the ASIO headquarters (Australia's answer to the CIA)

Which (for the new building) has been in the news of late. Was it your stuff that got hacked?

CAN is multi-master but I think you are proposing having a master that processes all info and then tells nodes what to do.

Not quite. What I have called MASTER handles user inputs. All high current functions are handled in the Roboteq node (unfortunately in that node everything will have to be all in one file, because that's all MicroBasic allows) and all display functions will be autonomously handled in DISPLAY. AUX just handles Roboteq power ON/OFF in response to MASTER - don't want autonomy here; I want just one switch to handle off/on for the whole system.

While there will always be a nominal master to generally control stuff I think it's more robust to distribute some of the processing to nodes. Thus some functions can continue of the master craps itself or the cable is cut. This is harder to implement of course and may not be appropriate, it depends on the system and you have 4 nodes which may be an appropriate level of distribution for this application.

On a wheelchair I think that there are only two functions that should continue if the master (or other critical node) craps out or the cable is cut: cut off power to the motors and display & log an error message. I think that I've got this level of distribution.

Another thing is a dead man's handle, for example a motor-control node should only drive the motor while it's getting commands to do so, if no command arrives for say 100mS it shuts the motor down or takes other appropriate action. If the Roboteq doesn't implement something similar you should add that function before it.

Yes, the Roboteq has a watchdog that shuts motor output. By default, it's set at 1 sec (and I repeat throttle and steering at 900 msec intervals if they're not 0 and not changing), but it could, and probably should, be shortened. In MASTER, if a confirmation message isn't received within a few tries, for example if Roboteq CAN goes off line, it does set Throttle = 0 and Steering = 0 (even before the watchdog kicks in) as well as sending an error message to DISPLAY. [I will also have a completely separate emergency shut down - a jack wired NC held open by plug attached to a lanyard on my wrist. The jack will be in series with the contactor coil - yank on the chord and halt, but better be wearing a seat belt as it will be a jolt.] ... to be continued

12

... continued

My apologises if you have already covered these issues above, there's a lot to read and I may have missed stuff.

No apology needed. Whether it's something already done, or already planned, or something I've not thought of, I think it's important that I get these comments. If nothing else, it forces me to explain what I think I've done. If I discover that I can't explain it, then I know that I'm probably not doing it right.

As for the hardware if this is a one-off I would be inclined to look at Bernt's board as it seems well designed. If you plan to take this further I think designing your own WC-specific boards might be a better way to go.

If this works out, there would probably be a few other users. One adventuresome soul has already asked for the bare boards so that he can experiment with it as he's planning to build a chair with Roboteq controller and LiFePO4 batteries at a higher voltage that commercial WC controllers can't handle. The numbers would never be large, however. I wasn't aware of Bernt's board until his note on this thread, but I will look at it with some care (and probably pester him for things I can't glean from what he's put on the web so far). It combines on one board everything that I have on 3 (MCU + CAN + DC-DC converter), but the form factor might not work. The attendant joystick on Rachele's chair is in a 6 cm (w) X 11 cm (l) X 5.5 cm (h) die-cast case. It's a good size; much better than the ever-larger joystick pods that are being marketed and that really get in the way. The joystick pot takes up almost 1/2 the available space in this box, yet it would hold my my stacked nano-sized boards and the ON/OFF and MODE buttons, the USR/ATTENDANT toggle and whatever connectors I end up with (even easier with a Hall-sensor joystick as they are shorter than the inductive unit in the DX-ACU). I think Bernt's board might not fit. DISPLAY will be in a small box that can be put on the back of the chair when an attendant is driving, or attached to Rachi's computer when she is driving, but that will be larger w x l and shallower, so Bernt's board would work well there. In any case, I already have four fully-functional Nano-sized CAN boards and that's plenty for now. After seeing on the MCP CAN forum how much trouble people have getting CAN to work, I was actually surprised at how problem-free this has been: one board didn't work because of a bad 17 pf capacitor (for the crystal), but otherwise (FLW?) it's been pretty smooth sailing. (An aside, I have made a small change to the CAN board design in the days since I posted. As I'll have two sockets on each box, and the end boxes only use one, I've removed the termination resistor and 2-pin header from the board and plan on using a termination plug. That gives me room to add a TVS and, if needed later, a common mode choke.)
Again, I really, really appreciate your feedback and hope you'll find time to keep them coming (and that y'all can put up with my verbosity).

Ciao,
Lenny

13

Dear Yellowsky,

Thanks for outlining how this works, and to Fabian Greif for coming up with the idea, to your group for adapting it for Arduino, and to Bernt for working on making it publicly available. My 1.5 years of German study nearly 50 years ago is not enough to let me read Herr Greif's pages, but between a few words and skimming some of the code I have an idea of how this was implemented - not that I could, in what remains of my lifetime, do the same.

I wonder how hard it would be to make a CAN2USB board so that a Roboteq MicroBasic script could be loaded at the same time, or even if it would be worthwhile. (Something to keep in the back of the mind for later.)

Ciao,
Lenny

14

Dear Lenny,

LROBBINS:
The attendant joystick on Rachele's chair is in a 6 cm (w) X 11 cm (l) X 5.5 cm (h) die-cast case. It's a good size; much better than the ever-larger joystick pods that are being marketed and that really get in the way. The joystick pot takes up almost 1/2 the available space in this box, yet it would hold my my stacked nano-sized boards and the ON/OFF and MODE buttons, the USR/ATTENDANT toggle and whatever connectors I end up with (even easier with a Hall-sensor joystick as they are shorter than the inductive unit in the DX-ACU). I think Bernt's board might not fit.

The board is 1.950 x 1.950 inch², just under 5x5cm². It should be < 15mm tall not counting the green phoenix connector. I will probably get rid of these too bulky connector, what type of connector would you put there? I hope the board does fit even with your joystick, doesn't it? If not, would a more elongated form factor help ?

... to be continued

15

Dear Lenny, dear Rob,

Here comes an unsorted list of answers to your text:

  • Your remarks on our sketched protocol tells me, that we are still far from something good. And Lenny and Rob, convinced me that we need to add higher level message integrity/delivery guarantees.

  • The need of a higher level CAN protocol: we all conclude that none of the well-known protocols fits our needs. Lenny and us started to design new protocols. I have difficulties to imagine that no-such protocol exists out there. I feel that the constraints are very common to lots of CAN-robotic projects. Does nobody know of an existing simple, well-tested protocol for those applications?

  • Robs clear example of Arduino analogRead() tells me, that, for a finalised product with serious security constraints, we need to get rid of Arduino software, and do a more secure reimplementation of a subset. But for prototypes and demonstrators, the Arduino approach stays still very attractive. If it was for driving my-self in a wheel-chair, I would probably stay forever with Arduino libraries, in an approach close to the one of John Williamson "wheelchairdriver.com" (and rely just on some amount of testing).

  • Robs analysis on your software is far better than what I could produce.

  • The appealing idea of a low cost mini "inertial navigation unit" is probably good for outdoor use with sometimes slipping wheels. But inside, you can not rely on the compass sensor used in conjuction with the gyros. Anyway, more different sensors gives better chance to detect abnormal, potentially dangerous situations (as once again slipping wheels).

  • The programming through the CAN-bus based on Yellowsky's modification of Fabian Greifs work is coming soon. My work consists of sorting out licences, attributions, deleted comments that existed in the original code, some more German to English translation and revert unnecessary edits introduced during debugging here. Just one interesting thing to point out: Yellowsky transformed parts of Fabian's CAN debugger firmware in a simple Arduino sketch that allows us to use as interface between the host PC and the CAN bus : standard Arduinos with CAN shield, your boards or the CANinterfacer. This does not give the full-speed CAN debugging as Fabian's CAN debugger, but it's sufficient for programming devices on the CAN bus.

Thank you all, I learned a lot from your comments,
Bernt

16

iyiyi this thread must get the gold medal for long posts :), here's another one.

Do you think that my pre-use check is not enough,

I like to catch problems as close to the source as possible.

that I should actually range check every analog read?

I'm not up on what they all do but that's quite possible.

Or even break out of the averaging as soon as one unreasonable value is read?

Not one bad reading, but don't add the bad one to the average, maybe count the bad ones and abort if you get more than 3, if it comes good assume it was noise and start averaging again but the bad readings did not get through. You've got Damping set to 20, I'm not sure what values you normally get but imagine if 10 was the current average and you get 1023 because a wire broke, that will seriously skew the average, get 5 in a row and you are in trouble.

As you say you do test later but if you were happily cruising at 500 and a wire breaks you will ramp up to 4500 in no time and the current code will think that is OK, it will reset to 2500 but that's still 5x faster than the user expected. By the time your daughter responds she could be down the stairs.

I wasn't aware of this convention.Among PerfectScript writers, preferred form is to write vVariableName

That format is Hungarian notation, disappeared years ago AFAIK and not popular with C programs I think.

Notation Example
Standard C / GNU notation mouse_weight
Hungarian notation iMouseWeight
Camel case (or lower Camel case) mouseWeight
Pascal case (or upper Camel case) MouseWeight
All capitals case MOUSE_WEIGHT

My preference is for C notation for class member names and local objects, camel case for most other things and all caps for defines and consts. It's not a hard rule of course and people have their own conventions.

Which (for the new building) has been in the news of late. Was it your stuff that got hacked?

No, my stuff was put in the first building when that was new. Hacking wasn't a problem then because there was no internet and the comms link around the building was optical fibre, pretty hard to tap into.

If I discover that I can't explain it, then I know that I'm probably not doing it right.

That's a very good technique, and quite true.

It combines on one board everything that I have on 3 (MCU + CAN + DC-DC converter),

That's a good thing for reliability, connectors are always a weak point.

Does nobody know of an existing simple, well-tested protocol for those applications?

Not me and I've looked a lot. There seems to be nothing between "ascii with a checksum" and full-blown Modbus or whatever.

This is a pet subject of mine and I've spent the last couple of years (on and off) working on designs, starting from a very complex master/slave protocol to a simple multi-master version. For most of that time the topology has been async serial over a redundant ring because with an RR you can cut the cable and it has almost no affect on the network traffic.

See here for a brief description of the idea

http://www.ardweenet.com/

After all that just a few weeks ago I decided to use CAN as the base level so most of my previous work is out the window :slight_smile: I still prefer an RR over a bus but truth is CAN must be very reliable given it's used in vehicles so I'll get over that.

So having decided on CAN, and given that the SAM3X has two CAN controllers, it was a natural thing to design a Due-like board to act as the "master" and that's where I am at now, distracted yet again from my network design :frowning:

My messages contain at most 4 bytes of data and the other space is used for the bit-wise complements of these.

The CAN frame has a CRC and it is very robust, I think you are wasting half your payload here. With the full 8 bytes you can represent all numeric data types up to float.

I'll have a better read through your protocol design doc.

Rob

17

The board is 1.950 x 1.950 inch², just under 5x5cm².

Yes, that would fit. How are you handling power for the (multi-node) system?

Yes, as Rob points out, connectors are a major headache and I'll describe some of my experience (and more or less crude fixes) when I reply to his post. As far as the Phoenix goes, I just generally dislike screw-clamp connectors in a high-vibration environment - I prefer to solder and de-solder if need be. If I had to use screw connections, I think I'd follow aircraft bus practice - a barrier strip with studs, extra-secure external star lock washers, and anti-vibration or jam nuts.
Ciao, Lenny

18

Hi again Bernt,

  • The need of a higher level CAN protocol: we all conclude that none of the well-known protocols fits our needs. ... I have difficulties to imagine that no-such protocol exists out there.

Perhaps the lack of same is understandable in the original context for which CAN was introduced: first heavy industrial vehicles and equipment like cranes, then for motor vehicles in general. Here, proprietary interests take precedence over inter-operability.

  • Robs clear example of Arduino analogRead() tells me, that, for a finalised product with serious security constraints, we need to get rid of Arduino software, and do a more secure reimplementation of a subset. But for prototypes and demonstrators, the Arduino approach stays still very attractive. If it was for driving my-self in a wheel-chair, I would probably stay forever with Arduino libraries, in an approach close to the one of John Williamson "wheelchairdriver.com" (and rely just on some amount of testing).

And as much of what Roy calls "sanity testing" in your program as can reasonably be added even if one avoids Arduino entirely. It's like having message-testing over and above what CAN already does. Can you really be sure that there's NO unusual circumstance in which your "perfect" code doesn't turn out to be less than perfect? (And a completely independent Mayday shut-down is probably in order too if you don't test to the tightest of compliance standards.)

  • The appealing idea of a low cost mini "inertial navigation unit" is probably good for outdoor use with sometimes slipping wheels. But inside, you can not rely on the compass sensor used in conjuction with the gyros. Anyway, more different sensors gives better chance to detect abnormal, potentially dangerous situations (as once again slipping wheels).

If you have slipping wheels indoors, the chair itself is badly engineered. Actually, for indoors (i.e. short term corrections), the gyros alone are adequate - the compass would only be used to correct drift over long tracts.

  • The programming through the CAN-bus based on Yellowsky's modification of Fabian Greifs work is coming soon. ..: Yellowsky transformed parts of Fabian's CAN debugger firmware in a simple Arduino sketch that allows us to use as interface between the host PC and the CAN bus : standard Arduinos with CAN shield, your boards or the CANinterfacer. This does not give the full-speed CAN debugging as Fabian's CAN debugger, but it's sufficient for programming devices on the CAN bus.

If that were a meal, my mouth would be watering (don't know if French has an expression like "mouth-watering" in English, or "acqualina in bocca" in italiano).
Ciao,
Lenny

19

Dear Rob Dear Rob (and everybody else),

Graynomad:
http://www.ardweenet.com/

After all that just a few weeks ago I decided to use CAN as the base level so most of my previous work is out the window :slight_smile: I still prefer an RR over a bus but truth is CAN must be very reliable given it's used in vehicles so I'll get over that.

So having decided on CAN, and given that the SAM3X has two CAN controllers, it was a natural thing to design a Due-like board to act as the "master" and that's where I am at now, distracted yet again from my network design :frowning:

I just finished reading your "http://www.ardweenet.com/". Very interesting!
It makes me question our perhaps too simple network topology:

  • How do today’s commercial wheelchairs do it? I think they use CAN, don't they? Do they use double CAN cabling? Do they even use ring topology?
    (How to get two separate cables to the wheelchair armrest?) Can somebody answer the same questions for cars?

I like your board. You built on Arduino compatibility, you chose the same TSR_1-2450 6.5 – 36 V switching power supply, and the size is to 1/20 of an inch the same than the CANinterfacer. RS-485 and RS-422 was a serious option for us too. And as you, we chose finally more modern CAN, designed for cars, having close requirements.

Quote from your web:

NOTE: One particularly important function required right now is to cast a critical eye over the network software design. I have a fairly large document that describes this that I can send to any experienced programmers with an interest, even better if you have this sort of networking experience, IE RS-485 based control networks.NOTE: One particularly important function required right now is to cast a critical eye over the network software design. I have a fairly large document that describes this that I can send to any experienced programmers with an interest, even better if you have this sort of networking experience, IE RS-485 based control networks.

I'm interested. Looks like there is a lot to learn from in there. (From your forum I got the following dead link: http://busnet.robgray.com/documentation/docs/BUSnet-dg-protocol-v0-2.pdf )

Thank you very much for the nicely done documentation on your website,
Bernt

20

Hi Lenny,

LROBBINS:
How are you handling power for the (multi-node) system?

There is that TSR 1-2450 switching DC/DC converter on board (6.5 – 36 V input, same as used by Rob on his board). There is no fuse on board, the converter can handle shorts.

Power for digital parts comes typically from the bus cabling, but you can connect elsewhere if needed. If any power electronics is associated to a node, it gets separate fat supply cabling and over-current protection.

On the small single handed sail boats (now preparing for Miniji and Access dinghies), we will have a CAN concentrator box with star cabling. A small sealed lead-acid battery goes in the same waterproof container, with an extra valve to the outside.

Have a nice day,
Bernt