Curious about theories from (especially) anyone with experience using CAN-BUS in custom automotive applications.
I put this in Bar Sport because I'm not looking for the solution to what will be a vaguely described problem, just a general quasi-diagnostic spitballing from "more-informed-than-laypersons", shall we say - ie we in a hobbyist microcontroller community.
Before anyone says it, yes, I know the general posting rules and I am simply not able to provide schematics or source code: I'm a firefighter, not a mechanic and as per City policy, access to these is "beyond the scope of my duties". In general, I'm not even supposed to chat with the mechanics as these conversations occur higher than my rank in the chain-of-command. Also, the rig described henceforth is not my usual apparatus, nor my usual fire station - I was just there yesterday for overtime and had to change over the rig to a spare Ladder Truck for the problem truck to be sent off to Mechanical Division.
With that out of the way, the vague description:
This is a very new apparatus, < 6 months in service. There have been no calls involving this truck to suggest damage from freezing, exposure to heat, chemicals, anything like that. The truck should be assumed in tip top, brand new condition.
This is a North American style, typical Quint (5 standard fireground functions in one apparatus) diesel Ladder/Aerial Truck. As such, it includes a Pump Panel with a modern Electronic Pressure Governor and a 32m aerial device and all the hydraulics for it and the outriggers, as well as a generator to supply electricity on scene for things like lights and fans.
It should be noted that in recent Apparatus build specifications, there are included several lithium ion battery chargers as the Fire Service is rapidly transitioning to battery powered electric saws, vehicle extrication cutters, rams and spreaders, thermal imaging cameras. I do not know if these chargers communicate on the CAN-BUS, but suspect not as they are not integral to the operation of the Apparatus per se, although the electronic pressure governor and aerial controls most certainly are.
Very recently, Drivers have been reporting "gremlins" in the electronics of the Truck. Seemingly inconsistent, unexpected abnormalities such as the turn signal signaling left when not called for, the emergency lights acting erratically when the siren function is changed, the whole array of dashboard engine warning signal lights illuminating during vehicle movement then turning off again, things like that.
In casual discussion with another Firefighter who has some experience in building custom cab-over-engine trucks, his opinion was that the twisted pair wiring for the CAN system was not properly shielded and CAN Node crosstalk was causing these unpredictable gremlins.
I was leaning more to the idea that or one or more nodes in a multi-master system were improperly chassis grounded or that something wasn't receiving the required voltage to allow the CAN system to correctly prioritize controller nodes. My thought is that the issues are erratic and seem to be increasing in frequency of occurences which on my amateur sniff test just doesn't sound like wire shielding failure but a larger issue.
I'd love to hear what the community thinks may be going on.
Given that this is a 6-month old piece of equipment in active use for life-saving duties, the obvious course of action is to engage with the manufacturer so they can resolve the problem. Yes, that's bleeding obvious - as these things sometimes are.
Agreed, which is why we spent an hour yesterday swapping out the apparatus to a spare Ladder to send the rig out for service to either the manufacturer or Mechanical Division (don't know which, maybe both, not my decision to make).
Yeah, sounds good. I don't think there's all that much more you could do. Whatever would help in tracking down the problem would involve poking around with measurement/analysis equipment. Apart from the risk of actually shorting something out accidentally, you would probably be breaking a couple of formal rules/regulations in your organization. It's not worth it to put your good relationship with your boss on the line just so you can give the manufacturer some hints in which direction to look (they should be more than capable of managing that, anyway!)
Agreed, we followed policy and acted on the direction of our shift Senior Officer. I was just curious from my "other life" point of view as a hobbyist in electronics what others here in the same hobby might think, given the vague and limited info I was able to supply. As I said in post #1, that's why this is in Bar Sport. I'm not looking for info to pass along to anyone responsible to fixing the issue, just chewing the rag on a Tuesday morning after shift.
Nope. Unless there is defective hardware involved. Which is unlikely. CANBUS is actually designed to work over a single wire. Which makes sense. In a dramatic vehicle accident there is a chance some of the wiring could be damaged. We want something that is extremely reliable even under the most adverse conditions.
And, I've tested exactly that. And, wrapping the single wire around an AC transformer. And, grossly exceeding the maximum wire length. And, using bare wire. And, using very tiny gauge wire.
The physical layer is very unlikely to be the problem.
My bet is on the typical bugs found with C / C++ programming: buffer overrun, uninitialized variables, use-after-free. The sorts of bugs that trigger undefined behaviour (aka "gremlins").
So how would this occur in your opinion, on controllers that were presumably purchased from a vendor? I know nothing of the quality of programming one might expect to receive in a "professional", off the shelf (probably?) component. Are they not pre programmed from the factory or am I misunderstanding how they are installed and used in custom vehicle installations?
1: The vendor actually not delivering a bug-free product.
2: Not mentioned before, but very plausible IMO: work on behalf of the systems integrator (the company that built this vehicle/device) that violates the interface specifications of the used modules/components, resulting in unanticipated behavior.
That's certainly within the realm of possibility. I wish I had some of the specifications to share here, alas, I do not. Everyone thinks fire trucks are cool (and they are ).
By the by, my station is getting a brand new Spartan Engine in a week or two! We're pretty pumped (pun intended).
We appreciate the ongoing support, thank you. I always like to thank folks in return for the work they do, whatever it is. A well functioning society depends on all our contributions: everyone's job is important or that job simply wouldn't exist.
The best support the community can give us, in my opinion, is to install and correctly maintain their smoke and CO alarms.
With almost 20 years service in a busy city, I can assure the community that the fire department alone is no substitute for a home with enough properly maintained smoke and CO alarms from a life safety perspective.
I always tell even the kids on station tours: when it comes to fire fighting, everyone, yes even kids has a job to do (tell a grown up if you see smoke or think there's a fire; don't hide in a closet or under a bed in the event of a fire; test your smoke and CO alarms; have and practice an escape plan with two ways out of every room; everyone's #1 priority is to get out of the house and stay out which is most often only possible if you have enough working smoke and CO alarms; call FD from outside the house).
As to the topic in this thread, I'm really hoping to be able to provide the answer to the problem by Friday. The Chief hates frontline apparatus sitting over the weekend when Mechanical is running at reduced staff and I know he's fuming about this issue on a brand new truck. I sure would be!
As am I. I am slowly reading through and trying to understand how the CAN protocol works, particularly the arbitration process on a CAN bus.
What might that look like in C code on an Arduino?
From pg 4 of a TI document:
Bus access is event-driven and takes place randomly. If two nodes try to occupy the bus simultaneously, access is implemented with a nondestructive, bit-wise arbitration. Nondestructive means that the node winning arbitration just continues on with the message, without the message being destroyed or corrupted
by another node.
Is this closer to SPI or UART from an Arduino perspective?
reading the doc, it is a bit confusing..
seems more like rs485 with everyone talking at once..
i use a proper scan tool, got a port usually under driver side dash..
changed some head gaskets, late model camaro..
did good job, had the heads machined..
would crank but wouldn't turn over..
took me 2 days, just one ground strap was off..
Those sorts of mistakes are easy to make with C / C++. This has several pages of hits just from this forum. Even experienced programmers will sometimes make those mistakes.
That can vary wildly.
This is a good time for an overview. I think that will bring some clarity.
CAN BUS sits at layer 2 in the OSI model. That means a node on the network can send a single frame of data to another node or broadcast to all nodes. A CAN BUS frame is as small as 11 bits and as large as 93 bits. That's it. That's the entirety of what CAN BUS brings to the picture.
To be useful CAN BUS has to be coupled with a processor. It turns out, Microchip has AVR processors with an integrated CAN BUS controller. They are nice.
There are various ISO automotive standards that add OSI layer 4. With that, arbitrarily large amounts of data can be moved.
So, the processor for a CAN BUS node would typically be programmed by the manufacturer. If it's an AVR processor then the same techniques used to program an Arduino board are used.
).