CAN bus traffic logger

Projects General Guidance
1

I'm working on a school based senior design project. The project is a custom vehicle that communicates over a 250k CAN bus. I am using https://www.sparkfun.com/products/10039 as the shield to interface with the bus.

For some reason I seem to be only reading about 40% of the messages. I got this figure by comparing the total message count of using the Arduino Uno+CAN adapter and a different adapter(a kavaser virtual serial port adapter) over the same time span. I should be logging 10000 messages or so in about 10 seconds, but the Arduino is only getting about 4000.

Is the Arduino just not fast enough to get all the messages? How can I ascertain whether or not this is the case, and if so, what other options do I have?

2

nr1224:
Is the Arduino just not fast enough to get all the messages?

That would be my guess. It's only a tiny teeny wimpy little microcontroller. Although it may be able to send and receive packets on a high speed network, that doesn't mean it can keep up with the whole of the network traffic if the network is busy.

The ability to keep up with the network traffic would depend enormously on how efficient your code is - perhaps if you post it, somebody can spot whether there is any scope to make it more efficient.

3

I read somewhere that the maximum serial communication speed for a 16Mhz arduino is 115200 bps.
115200/250000=46%

I have a feeling this limitation is based on memory and message buffer. (the buffer is filling up)

4

I've attached my code for everyone to look at, but it sounds like I was right about it not being fast enough. Would simply moving up to a faster/bigger micro-controller be a smart move? If so, what would be a good choice to remain Arduino compatible, a Maple perhaps?

can_data_logging.ino (2.49 KB)

5

I'm willing to take a look at your code but only if you first answer a question. If you just want to log a stream of CANBUS frames, why are you not just using the Kvaser? If I remember correctly, it even comes with a logging application specifically designed to collect huge quantities of data.

6

That's a great question with a simple answer. The vehicle in question is a high performance electric motorcycle that might eventually go as fast as 160mph. The Kvaser is simply a usb dongle plugged into a computer, and while it performs great, there is no way to stick a whole x86 computer into such a small enclosure.

7

Let's start with some basic math. We'll assume no overhead with CANBUS...
250000 bits per second / 8 bits per byte = 31250 bytes per second

http://forum.arduino.cc/index.php/topic,77378.0.html
Occasional write delays of 200 milliseconds.
31250 bytes per second * 0.200 seconds = 6250 bytes lost

Is that an acceptable loss?

If not, you will be learning to use @fat16lib's beta library (be sure to read past that post for information about your SD card choice)...
http://forum.arduino.cc/index.php?topic=98898.msg744496#msg744496

8

Ok, I will try and integrate that and see what kind of performance boost I get.

As I mentioned earlier, is it possibly simpler to buy a faster controller that could catch everything, and if so, which one?

9

nr1224:
As I mentioned earlier, is it possibly simpler to buy a faster controller that could catch everything, and if so, which one?

Arduino Uno

You first should establish that a faster controller will make a difference. A little more math will tell you that the Uno processor (an ATmega328 clocked at 16 MHz) probably has enough horsepower to do the work.

At this point in the discussion the bottleneck is the SD card. You can try to overcome the SD card's limitation by using @fat16lib's library and buffering data. The potential pitfall is the amount of SRAM available on the Uno. You simply may not have enough space to buffer data until the SD card catches up.

One alternative is to use external memory that does not have the delay. FRAM is a good example. It's more expensive and using it will require you to develop more code but writes are performed faster than the Uno processor will be able to send data.

10

I understand what you're saying, I don't really want to move past the arduino until I absolutely must.
However, from my perspective, I've got two options. I can stick it out with the arduino and more or less fight with the hardware to do what I want it to, investing time and money into it...OR...I can just switch over to a RasPi and more or less avoid this problem entirely because it has enough flash RAM to hold a gigantic buffer of information instantly, and then writing it all to the SD card in one swoop once I'm done logging.

I'm a mediocre coder at best, I'm not really sure what the best choice is. Is FRAM significantly expensive? Is it hard to find and interface with?

11

Is it hard to find and interface with?

https://www.google.com/search?q=fram+site%3Aforum.arduino.cc
http://forum.arduino.cc/index.php?topic=125219.0

Is FRAM significantly expensive?

http://www.mouser.com/Semiconductors/Memory/F-RAM/_/N-488wv?Keyword=f-ram&FS=True
Looks like about $15 per megabit.

If you go this route you will have to use Serial to dump the data to a PC.

12

nr1224:
I can just switch over to a RasPi and more or less avoid this problem entirely because it has enough flash RAM to hold a gigantic buffer of information instantly, and then writing it all to the SD card in one swoop once I'm done logging.

That would certainly be a simple low risk solution.

13

I've been doing some speed testing, and it appears that my code does in fact take too long to write to the SD card, explaining the missing packets.

I want to attempt to load the data into a buffer, and somehow multitask so that the Arduino can still collect data as it writes to the card...interrupts perhaps?

What is the basic overview of allocating a buffer in RAM and doing two tasks at once?

14

I suggest you start by replacing the following code with something that does not touch the heap...

String output;
...
void create_string()
{
  output = ""+String(message.id, HEX);
  
  for(int i = 0; i<8; i++)
  {
    output += " "+String(message.data[i], HEX);
  }
  
  /*
  output += " "+String(message.data[0],HEX);
  output += " "+String(message.data[1],HEX);
  output += " "+String(message.data[2],HEX);
  output += " "+String(message.data[3],HEX);
  output += " "+String(message.data[4],HEX);
  output += " "+String(message.data[5],HEX);
  output += " "+String(message.data[6],HEX);
  output += " "+String(message.data[7],HEX);
  */
  
}

void write_string()
{
  data.println(output);
  //Serial.println(output);
  /*char buffer[output.length()+1];
  output.toCharArray(buffer, output.length()+1);*/
}
15

nr1224:
I want to attempt to load the data into a buffer, and somehow multitask so that the Arduino can still collect data as it writes to the card...interrupts perhaps?

Interrupts add complexity and overhead. The complexity can easily introduce bugs that are extremely difficult to fix. The overhead can result in your application being even slower.

16

nr1224:
I've been doing some speed testing...

I assume that testing includes some basic profiling. What are the results? What is the most time consuming code?

17

I've found that I receive about 1000 messages a second; my code would need to be operate in under a millisecond, from grabbing the data to writing it, in order to be fast enough. Currently it takes 3-5 milliseconds, explaining my missing data.

I apologize for not totally understanding you, when you say avoid the heap with the String creating functions, what exactly do you mean?

18

The String class allocates memory dynamically from the heap as required, and returns it to the heap when it's no longer required. This dynamic allocation has a significant overhead. There is also a risk that due to the Arduino's necessarily simple memory management, allocating and deallocating heap memory can leave the heap fragmented which can lead to allocations failing even though there is still memory available, because the available memory isn't all contiguous. In older versions of the IDE there is an added issue that there was a bug in the heap management code which could cause a memory leak.

The basic premise of the String class is to make it easier to manipulate strings, which is fine when you're in a VM with massive memory and CPU to spare but an expensive luxury when you're in a microcontroller with only a couple of KB of RAM to play with.

My suggestion is that you stop using the String class and use c-strings (null-terminated char arrays) instead. There is a rich runtime library of string manipulation functions which enable you to do anything that the String class does. Many of the String methods eventually end up invoking the underlying c-string library anyway, and in most cases it's no harder to access them directly.

(This is another case where IMO the Arduino designers were excessively inspired by Processing and tried to make C++ look like Java - I think it's completely misguided.)

19

nr1224:
I've been doing some speed testing, and it appears that my code does in fact take too long to write to the SD card, explaining the missing packets.

Right, and I think you'll find that you can't really get around that because of the nature of the SD card interface. A while ago, someone asking a question here, was told that the interface chip for the SD card is buffered, and that some number of bytes are stored before writing, and that this does not always take the same amount of time, due to varying lengths of data.

I want to attempt to load the data into a buffer, and somehow multitask so that the Arduino can still collect data as it writes to the card...interrupts perhaps?

What is the basic overview of allocating a buffer in RAM and doing two tasks at once?

You still need time to perform all tasks, and if the SD write takes up a lot of time, there will be a problem regardless of whether or not you interleave the tasks.

Someone mentioned FRAM, and that's a pretty good way to do it, I think, but as pointed out, it's a tad pricey. The only real reason to use FRAM is for its non-volatility. If you want to log to RAM, why not consider a static RAM. Yes, it's volatile, but unless you plan on gathering the messages, powering off, then powering on and offloading the data, it isn't really a problem.

I'd probably look at static RAM, byte-wide, for gathering the messages, an SD card for offloading after the test run, and if you still find it a bit slow, you could go for an Arduino Due. With its 96K bytes of RAM, you might not even need an external SRAM. Further, the SD card can be in a USB adapter, to attach to the OTG-capable USB port for offloading, and of course, it runs at 84 MHz, and is Arduino compatible at the C++ level.

The ones from China on Ebay tend to be less than 3 times the price of an Uno. Should be a similar ratio for the US ones.

20

If the bottleneck is writing to the SD card you could use the approach used by the hardware Serial driver, and handle the data reception entirely within interrupts. That would leave the main context free to do blocking writes to the SD card. As long as the SD card can keep up with the sustained data rate and you don't get any bursts that exceed your available SRAM, you should be OK. Getting that working is going to be quite an advanced project.

I don't know how hard it would be to rewrite the SD library to support asynchronous operation, but it might be possible. This would also be an advanced project.

A cheap and cheerful option would be to spew the data out over a serial port, and connect that to an ArduLog to capture it to an SD card. Its still going to be a performance challenge, but avoids you needing to write your own interrupt based implementation.