Struggling with my BMW IBUS projects. Serial.read related issues.

Projects General Guidance
41

Riva:
Does some device on the bus send the same message on a regular basis ?

You get status request messages sent out periodically to check that that modules are still active, but nothing more frequent than 3 second intervals. You could potentially miss an awful lot of traffic in that period. I guess it's still good practice to try and include some sort of error recovery though.

My problem at the moment is that when it gets out of sync, it quite often causes the arduino to reset, or simply becomes unresponsive, so it's hard to debug. I don't know for sure, but think it's the IBUSbyte array getting full and over writing into program memory. I've increased the array size to 256 bytes to allocate more space, but it still locks up during busy times on the bus. I guess I could make it even bigger, but assumed the biggest it would ever get is when it reads a length byte of 0xFF and tries to read in 255 bytes. Actually, 256 bytes isn't quite big enough. It will already have read in the source and length bytes, so the array could potentially get 257 bytes. I'll increase the array size to 300 just to be sure.

This is one of the reasons I would prefer to not use the length byte to dictate how many bytes to read in before flagging the message as complete. If for any reason the length byte is wrong, the whole system falls over because you start reading in bytes from other messages etc.

Thanks,

Ian.

42

ian332isport:
as I can't figure out how to measure the idle time.

I've just looked back st some code you posted (to refresh my memory). I think you have just been using Serial.read() to get the IBus data.

I would try to measure the time between Serial.available() > 0 and the next time it is > 0. If the gap is above X then it represents an idle period. I'm not sure if this would be reliable but it seems the simplest place to start. If that fails you probably need to write some interrupt routine which is inevitably more complex. My idea is something like this

prevByteMicros = micros();
if Serial.available() > 0 {
   curByteMicros = micros();
   if (curByteMicros - prevByteMicros >= blankInterval) {
      newMessage = true;
      prevByteMicros = curByteMicros;
   }
   else {
     newMessage = false;
   }
}

...R

Edit to change millis to mcros as the interval is short. ...R

43

Robin2:
I would try to measure the time between Serial.available() > 0 and the next time it is > 0. If the gap is above X then it represents an idle period. I'm not sure if this would be reliable but it seems the simplest place to start. If that fails you probably need to write some interrupt routine which is inevitably more complex. My idea is something like this

prevByteMicros = micros();

if Serial.available() > 0 {
  curByteMicros = micros();
  if (curByteMicros - prevByteMicros >= blankInterval) {
     newMessage = true;
     prevByteMicros = curByteMicros;
  }
  else {
    newMessage = false;
  }
}




...R

Edit to change millis to mcros as the interval is short. ...R

Hi Robin,

Thanks very much for this idea. I've already been playing around with similar code, but still no joy.

One thing that I think will stop this working is that you have to wait for the next Serial.available() > 0 to stop the timer. As soon as we get this far, surely it's already too late as the next message is already in the buffer. If we could start a timer when Serial.not available, that may work. If Serial.not available for more than 900 micro seconds, that would indicate an idle period and give enough time to process the message before the next one arrives.

Thanks,

Ian.

44

I think my idea will work. It will measure the time from the start of the last byte of the previous message to the start of the first byte of the new message. You will probably need a bit of experimenting to find the correct interval. (My "interval" will be the time for one byte plus the real interval).

It doesn't matter if the new message has already started to arrive. All you need to know is that byte X is the first byte and you will know that before you do Serial.read().

I'm assuming your code reads bytes one at a time.

...R

45

Hi Robin,

Where would you put the Serial.read() command in that ?

Thanks,

Ian.

46

Probably after the if (Serial.available > 0) { }

I think I would use "newMessage" being true to reset the index for where the bytes are stored and to restart the process of interpreting the data - for example to find the length byte.

Perhaps you can put something together and post it for review.

...R

47

This is what I'm trying at the moment. Initially I didn't think it was doing anything, but when I dropped the blankInterval down below 10 I started to get random newMessage results. With blankInterval set at 4, I just get a continual stream of 'New Message' in the serial monitor. blankInterval between 5 and 15 seem to just give random 'New Message' or 'read next byte' and 15 and above just gives 'read next byte' for each HEX byte that is sent.

I'm not quite sure what this is telling us, and I don't understand why the blankInterval needs to be so low before it starts to react.

#include <SoftwareSerial.h>

//Constant Definitions - there won't change.

SoftwareSerial mySerial(7, 8); // 7 is Rx, 8 is Tx
const int CS_LWAKE = 4; // HIGH enables MCP2025 chip
byte IBUSbyte[300];

//Variable definition - these can be changed by the program.

int i = 0;
int blankInterval = 10 ;
unsigned long prevByteMicros;
unsigned long curByteMicros;
boolean newMessage;

void setup()                     
{ 
  Serial.begin(9600, SERIAL_8E1);
  mySerial.begin(9600);
  mySerial.println("--IBUS receive test--");
  pinMode (CS_LWAKE, OUTPUT); // initialize pin.
  digitalWrite (CS_LWAKE, HIGH); // write pin high.


} 

void loop() 
{ 
  prevByteMicros = micros();
  if (Serial.available() > 0) {
    curByteMicros = micros();
    if (curByteMicros - prevByteMicros >= blankInterval) {
      newMessage = true;
      prevByteMicros = curByteMicros;
    }
    else {
      newMessage = false;
    }
    if (newMessage == true) {
      mySerial.print("NewMessage");
    }
    else if (newMessage == false) 
    {
      mySerial.println("read next byte ");
      ReadIBUS();
    }
  }

}// end of main loop 

void ReadIBUS()
{
  IBUSbyte[i++] = Serial.read();
}
48

Robin2:
Probably after the if (Serial.available > 0) { }

I think I would use "newMessage" being true to reset the index for where the bytes are stored and to restart the process of interpreting the data - for example to find the length byte.

Perhaps you can put something together and post it for review.

...R

Ooops, crossed posts.

I'll move things around and have another try.

Ian.

49

Robin2:
Probably after the if (Serial.available > 0) { }

I think I would use "newMessage" being true to reset the index for where the bytes are stored and to restart the process of interpreting the data - for example to find the length byte.

Perhaps you can put something together and post it for review.

...R

If I put ReadIBUS() after the if (Serial.available > 0) { } (assuming you mean completely outside the if statement), it just continually loops through ReadIBUS() (as you would expect being in the main loop). If I put it directly after if (Serial.available > 0) { I get the same results as before.

void loop()
{
prevByteMicros = micros();
if (Serial.available() > 0) {
ReadIBUS();
curByteMicros = micros();
if (curByteMicros - prevByteMicros >= blankInterval) {
newMessage = true;
prevByteMicros = curByteMicros;
}
else {
newMessage = false;
}
if (newMessage == true) {
mySerial.println("New Message ");
i = 0;
}
else if (newMessage == false)
{
mySerial.println("read next byte ");

}

}

}// end of main loop

50

Hi, Ian,

I've been thinking a bit more about this. It seems to me that (assuming the gap between messages is the correct delimiter) the gap should be all that needed to separate messages. If so this code should work. It should save 8 messages and then start overwriting them so it always has the most recent 8. For testing, perhaps you could modify loop() so it stops calling getIBusStuff() after 8 messages have been read. Obviously this isn't tested.

Because the interval is in microsecs very short numbers will be completely meaningless.

ibusInterval = 1800; // microsecs
byte ibusData[8][32];  // 8 rows of 32 bytes - assumes max message is 32 bytes
byte ibusRow = 7;
byte ibusCol = 0;


void loop() {
  getIBusStuff();
  // other stuff;
}

void getIbusStuff() {
  if Serial.available() > 0 {
    curByteMicros = micros();
    if (curByteMicros - prevByteMicros >= blankInterval) {
      prevByteMicros = curByteMicros;
      startNewIbusMessage();
    }
    byte inByte = Serial.read();
    saveIbusByte(inByte);
  }
}

void startNewIbusMessage() {
  ibusRow ++;
  if (ibusRow > 8) {
    ibusRow = 0;
  }
  ibusCol = 0;
}

void saveIbusByte(byte inb) {
  ibusData[ibusRow][ibusCol] = inb;
  ibusCol ++;
  if (ibusCol > 31) {
    ibusCol = 31;
  }
}

...R

Edit ... the above code has errors. There is a working version in Reply# 56, below ...R

51

Massive thanks Robin.

I'll give it a try this evening and report back.

I've also been looking at Serial.readBytes(). It appears to read bytes into a buffer until it has the number required, or a timeout is reached. If I set the number required high and rely on the timeout it may work. Unfortunately it uses millis() not micros() for the timeout. I'm currently having a fiddle with the Stream.cpp file to see if I can change it to work with micros(). This looks to be the bit of code responsible.

// private method to read stream with timeout
int Stream::timedRead()
{
  int c;
  _startMillis = millis();
  do {
    c = read();
    if (c >= 0) return c;
  } while(millis() - _startMillis < _timeout);
  return -1;     // -1 indicates timeout
}
52

As far as I know Serial.readbytes() is a a blocking function and I don't see any advantage in using it with a data stream as slow as 9600 baud.

The timeout would need to be long enough to wait for the whole of a long message so I think it would get in the way if you inadvertantly start sampling in the middle of a message.

One of the things I like about my suggestion (always assuming it works :slight_smile: ) is that it deals with all bytes with the same code, doesn't require special cases and will jump to start a new message whenever the delimiting gap occurs.

Figuring out the content can come later after the message has been received.

...R

53

I reckoned I might be able to write a short PC program to simulate thee output of the IBUS system in a very basic way - just sending a few strings of different lengths with a gap between them - to test my Arduino code. Not surprisingly there were a few silly mistakes in the code I posted earlier. Following is a complete sketch that seems to work. I have written it for a Mega because it has multiple hardware Serial ports. If you want to write to the Serial Monitor with SoftwareSerial just replace the references to Serial1 with whatever you call your instance of SoftwareSerial.

In case others are interested I had the Mega connected to the PC through the normal USB cable and I used an FTDI cable to connect Rx, Tx and GND for Serial1 to another USB connection on the PC. That allowed me to view the output through the FTDI cable on the Serial Monitor while another program (written in JRuby) sent data to the Mega over the normal USB cable.

unsigned long ibusInterval = 1800; // microsecs
unsigned long prevByteMicros = 0;
unsigned long curByteMicros = 0;

byte ibusData[8][32];  // 8 rows of 32 bytes - assumes max message is 32 bytes
byte ibusRow = 7;
byte ibusCol = 0;
int recentMessage = -2;
int messageWritten = -1;

void setup() {
  Serial.begin(9600);
  Serial.println("Serial0 works");
  
  Serial1.begin(9600);
  Serial1.println("Serial1 works");


}


void loop() {
  getIbusStuff();
  printIbusStuff();
  // other stuff;
}

void getIbusStuff() {
  if (Serial.available() > 0) {
    curByteMicros = micros();
    if (curByteMicros - prevByteMicros >= ibusInterval) {
      startNewIbusMessage();
    }
    prevByteMicros = curByteMicros;
    byte inByte = Serial.read();
    saveIbusByte(inByte);
  }
}

void startNewIbusMessage() {
  ibusRow ++;
  if (ibusRow > 7) {
    ibusRow = 0;
  }
  ibusCol = 0;
  recentMessage ++;
  if (recentMessage > 7) {
    recentMessage = 0;
  } 
}

void saveIbusByte(byte inb) {
  ibusData[ibusRow][ibusCol] = inb;
  ibusCol ++;
  if (ibusCol > 31) {
    ibusCol = 31;
  }
}

void printIbusStuff() {
  if (recentMessage >= 0) {
     if (recentMessage != messageWritten) {
        Serial1.print("Msg ");
        Serial1.print(recentMessage);
        Serial1.print("  ");
        Serial1.write(ibusData[recentMessage], 15);
        Serial1.println();
        messageWritten = recentMessage;
     }
  }
}

...R

54

Hi Robin,

Right, I've loaded your program into the Arduino and after a couple of small changes to the Serial setup (need 8E1 for IBUS), I was ready for testing. As soon as I power up, I see the 'Serial Works' prompt. If I send one 6 byte message (50 04 68 32 11 1F) I get nothing on the serial monitor. I'm assuming this is the program getting in sync with the gap between bytes.
If I send the same 6 byte message I get:

Msg 0 Ph2
Msg 1 P
Msg 2
Msg 3 h
Msg 4 2
Msg 5

If I send the same message again, I get this:

Msg 6
Msg 7 P
Msg 0 h2
Msg 1 h
Msg 2 2
Msg 3

and the same again:

Msg 4
Msg 5 P
Msg 6
Msg 7 h
Msg 0 2h2
Msg 1

It looks like the same characters, but the order is a little mixed up. I'm also assuming these are ASCII which explains some of the gaps. Looking at the ASCII table, 0x04 is EOT and 0x11 is DS1 etc. Is it possible to output the bytes in hex ?

I've had a play with the interval and 900 seems to be a bit more reliable, but it's hard to see exactly what's coming in while it's ASCII. These are consecutive 6 byte messages (same as above). Note that the first message comes in straight away without ignoring the first message. Also note that the first message is only receiving 5 bytes, not the 6 that were sent. It looks to be 6 from there on in though.

Msg 0 P
Msg 1
Msg 2 h
Msg 3 2
Msg 4

Msg 5
Msg 6 P
Msg 7
Msg 0 h
Msg 1 2
Msg 2

Msg 3
Msg 4 P
Msg 5
Msg 6 h
Msg 7 2
Msg 0

Msg 1
Msg 2 P
Msg 3
Msg 4 h
Msg 5 2
Msg 6

It certainly looks like you're onto something though.

55

ian332isport:
If I send one 6 byte message (50 04 68 32 11 1F) I get nothing on the serial monitor.

Yes, this is a DEMO program - not a finished work :slight_smile:
You need to send at least two messages for there to be a gap between messages.

How are you generating and sending the messages? - I may be able to improve my PC program
Can you try just sending printable characters for testing?

Yes you can get it to print the byte values rather than the characters but this late at night I can't remember how. Maybe you need to go through the array byte by byte and use Serial.print(nnn, HEX).

...R

56

Hi Robin,

I fully understand this is only DEMO software. I really do appreciate your help with this and I'm certainly not expecting a fully finished program.

I'm using a piece of software called Navcoder to send messages ( http://www.navcoder.com/ ). You can try the free version, but I don't recall if the message tester or Steering wheel emulator work without registering the software.

I've never used it, but another option is IBUSAnalyser. I've put a copy in my Dropbox if you want to give it a try.

https://dl.dropboxusercontent.com/u/100050743/I-BUSDecoder1008SRC.zip

I'll try sending regular characters and see what happens. Probably won't be until tomorrow evening now.

Thanks again,

Ian.

57

I meant to say earlier that you seem to have a mania for reducing the interval (you said 900 usecs) whereas my instinct would be to increase it. :slight_smile:

I'm not sure I'm going to bother with NavCoder but I might write a sketch to send data from another Arduino as I could control it more closely than I can with my PC software.

...R

58

I've spent a bit more time on my simulation. I've written a short program for an Uno that I use to send the data to the Mega in preference to sending the same data from my PC. Using the Arduino gives me better control of the timing.

After a great deal of fiddling around (which was eventually solved by using Serial.flush() on the Uno) the Mega now seems to receive data reliably with a gap of 2 millisecs between messages. The only change to the code I posted most recently was to comment out 3 of the print statements to speed things up.

     if (recentMessage != messageWritten) {
     //   Serial1.print("Msg ");
     //   Serial1.print(recentMessage);
     //   Serial1.print("  ");
        Serial1.write(ibusData[recentMessage], 15);
        Serial1.println();
        messageWritten = recentMessage;
     }

I venture to suggest that if this code doesn't work with your real data it is because we still don't know how the real system works.

...R

59

Hi Robin,

I've set the delay back to 1800 :smiley: and commented out the Serial.print lines as you suggested. Unfortunately I'm getting the same results as before when sending real data from Navcoder. I've also tried sending data (regular ASCII characters) directly from the Arduino serial monitor window, but don't get anything sensible or repeatable. I don't have any control over the gap between bytes when using the serial monitor though, so that may be the problem.

I've been in touch with the guy behind an existing IBUS interface device and he has confirmed that the gap between messgaes is the correct method. This is the info he provided.

Detecting the inter packet gap is they only way. Once in a while you will get a corrupt message, so you need to trap the various possibilities :

Good packet, length matches actual length.
Short packet, inter packet gap comes before 'length' bytes received.
Over run, more then length bytes received, probably caused by misinterpreting the length byte.
Invalid CRC, corruption within the packet.

You can also have a sanity check on the length byte, the maximum data length is 31 bytes, so the maximum possible valid iBus message is:

Source Byte, Length Byte, Dest Byte, Function Byte, 31 Data bytes, Check Sum byte.

There is some ambiguity in the maximum length of the data, some say 32 bytes + function, some say that the 32 bytes includes the function byte. Some third party devices send 32 data bytes, but this can cause problems with some BMW devices, for example the electric mirrors on the E46 can start to go up and down if you send 32 data bytes!! But if you see a length of over 36, you can be sure it's wrong!

If you work on a minimum inter packet gap of 1.5 milliseconds you should be OK, but I've found that it's best to wait for 10 milliseconds before sending any message to ensure that the bus has time to process messages from other devices.

I'll have a play with the code and see if I can get it to do anything sensible with Navcoder messages.

Thanks,

Ian.

60

Hi Ian,

I had a look at the Navcoder link. It's a Windows program so that rules that out.

At the risk of sounding like I am repeating myself (from a few pages back) I think it would be worth while writing a short sketch that captures (say) 100 bytes from Navcoder together with the intervals between each of them (in Micros) and then displays all on the Serial Monitor.

Do you have any way (separate from your Arduino sketch) to prove that the output from Navcoder complies with the IBUS timing requirements. I ask, because with serial buffers and the USB system the timing might go a bit wonky.

I'm assuming you only have one Arduino. If you have a second one I can post the code I transmit to demo the code I have already posted.

...R