Emulating read serial data.

Hi all,

I'm writing some code that will capture some serial data that is a series of variables that my cars ECU spits back— I then need to chop this up so I can access various things like temperature, RPM etc

I have captured this data using some software— and I think it looks right?

02d2 0000 0000 0000 0000 0000 9393 0101
03df 03e0 02dc 0301 8401 d378 00c7 0000
0000 03e8 03e8 03e1 0064 0000 0064 03ed
0064 0064 0064 00a0 0000 fffa 0002 0020
0000 03e0 0064 0000 0064 03fe 0000 0000
0000 00e6 0000 009c 0000 03e0 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0064 0000 03e0 03e0 0000 0000 0000 0000
0000 0000 0000 0000 0000 0003 c700 0000
0000 0000 0000 0000 0000 0000 03fe 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 03e8 03e8
03e8 03e8 03e8 03e8 03e8 03e8 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000

Here's how it's chopped up: http://home.comcast.net/~whaussmann/RS232_MS2E/RS232_RTvar.htm

What I want to do is get that data in to an array so that I can play with it— it's not practical for me to sit in the car whilst I try out the various things I want to do with it.

What's the easiest way for me to get that in to an array? By hand? The data looks like it's in pairs of hex values? I am not sure.

James

If you place the data in a array (unsigned character array), and make a structure for the car data, you can make of pointer of the structure and point it to the array. Or use memcpy.

A structure is the best solution, since the car data mixes 8-bit and 16-bits variables.

But I can't match the captured data with the car data. They don't seem to be alike.

Are you wanting to work with this data in some other platform, or are you hoping to analyze it on the Arduino? I'm thinking the former, because analyzing this many real-time values, with floating-point scales and offsets, will be challenging for an AVR.

A quick way to get it into a C array would be to paste it into a text editor, and search and replace a single space for ",0x". Then add some more "0x"'s at the beginning of the lines if you have to, delete the trailing comma if it appears, put some brackets around it and declare an unsigned integer array array. The same thing will work for a Perl list, except you'd enclose it in parentheses.

You'll be left with an array of 16-bit integers, some of which will describe integer values, some will describe two single-byte values, some will describe part of a 32-bit value, and some will be in bits. You have a big clerical job ahead of you to extract meaningful data from all of it.

The previous poster notes that the data don't entirely make sense viewed in the framework of the web page you point to. I agree. I think that I can extract that the data was taken a few seconds after starting the digital device, that it was a pleasant 23 degrees C outside that day, the coolant temperature was only a few degrees above ambient, the car wass about 100 to 150 meters above sea level, and the engine was off. After that, I can't really find much correspondence. The map shows 154 bytes; your response is 408 bytes. Something's different. I'm thinking that the webpage describes an older version of Megsquirt 2, and you're interrogating a Megasquirt 3 controller. You don't have the right map.

All I want to do is cherry pick a couple of values and fire them out to a display via SPI — so I'll only be calculating on two at most.

I was hoping to split it in to bytes so that it's exactly as it is when I read it in via the serial port. I guess I'll have to do that by hand.

I've realised now that the MS3 which I have has a much bigger output — 408 bytes exactly as you say. The first set of values are all the same though.

You got all the values right — 23 degrees outside temp and the engine was 30 degrees as it was a few hours after it had been run.

I have since found a library that will work with the MS — but it's a bit heavy for what I want to do and as with everything this is a learning thing for me. I'm also planning to use SoftwareSerial for this too. It does give me the code that combines the bytes to form the 16-bit integers so that's a plus point.

JamesCarruthers:
I was hoping to ... I guess I'll have to do that by hand.

That sounds a little whiny. You're programming a microcontroller in C now, dude - the sport of kings. There's no room for that kind of talk here.

The goal is to convert a series of ASCII hexadecimal characters, along with some whitespace, into an array of bytes. A low-level way to do it is to examine each character in turn, and decide whether or not it's a hex digit. If it's not, ignore it; if it is, then convert it to binary. A low-level way to convert it to binary is to examine it to find out whether it's a decimal digit, and convert it in an appropriate fashion; if it's not, it's a hex digit, and it gets converted a different way. When the first half of a byte is converted, stash it in some temporary place, get the second half of the byte, and combine them into one byte. Do that until all the characters are gone. Voila - 408 bytes.

... this is a learning thing for me.

That's good to know. It means that you don't want us to write your code for you.

A few things could go wrong. If you collect the whole reply, and then start to process, you've burned almost a kilobyte of RAM - half of the RAM in the Arduino. If you then store 408 bytes someplace else, that's around 70% of your RAM gone, before you start processing data. I see a couple of ways to manage this:

  • Process the serial data on the fly, as it becomes available. The Megasquirt 3 appears to support baud rates up to 115200. Guessing ten bytes per character, that means that a byte comes in in about 87 microseconds, or about 1388 machine cycles. That sounds like more than enough time to process the incoming bytes in to an array
  • Store the whole message, and then store the data bytes right on top of it. Each data byte comes from an average of more than two characters, so the data won't overrun the message. It looks like you don't need to keep both things in memory at once, so that could work. If you want to fetch new data while still using old data, though, this plan isn't the right one.

If you process on the fly, and something else takes too long, it could be tricky to troubleshoot. But it sure leaves a lot more RAM available, and I can't see a good reason for keeping the ASCII around once the byte array exists. Running out of RAM is tricky to troubleshoot, too.

Because this is a learning thing for you, here's a treat: you might find the meaning of the additional bytes in array at this url: Megasquirt Downloads Firmware and Software - Megasquirt EFI. A portion of the file "ms3.ini" seems to describe it in a way that I can't fully fathom. But, it sure looks a lot like something that somebody did "text to columns" on in Excel to get the map you showed us, and it maps out to 408 bytes. Extracting it may be a bit of a challenge.

I am not sure if I am going to repeat what someone has to say but, here goes. You probably don't want to know every single bit of information that is being spit out of the ECM. Therefore you only need to define the variables that you care about.

barometer S16  16 0.1, 0 kPa real time barometer for altitude correction

For the barometer a 16bit value, you are going to have a High byte and a Low byte, the bytes are at index 16 and 17. You should use a counter with serial.read and when you get to values 16 and 17, save them to capture your barometer information. You can use or discard any of the stream that is returned by counting the bytes as they come in threw the serial port.

A lot of times there is a math formula to get your units to come out correctly, your barometer bytes may be actually KPA reading and not require math, you will have to find that yourself.

I have a project that is somewhat similar, here is just a snippet of my code that read serial data returned and processes it. It will not compile but, it should give you some food for thought.

 if (Serial1.available()) 
    {
      static int counter = 0;
      byte inByte = Serial1.read();
      buf[counter] = inByte;
      counter++; 
      /////////////////////////////////READ RPM, BATTERY, AND THROTTLE/////////////////////
      if (counter == messagelength)     // adjust to index size expected
      {
        
         RPM = (((buf[rpm1] * 256) + buf[rpm2]) / 2.56);   //example of HIGH byte and LOW byte addition////////
        Battery = (buf[BATT] / 12.7);
        Throttle = ((buf[TPS] - 55)*125 / (256-55)); 
  }
}

Thank you both for your replies — certainly no whining for me.

The serial data I posted has been converted to hex by the terminal software — the serial data returned is a load of bytes as you'd expect. So the whole lot is received in about 3.5ms?

It's a great idea to do the maths as it goes and to just pick out the values I need rather than storing it in RAM and then going through it afterwards — the cycle to go through the read data can be cut short once I've found the two values I want to display. Though I guess the reality of this is wether or not me adding the if statements slows it down. However I don't think it'll be an issue for what I want to do with it.

I've found the portion of the MS3.INI which defines the byte offset and scalar and translate values, it's roughly the same as the table I posted. I don't need or want to use most of the additional values but a few are useful none-the-less. (otherwise I would have just bought an MS2 and not a MS3, it's cheaper)

There also seems to be a different way of asking for the data if you only want a portion of it.

;full version that works anywhere - be sure to change the no. bytes requested
   ochGetCommand    = "r\x00\x07\x00\x00\x01\x98"

Thanks for all your help — I am waiting for a couple of components to finish off the hardware and then I'm good to start writing the code properly.