serial communication data corruption

Projects Networking, Protocols, and Devices
1

i use serial communication between arduino mega and netduino
the data i receive is fragmented or merged also sometimes corrupted

a packet dump which were sent by the mega

Packet length : 8
Packet destination : netduino
Packet source : mega
Packet Type : Ping
Packet CheckSum : 11
0x8, 0x1, 0x2, 0x0, 0xB, 0x0, 0x0, 0x0

0 218 701 713 195 82 518 1023 833 583 367 799 1023 736 343 500
Packet length : 72
Packet destination : netduino
Packet source : mega
Packet Type : Analog update
Packet CheckSum : 2335
0x48, 0x1, 0x2, 0x1, 0x0, 0x0, 0x0, 0x0, 0xDA, 0x0, 0x0, 0x0, 0xBD, 0x2, 0x0, 0x0,
0xC9, 0x2, 0x0, 0x0, 0xC3, 0x0, 0x0, 0x0, 0x52, 0x0, 0x0, 0x0, 0x6, 0x2, 0x0, 0x0,
0xFF, 0x3, 0x0, 0x0, 0x41, 0x3, 0x0, 0x0, 0x47, 0x2, 0x0, 0x0, 0x6F, 0x1, 0x0, 0x0,
0x1F, 0x3, 0x0, 0x0, 0xFF, 0x3, 0x0, 0x0, 0xE0, 0x2, 0x0, 0x0, 0x57, 0x1, 0x0, 0x0,
0xF4, 0x1, 0x0, 0x0, 0x1F, 0x9, 0x0, 0x0

an example of what i recieve

[06/01/2011 00:00:11 Undetermined->Undetermined]Packet Length : 72 From : 2 To : 1 with type : 1 and actual len : 82
[06/01/2011 00:00:12 Undetermined->Undetermined] 0x48 0x01 0x02 0x01 0x82 0x02 0x00 0x00 0x79 0x02 0x00 0x00 0x81 0x02 0x00 0x00 0x46 0x02 0x00 0x00 0x1E 0x02 0x00 0x00 0xFE 0x01 0x00 0x00 0xE9 0x01 0x00 0x00 0xC0 0x02 0x00 0x00 0x0F 0x03 0x00 0x00 0x49 0x03 0x00 0x00 0xC1 0x02 0x00 0x00 0x93 0x02 0x00 0x00 0xAF 0x02 0x00 0x00 0x96 0x02 0x00 0x00 0x82 0x02 0x00 0x00 0x4A 0x02 0x00 0x00 0xB0 0x08 0x00 0x00
[06/01/2011 00:00:16 Undetermined->Undetermined]Packet Length : 72 From : 2 To : 1 with type : 1 and actual len : 50
[06/01/2011 00:00:16 Undetermined->Undetermined] 0x48 0x01 0x02 0x01 0x07 0x02 0x00 0x00 0x0F 0x02 0x00 0x00 0x04 0x02 0x00 0x00 0x43 0x02 0x00 0x00 0x48 0x02 0x00 0x00 0xD3 0x01 0x00 0x00 0xC5 0x01 0x00 0x00 0x97 0x02 0x00 0x00 0xFD 0x02 0x00 0x00 0x3D 0x03 0x00 0x00 0x06 0x03 0x00 0x00 0x0D 0x03
[06/01/2011 00:00:16 Undetermined->Undetermined]Packet Length : 0 From : 26 To : 0 with type : 3 and actual len : 32
[06/01/2011 00:00:16 Undetermined->Undetermined] 0x00 0x00 0x1A 0x03 0x00 0x00 0xDC 0x02 0x00 0x00 0x0A 0x03 0x00 0x00 0x23 0x03 0x00 0x00 0xB4 0x05 0x00 0x00 0x08 0x01 0x02 0x00 0x0B 0x00 0x00 0x00 0x0D 0x0A
[06/01/2011 00:00:21 Undetermined->Undetermined]Packet Length : 72 From : 2 To : 1 with type : 1 and actual len : 46
[06/01/2011 00:00:21 Undetermined->Undetermined] 0x48 0x01 0x02 0x01 0x21 0x02 0x00 0x00 0x19 0x02 0x00 0x00 0xFC 0x01 0x00 0x00 0x27 0x02 0x00 0x00 0xBB 0x01 0x00 0x00 0x98 0x01 0x00 0x00 0x80 0x01 0x00 0x00 0x43 0x02 0x00 0x00 0x9F 0x02 0x00 0x00 0xF7 0x02 0x00 0x00 0x51 0x02
[06/01/2011 00:00:21 Undetermined->Undetermined]Packet Length : 0 From : 35 To : 0 with type : 2 and actual len : 36
[06/01/2011 00:00:21 Undetermined->Undetermined] 0x00 0x00 0x23 0x02 0x00 0x00 0x1C 0x02 0x00 0x00 0xC9 0x01 0x00 0x00 0xE8 0x01 0x00 0x00 0xC9 0x01 0x00 0x00 0x78 0x08 0x00 0x00 0x08 0x01 0x02 0x00 0x0B 0x00 0x00 0x00 0x0D 0x0A
[06/01/2011 00:00:26 Undetermined->Undetermined]Packet Length : 72 From : 2 To : 1 with type : 1 and actual len : 82
[06/01/2011 00:00:26 Undetermined->Undetermined] 0x48 0x01 0x02 0x01 0x0F 0x02 0x00 0x00 0x43 0x02 0x00 0x00 0x36 0x02 0x00 0x00 0x0C 0x02 0x00 0x00 0xFD 0x01 0x00 0x00 0xE0 0x01 0x00 0x00 0xD4 0x01 0x00 0x00 0xA8 0x02 0x00 0x00 0x1D 0x03 0x00 0x00 0x40 0x03 0x00 0x00 0x0A 0x03 0x00 0x00 0xFD 0x02 0x00 0x00 0x20 0x03 0x00 0x00 0xE8 0x02 0x00 0x00 0x17 0x03 0x00 0x00 0x2C 0x03 0x00 0x00 0x0B 0x07 0x00 0x00 0x08 0x01 0x02 0x00 0x0B 0x00 0x00 0x00 0x0D 0x0A

notice that at the first packet i receive the packet is pretty much fine but i've 10 extra bytes
notice the second packet were i received only 50 bytes where i was supposed to recieve 72 but merging it with the third packet that makes 2 full packets of 72 and 8 with 2 extra bytes in the end

2

code? for both sides.
How is hardware connected?

3

tx1-rx1 tx2-rx2

arduino
i dump the packet before sending it so ik for sure it's fine before sending
my setup Serial.begin(115200,SERIAL_8N2);
i send with Serial.write(packet,packet[0]); //offset 0 is packet length

netduino
setup new SerialPort(SerialPorts.COM2, 115200, Parity.None, 8, StopBits.Two);
handler

              while (Serial.BytesToRead > len)
                {
                    var buf = new byte[len - 1];
                    Serial.Read(buf, 0, 1);
                    for (int i = 0; i < len; i++)
                        result.Add(buf[i]);
                }
                if (result.Count > 0)
                {
                    HandleResponse((byte[]) result.ToArray(typeof (byte)));
                }

point is, i use same exact code on netduino with finger print module, emic2, AVR shield and others and it's working correctly, so :\

4

okay i've written the code like 10 times and it still keeps screwing up and rarely is delivered as is
i've tried using a common ground, changing the code with custom queue for fragmentation in netduino due to the recv. buffer timeout (which should not be reached if the mega is sending with constant good time frame), using slower baudrate, using one or two stop bits)

i really need to get this part done to get to the rest of the project, if anyone have anything in mind for me to try then please lemme know, even if it sound stupid, i'm out of ideas

5

"Tried" a common ground? It won't work at all without a ground connection.

while (Serial.BytesToRead > len)

How big is this len? 72 bytes for the biggest packet? You know that's bigger than the serial buffer on most Arduinos? This may never become true. I don't know about the Netduino - it may have a bigger buffer. It's still poor practise to let the hardware buffer fill up like that. You should be processing the incoming chars into your own buffer as soon as they arrive, one at a time.

6

they have same supply so i guess that already men they have a common ground and hence it worked but still bugged

yeah i rewrote that in a custom queue where i buffer the bytes as i receive them on netduino (no documentation on that and the property is accessible but throw exception when you try to change it, that ain't even the problem)

here is a static small packet 0x06 0xF1 0xF2 0xF3 0xF4 0xF5 to test with, here is what i receive

[06/01/2011 00:00:13 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 241 and actual len : 13
[06/01/2011 00:00:14 Undetermined->Undetermined] 0x0D 0x0A 0x06 0xF1 0xF2 0xF3 0xF4 0xF5 0x0D 0x0A 0x06 0xF1 0xF2
[06/01/2011 00:00:14 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 13 and actual len : 13
[06/01/2011 00:00:14 Undetermined->Undetermined] 0x0D 0x0A 0x06 0x0D 0x0A 0xF1 0xF2 0xF3 0xF4 0xF5 0x0D 0x0A 0xF1
[06/01/2011 00:00:31 Undetermined->Undetermined]Packet Length : 242 From : 244 To : 243 with type : 245 and actual len : 242
[06/01/2011 00:00:32 Undetermined->Undetermined] 0xF2 0xF3 0xF4 0xF5 0x0D 0x0A 0xF1 0xF2 0xF3 0xF4 0xF5 0x0D 0x0A 0xF1 0xF2

that 0D 0A is being sent so much that i doubt that is noise, so yeah i receive the packet but there is something wrong that screws everything up
here is the new code !

mega

const byte testarray[] = {0x06,0xF1,0xF2,0xF3,0xF4,0xF5};
and inside the loop
Serial.write(testarray,6);
delay(500);

netduino

            Serial.DataReceived += (sender, args) =>
            {
                incomingfirstbyte = (byte)Serial.BaseStream.ReadByte();
                while (incomingfirstbyte < 8 && incomingfirstbyte != 0x0D && incomingfirstbyte != 0x0D)
                {
                    if(Serial.BytesToRead < 1) return;
                     incomingfirstbyte = (byte)Serial.BaseStream.ReadByte();    
                }
                SerialQueue.push(incomingfirstbyte);
                var fragment = new byte[Serial.BytesToRead];
                Serial.Read(fragment, 0, fragment.Length);
                SerialQueue.push(fragment);
                if (SerialQueue.CanPop()) HandleResponse(SerialQueue.pop());
            };

and here is the custom queue

public class CostumeQueue
    {
        private int pointer = 0;
        private readonly int size;
        private readonly byte[] internalByteArray;

        public CostumeQueue(int size)
        {
            this.size = size;
            internalByteArray = new byte[size];
        }

        public void push(byte[] fragment)
        {
            var len = fragment.Length;
            if (CanPush(len))
            {
                Array.Copy(fragment,0,internalByteArray,pointer,len);
                pointer += len;
                return;
            }
            throw new Exception("there isn't enough space to push");
        }

        public void push(byte thebyte)
        {
            if (CanPush(1))
            {
                internalByteArray[pointer] = thebyte;
                pointer++;
                return;
            }
            throw new Exception("there isn't enough space to push");
        }

        public byte[] pop()
        {
            if (CanPop(internalByteArray[0]))
            {
                var result = new byte[internalByteArray[0]];
                Array.Copy(internalByteArray,0,result,0,internalByteArray[0]);
                var temp = new byte[size];
                Array.Copy(internalByteArray,result.Length,temp,0,pointer-result.Length);
                pointer -= result.Length;
                Array.Copy(temp,internalByteArray,0);
                return result;
            }
                throw new Exception("there isn't enough queue to pop");
        }

        public bool CanPop(int len)
        {
            return pointer >= len;
        }

        public bool CanPop()
        {
            return pointer >= internalByteArray[0] && pointer > 4;
        }

        public bool CanPush(int len)
        {
            return size >= pointer + len;
        }
    }
7

i've been trying to optimize my code, gone back to fast baudrate with default settings, and no longer recieve the length of the first byte, i've end up with this
(everything is correct except those first 2 bytes)

[06/01/2011 00:00:29 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 241 and actual len : 8
[06/01/2011 00:00:29 Undetermined->Undetermined] 0x0D 0x0A 0x06 0xF1 0xF2 0xF3 0xF4 0xF5
[06/01/2011 00:00:29 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 241 and actual len : 8
[06/01/2011 00:00:29 Undetermined->Undetermined] 0x0D 0x0A 0x06 0xF1 0xF2 0xF3 0xF4 0xF5
[06/01/2011 00:00:30 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 241 and actual len : 8
[06/01/2011 00:00:30 Undetermined->Undetermined] 0x0D 0x0A 0x06 0xF1 0xF2 0xF3 0xF4 0xF5
[06/01/2011 00:00:30 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 241 and actual len : 8
[06/01/2011 00:00:30 Undetermined->Undetermined] 0x0D 0x0A 0x06 0xF1 0xF2 0xF3 0xF4 0xF5
[06/01/2011 00:00:31 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 241 and actual len : 8
[06/01/2011 00:00:31 Undetermined->Undetermined] 0x0D 0x0A 0x06 0xF1 0xF2 0xF3 0xF4 0xF5
[06/01/2011 00:00:31 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 241 and actual len : 8
[06/01/2011 00:00:31 Undetermined->Undetermined] 0x0D 0x0A 0x06 0xF1 0xF2 0xF3 0xF4 0xF5
[06/01/2011 00:00:32 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 241 and actual len : 8
[06/01/2011 00:00:32 Undetermined->Undetermined] 0x0D 0x0A 0x06 0xF1 0xF2 0xF3 0xF4 0xF5
[06/01/2011 00:00:32 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 241 and actual len : 8
[06/01/2011 00:00:32 Undetermined->Undetermined] 0x0D 0x0A 0x06 0xF1 0xF2 0xF3 0xF4 0xF5
[06/01/2011 00:00:33 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 241 and actual len : 8
[06/01/2011 00:00:33 Undetermined->Undetermined] 0x0D 0x0A 0x06 0xF1 0xF2 0xF3 0xF4 0xF5
[06/01/2011 00:00:33 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 241 and actual len : 8
[06/01/2011 00:00:33 Undetermined->Undetermined] 0x0D 0x0A 0x06 0xF1 0xF2 0xF3 0xF4 0xF5
[06/01/2011 00:00:34 Undetermined->Undetermined]Packet Length : 13 From : 6 To : 10 with type : 241 and actual len : 8
[06/01/2011 00:00:34 Undetermined->Undetermined] 0x0D 0x0A 0x06 0xF1 0xF2 0xF3 0xF4 0xF5
8

that 0D 0A is being sent so much that i doubt that is noise

0D 0A are the CR and LF characters. You don't have a stray Serial.println() somewhere in the sending code by any chance?

9

no, i also suspected that but it should send it in the end not in the begging but yes that's \r\n
i'm testing with the bare minimum to just see what's wrong and no i use serial.write and not println or writeln (if that even exists)

here
the definition
const byte testarray[] = {0x06,0xF1,0xF2,0xF3,0xF4,0xF5};

the usage inside the loop
Serial.write(testarray,6);
delay(500);

and there is nothing in the documentations mention anything about sending a \r\n at the very first of each call

but what really beaks my noodles is that the same code works with finger print module and emic module without the slightest problem, and those cart ret and new line isn't send anywhere, packets are sent perfectly even that i handle them poorly they never caused any problem

also you guys has been so helpful so i'm giving you all karma once this is solved :smiley:

10

On the Mega, you are using the serial monitor to check the packet contents before they are sent, is that right?

Which serial port are you using for connection to the Netduino?

11

okay i've rewrote everything that is related to it, and found an un-commented println which should have been commented, i'm seriously think i should commit suicide, sorry guys that i bothered you with this, that extra bytes screwed everything up

12

http://snippets-r-us.com/

Pete

13

andrewxxx:
okay i've written the code like 10 times and it still keeps screwing up and rarely is delivered as is
i've tried using a common ground, changing the code with custom queue for fragmentation in netduino due to the recv. buffer timeout (which should not be reached if the mega is sending with constant good time frame), using slower baudrate, using one or two stop bits)

i really need to get this part done to get to the rest of the project, if anyone have anything in mind for me to try then please lemme know, even if it sound stupid, i'm out of ideas

You do realize that the Serial library on the mega by default uses 64 byte buffers. If you send a packet of 75 bytes, the call Serial.print(bf[],bufsize); hangs until the first 11 bytes have been transmitted. Then the call returns while the H/W is sending the rest of the buffer. I have know knowledge of the limitations on the netDuino.

I think you need to implement some type of handshaking. CTS/RTS between the two devices. I implement it for Arduino 1.0.5. I haven't yet updated it for 1.6.5(still working thru all my library customizations).

Chuck.

Check out my Kickstarter Project Memory Panes an expansion RAM Shield for Mega2560's. It adds 1MB of RAM for those projects where 8KB is not enough.

14

i've seen that limitation in the software serial library and edited it, changed the types to int16 and made it 500 byte long but didn't know where to do the same for the serial library and not the software serial

so by allowing enough time for the h/w to send the rest of the buffer that should work ? it's pretty straight forward to calculate how much will it takes given the baudrate and the serial communication efficiency (priaty, stop bit etc..) and it should be fine, am i right ?

15

andrewxxx:
i've seen that limitation in the software serial library and edited it, changed the types to int16 and made it 500 byte long but didn't know where to do the same for the serial library and not the software serial

so by allowing enough time for the h/w to send the rest of the buffer that should work ? it's pretty straight forward to calculate how much will it takes given the baudrate and the serial communication efficiency (priaty, stop bit etc..) and it should be fine, am i right ?

I had to write the cts/rts code because Windows could not accept 400 bytes at 9600 baud without dropping chars. it was inconceivable that a 16mhz 8bit could flood a 2ghz 32bit at 9600 baud, but I had to drop to 1200 Baud before windows could keep up. Hopefully the netduino gives more priority to comm traffic.

Is the netduino multitasking? can you guarantee the Netduino Serial IO has big enough buffers to handle a single packet without a task switch?

I would implement handshaking if the data was critical.

Chuck

16

i don't have a problem at the netduino, it has much bigger buffer (to the degree that i didn't bother checking how big) but to be safe i use a custom queue that i've posted it's code at this topic where it can handle 1024 bytes (you could go as big as your memory can take) and all you do is pushing new data and trying to pop whenever all fragments are received correctly

for the arduino i've added a delay to wait until the packet is all sent from the hardware so it would guarantee that packets won't be lost

i've moved static data i.e. strings and such, and only send it's index in the packet rather than the actual data so the 64 bytes would be enough as i've unlocked the recv. buffer size on the software serial but not the serial serial :smiley: