Corrupt serial data in very simple program

Hey guys! I’m a long-time user of the Arduino, but this is my first post. I’ve searched high and low for a solution, but this one has me stumped.

Problem: When sending serial data to the Arduino after booting my Windows XP machine, the data the Arduino thinks it received is different than what was sent.
Strangeness: If I upload the same sketch to the Arduino, the data received matches the data sent and everything works as intended.

Details: The goal of my project is to control my TV via an IR LED connected to the Arduino. The Arduino is sent a 8 bytes over serial from a Python script running on the same PC the Arduino is connected to.

Here is the Arduino code. I can post the Python script, but I don’t think it’s too relevant.

#include <IRremote.h>

IRsend irsend;

void setup() {
  Serial.begin(9600);
  pinMode(3, OUTPUT);
}

void loop() {
   while (Serial.available() > 0) {
       union u_tag {
        byte b[8];
        unsigned long long ulval; 
       } u;
       u.b[0] = Serial.read();
       u.b[1] = Serial.read();
       u.b[2] = Serial.read();
       u.b[3] = Serial.read();
       u.b[4] = Serial.read();
       u.b[5] = Serial.read();
       u.b[6] = Serial.read();
       u.b[7] = Serial.read();
       
       for(int i = 0; i < 8; i++){
            Serial.write(u.b[i]);
       }
       unsigned long long val = u.ulval;
       irsend.sendPanasonic(val, 48);

      delay(1000);
      Serial.flush();
   }
   delay(2000);
}

Thanks,
Matt

See if there is at least one character in the serial buffer, then read eight of them. Not sensible

Thanks for the quick reply

So your suggestion would be to do while(Serial.available() > 7){...} ?

I've tried that as well and have seen the same results :(

Then start with one character and make that work first. If you can reliably read one, you can reliably read many. This implies that you only read if there is something to read.

You need to post some of your dbug output

Just out of curiosity, why is the definition of the union inside the while block? The definition is static, and should be outside the function. Creating an instance of the union doesn't even belong in the while loop.

There’s no good reason for the union to be inside the while loop. I changed my code so that it’s declared outside now.

A few notes:

  • I’ve moved from my HTPC to my laptop where I’m running XP in VirtualBox (more convenient) and the problem still exists. This more or less eliminates the PC hardware and any driver/software conflicts.
  • My sample programs below are only sending a long value (32 bits) because I’m seeing the same problem I did with the unsigned long long (64 bits) and why not make things a little simpler?
  • I didn’t see the problem when sending/receiving single bytes.

Here is my serialTest sketch

#define NUM_BYTES 4 // Number of bytes we're testing with

union u_tag {
  byte b[NUM_BYTES];
  unsigned long ulval; 
} u;

void setup() {
  Serial.begin(9600);
}

void loop() {
   while (Serial.available() > (NUM_BYTES - 1)) {
       // Read data in
       for(int i = 0; i < NUM_BYTES; i++) {
         u.b[i] = Serial.read();
       }

       delay(1000);
       
       // Write received data out
       for(int p = 0; p < NUM_BYTES; p++){
        Serial.write(u.b[p]);
       }
   }
}

Here is my serialTest.py script

import d2xx
import struct
import binascii
import time
import sys

def serialCom(h, hexData):
	structFmt = "<L"
	dataSize = (len(hexData) - 2) / 2
	print dataSize
	print "HexData = " + str(hexData)
	try:
		longData = long(hexData, 16)
		# Pack value as binary data string to make d2xx happy
		binStr = struct.pack(structFmt, longData)
		# Write data to Arduino
		h.write(binStr)
		
		# Debug output
		outputFile = "c:\serialTestOutput.txt"
		f = open(outputFile, "w")
		sentStr = "Sent: " + str(longData)
		print sentStr
		f.write(sentStr + "\n")
		f.close()
		
		# Give things a sec to settle
		time.sleep(1)
		
		# Read in serial data from Arduino
		readBinStr = h.read(dataSize)
		
		# Debug output
		f = open(outputFile, "a")
		rcvdStr = "Received: " + str(struct.unpack(structFmt,readBinStr))
		print rcvdStr
		f.write(rcvdStr + "\n")
		f.close()
	except Exception, err:
		print "exception occurred: " + str(err)
		#pass

def setup():
	h = d2xx.open(0)
	h.setBaudRate(9600)
	return h

def main():
	h = setup()
	# eventually need to send something like this: 0x40040100BCBD
	serialCom(h, '0x40040100')
	
	time.sleep(3)

if __name__ == "__main__":
	main()

Here are the results

Uploaded sketch to Arduino and ran Python script
==BEGIN_OUTPUT==
Sent: 1074004224
Received: (1074004224,)
==END_OUTPUT==

Restarted XP and ran Python script
==BEGIN_OUTPUT==
Sent: 1074004224
// the script hangs here waiting for 4 bytes on the serial buffer
==END_OUTPUT==

Killed the Python script and ran it again
==BEGIN_OUTPUT==
Sent: 1074004224
Received: (35725312,)
==END_OUTPUT==

Reran the Python script
==BEGIN_OUTPUT==
Sent: 1074004224
Received: (100802848,)
==END_OUTPUT==

Reran the Python script
==BEGIN_OUTPUT==
Sent: 1074004224
Received: (1107493128,)
==END_OUTPUT==

-------------------------------------
Edited the Python script to run do the send and receive of the same data 3 times

==BEGIN_OUTPUT==
Sent: 1074004224
// the script hangs here waiting for 4 bytes on the serial buffer
==END_OUTPUT==

==BEGIN_OUTPUT==
Sent: 1074004224
Received: (34676736,)


Sent: 1074004224
Received: (35725438,)


Sent: 1074004224
Received: (17330694,)
==END_OUTPUT==

The binary representation of the sent number:
1000000000001000000000100000000

The binary representation of the 3 received numbers respectively:
10000100010010000000000000
10001000010010000001111110
01000010000111001000000110

First thing I see is that you have a fundamental misunderstanding about how serial data is sent/received.

Serial delivery follows the USPS model of delivery, not the UPS model. UPS guarantees to deliver your package. USPS guarantees to try to deliver your package. Big difference.

You need to account for possible lost bytes. Nothing in your code allows for this.

Second, I see no good reason for printing the values sent and received without delimiters. Doing so makes it very difficult to see where a byte got missed/lost/misunderstood. Spaces make fine delimiters. See how many of them I've used in this message, to make it more understandable?

I'm not an expert on Python by any means. I can't see how

        rcvdStr = "Received: " + str(struct.unpack(structFmt,readBinStr))

results in

Received: (1074004224,)

Where are the parentheses and comma coming from?

Perhaps the thing you need to do is look into whether the issue is on the Python end or the Arduino end. On the Python end, print the individual bytes being sent, with spaces between them. Print the received data with space between them. Do this in addition to the packed sent and packed received printing that you are doing now. Perhaps a clue will position itself so that you (or we) can trip over it.

The parentheses and comma are due to struct.unpack returning a Python tuple.

I have the problem mostly narrowed down to the Arduino. I'm using a program (called USBlyzer) to sniff the serial data and my Python script is sending the data correctly.

I will try to modify my code to do the things you've suggested to help with debugging. I'll also look into accounting for lost bytes.

Thanks for your help and sharp sarcasm. I lol'd