Long arrays, Serial, and the case of missing bytes

Forum 2005-2010 (read only) Software Syntax & Programs
1

Hi all,

Please consider the following code (tested on Arduino Diecimilla with a 328):

// counttest.pde
// CASE1 - byte array, read back long element and print it
// CASE2 - byte array, dump 100 long elements
// CASE3 - long array, read back element and print it
// CASE4 - long array, dump 100 elements
#define CASE4

static const long SERSPEED=115200;

#define NUMELEM 100
static const int NUMLONG= NUMELEM;
static const int SUL = sizeof(unsigned long);
// in bytes
static const int SIZEB = NUMELEM*SUL;

#if ( defined(CASE1) || defined(CASE2) )
static byte tarray[SIZEB]; 
#endif

#if ( defined(CASE3) || defined(CASE4) )
static unsigned long tarray[NUMLONG]; 
#endif

// where are we in tarray? location index..
static unsigned int tli = 0;
// long element variable
static unsigned long telem = 0;
// temporary
static unsigned long tmpl = 0;

void setup() 
{ 
  Serial.begin(SERSPEED); 
  Serial.println("AAAAAA"); // start marker 
  delay(2000);
}

void loop() 
{
  getNextElem();

#if defined(CASE1)  
  tmpl = *(unsigned long *) &tarray[tli];
  Serial.println(tmpl, DEC);
#endif 

#if defined(CASE3)  
  tmpl = tarray[tli];
  Serial.println(tmpl, DEC);
#endif 

#if ( defined(CASE2) )
  if (tli == 0) {
    Serial.println("------");
    Serial.write(tarray, SIZEB);  
  }
#endif 

#if ( defined(CASE4) )
  if (tli == 0) {
    Serial.println("------");
    Serial.write((uint8_t *)tarray, SIZEB);  
  }
#endif 

  //delay(10); 
}

void getNextElem()
{  
  telem++;
  
#if ( defined(CASE1) || defined(CASE2) )
  *(unsigned long *) &tarray[tli] = telem;
  tli += SUL;  
  if (tli == SIZEB) {
    tli = 0; 
  }
#endif 

#if ( defined(CASE3) || defined(CASE4) )
  tarray[tli] = telem;
  tli++;  
  if (tli == NUMLONG) {
    tli = 0; 
  }
#endif 

}

Basically, a long counter gets incremented and its value gets stored in an array; afterwards, either the latest value - or a dump of 100 latest values - gets sent via serial.

Cases 1 and 3 return a single value, and they produce the same output:

$ cat /dev/ttyUSB0  # (line 1)
AAAAAA
0 
0
0
0
... 
0    #  (line 100)
0
1
2
3
4
5
6
7
8
9
10
11
...

Apart from the beginning zeroes (and sometimes an extra line end seems to get inserted), it seems that values are shown in proper order as expected.

Cases 2 and 4 return a binary dump, and they also produce the same output:

cat /dev/ttyUSB0 | hexdump -v -e '"%07.1_ax \t"' -e '4/1 "0x%02x, " "   " 4/1 "0x%02x, " "\t"' -e '4/1 "%_p " "   " 4/1 "%_p "' -e '"\n"'
      0       0x41, 0x41, 0x41, 0x41,   0x41, 0x41, 0x0d, 0x0a,      A A A A   A A . .
      8       0x2d, 0x2d, 0x2d, 0x2d,   0x2d, 0x2d, 0x0d, 0x0a,      - - - -   - - . .
     10       0x01, 0x00, 0x00, 0x00,   0x02, 0x00, 0x00, 0x00,      . . . .   . . . .
     18       0x03, 0x00, 0x00, 0x00,   0x04, 0x00, 0x00, 0x00,      . . . .   . . . .
     20       0x05, 0x00, 0x00, 0x00,   0x06, 0x00, 0x00, 0x00,      . . . .   . . . .
     28       0x07, 0x00, 0x00, 0x00,   0x08, 0x00, 0x00, 0x00,      . . . .   . . . .
     30       0x09, 0x00, 0x00, 0x00,   0x0a, 0x00, 0x00, 0x00,      . . . .   . . . .
     38       0x0b, 0x00, 0x00, 0x00,   0x0c, 0x00, 0x00, 0x00,      . . . .   . . . .
     40       0x0d, 0x00, 0x00, 0x00,   0x0e, 0x00, 0x00, 0x00,      . . . .   . . . .
     48       0x0f, 0x00, 0x00, 0x00,   0x10, 0x00, 0x00, 0x00,      . . . .   . . . .
     50       0x00, 0x00, 0x00, 0x12,   0x00, 0x00, 0x00, 0x00,      . . . .   . . . .
     58       0x00, 0x00, 0x14, 0x00,   0x00, 0x00, 0x15, 0x00,      . . . .   . . . .
     60       0x00, 0x00, 0x16, 0x00,   0x00, 0x00, 0x17, 0x00,      . . . .   . . . .
     68       0x00, 0x00, 0x18, 0x00,   0x00, 0x00, 0x19, 0x00,      . . . .   . . . .
     70       0x00, 0x00, 0x1a, 0x00,   0x00, 0x00, 0x1b, 0x00,      . . . .   . . . .
     78       0x00, 0x00, 0x1c, 0x00,   0x00, 0x00, 0x1d, 0x00,      . . . .   . . . .
     80       0x00, 0x00, 0x1e, 0x00,   0x00, 0x00, 0x1f, 0x00,      . . . .   . . . .
     88       0x00, 0x00, 0x20, 0x00,   0x00, 0x00, 0x21, 0x00,      . .   .   . . ! .
     90       0x00, 0x00, 0x22, 0x00,   0x00, 0x00, 0x23, 0x00,      . . " .   . . # .
     98       0x00, 0x00, 0x24, 0x00,   0x00, 0x00, 0x25, 0x00,      . . $ .   . . % .
...

Now the interesting in these cases is this - note what happens whenever counter is at 0x10 and starts increasing:

38       0x0b, 0x00, 0x00, 0x00,   0x0c, 0x00, 0x00, 0x00,
40       0x0d, 0x00, 0x00, 0x00,   
0x0e, 0x00, 0x00, 0x00,
0x0f, 0x00, 0x00, 0x00,   
0x10, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00,                   // 0x11 missing!
0x12, 0x00, 0x00, 0x00, 
0x00, 0x00, 0x00,                   // 0x13 missing!
0x14, 0x00, 0x00, 0x00, 
0x15, 0x00, 0x00, 0x00, 
0x16, 0x00, 0x00, 0x00, 0x17, 0x00, ... 
...
some 1000 bytes below:
....
448       0x00, 0x00, 0x0c, 0x01,   0x00, 0x00, 
0x0d, 0x01, 0x00, 0x00, 
0x0e, 0x01, 0x00, 0x00, 
0x0f, 0x01,  0x00, 0x00, 
0x10, 0x01, 0x00, 0x00, 
0x01, 0x00, 0x00,                   // 0x11 missing!
0x12, 0x01, 0x00, 0x00, 
0x01, 0x00, 0x00,                   // 0x13 missing!
468       0x14, 0x01, 0x00, 0x00,   0x15, 0x01, 0x00, 0x00,      
470       0x16, 0x01, 0x00, 0x00,   0x17, 0x01, 0x00, 0x00,
...

Well, I'm completely puzzled!?

  • Cases 1 and 3, which read element by element, show that data is apparently entered correctly in the array - otherwise there would be abrupt changes in values if there are missing bytes
  • Cases 2 and 4 have the exact same problem, so it is not due to the how the array is declared?

I'm not aware that 0x11 and 0x13 should have some "special character" meaning - could it be that they maybe get interpreted as commands to the FTDI chip, so it "eats" them during transmission?

Edit: So the questions are:

  • Does anyone know why 0x11 and 0x13 are dropped when sending data via Serial.write?
  • Does anyone know why in cases 1/3, the very first run shows zeroes?

Thanks,
Cheers!

2

Does anyone know why 0x11 and 0x13 are dropped when sending data via Serial.write

aka "XON" and "XOFF".

3

Hi AWOL,

Thanks for your answer!!

aka "XON" and "XOFF".

That was namely it - I usually just use screen to capture output from the Arduino, as in:

screen -L /dev/ttyUSB0 115200

I had no idea that it, by default, actually turns XON/XOFF flow control.. Here is what I did to check:

# run screen 
$ screen -L /dev/ttyUSB0 115200

# get port info via stty
$ stty -F /dev/ttyUSB0 -a

speed 115200 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^H; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>;
start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; flush = ^O;
min = 100; time = 2;
-parenb -parodd cs8 -hupcl -cstopb cread clocal -crtscts
-ignbrk brkint ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl ixon -ixoff -iuclc -ixany -imaxbel
-iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon iexten -echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke

# ixon is ON, turn it off using stty
$ stty -F /dev/ttyUSB0 -ixon

# check again - it is now off: 
$ stty -F /dev/ttyUSB0 -a

speed 115200 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^H; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>;
start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; flush = ^O;
min = 100; time = 2;
-parenb -parodd cs8 -hupcl -cstopb cread clocal -crtscts
-ignbrk brkint ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel
-iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon iexten -echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke

# we could run /cat/devttyUSB0, and 0x11 and 0x13 will be present in the stream 

# now screen again:
$ screen -L /dev/ttyUSB0 115200

# no 0x11, 0x13 seen - check again with stty - ixon has been turned back on: 
$ stty -F /dev/ttyUSB0 -a

speed 115200 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^H; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>;
start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; flush = ^O;
min = 100; time = 2;
-parenb -parodd cs8 -hupcl -cstopb cread clocal -crtscts
-ignbrk brkint ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl ixon -ixoff -iuclc -ixany -imaxbel
-iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon iexten -echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke

The trick was to turn off XON/XOFF flow control in screen, for which the manual says:

-f, -fn, and -fa - turns flow-control on, off, or "automatic switching mode".
...
If a tty (character special device) name (e.g. "/dev/ttya") is specified... An optional parameter is allowed consisting of a comma separated list of flags in the notation used by stty(1):
...
ixon or -ixon Enables (or disables) software flow-control (CTRL-S/CTRL-Q) for sending data.

So, just to note: screen -fn /dev/ttyUSB0 115200 will not work, but this command line will, finally, work:

screen -L /dev/ttyUSB0 115200,-ixon

But as far as it says on AN232B-04_DataLatencyFlow.pdf (via CCS :: View topic - FT232 and handshaking.):

The Need For Handshaking
USB data transfer is prone to delays that do not normally appear in systems that have been used
to transferring data using interrupts.
...
The FT232R, FT2232C (in UART mode)
and FT232BM chips can use RTS/CTS, DTR/DSR hardware or XOn/XOff software handshaking.
It is highly recommended that some form of handshaking be used.

So, I guess a better command line would be one that would specifically enable, say, RTS/CTS while turning off XON/XOFF:

screen -L /dev/ttyUSB0 115200,-ixon,crtscts

but I'm not sure if it makes a difference though - since the rts/cts+dtr/dsr of the FT232 chip actually are not soldered on the Duemillanove, they go on pads (arduino-duemilanove-schematic.pdf). So, hopefully this command at least keeps some handshaking between the PC and the FT232.

The page FTDI Chip Commands made me think whether the same settings could be forced from the Arduino side... But apparently not, as 328 datasheet doesn't even mention flow control in context of its usart.

Finally, Arduino playground - (Re)programming the default settings into the FTDI FT232RL chip notes that MProg (now FT_Prog) is needed to control the settings of the FT232; but there is no mention of XON/XOFF (although there is of rts/cts+dtr/dsr) in AN_124_User_Guide_For_FT_PROG.pdf.

In any case, I'm glad to have this problem clarified.. now, on to the mysterious zeroes :slight_smile:

Cheers!