Serial readBytes reading less bytes than requested

Using Arduino Storage
1

I am using an Arduino Due with an SD where i store .wav files. A utility is already made that can read data from the serial port. It receives a large CHUNK that reads in 4096 Bytes pieces:

if (strcmp(command, "CHUNK") == 0) {
// t_chunk is index of the chunk. Given that the chunk size (524288 == 512Kb) is larger than the wav file (482278) data will only be sent in CHUNK 0 with all the data.
// to_receive is the size of data being sent, in this case 482278
char fname_buffer[12];
t_chunk = atoi(SerialIO.get_param(0));
to_receive = atoi(SerialIO.get_param(1));
sprintf(fname_buffer, "tmp_%lu.up", t_chunk);
_tmp_file = BingMachine.SD.open(fname_buffer, O_WRITE | O_TRUNC | O_CREAT);
total_read = t_chunk * FIO_CHUNK_SIZE;
this_read = 0;
_error = OK;
while (this_read < to_receive) {
    if (to_receive - this_read < FIO_TX_SIZE) { //FIO_TX_SIZE == 4096
        to_read = to_receive - this_read; 
    } else {
        to_read = FIO_TX_SIZE;
    }    
    read = Serial.readBytes(_buffer, to_read);    
    if (read == 0) {
        // This is a timeout handler                    
        }    
    } else {        
        _tmp_file.write(_buffer, read);
        this_read += read;
        total_read += read;                    
    }
}
_tmp_file.close();

When the last piece of data is read, and it is smaller than 4096, the amount read (stored in read is smaller than the ammount requested to_read.

DEBUG: Received 4096 bytes
DEBUG: Writing
DEBUG: Wrote 479232 out of 482278
DEBUG: Waiting
DEBUG: Received 3010 bytes
DEBUG: Writing
DEBUG: Wrote 482242 out of 482278
DEBUG: Waiting
DEBUG: Received 0 bytes

The code requests 3046 bytes but instead it receives 3010. Serial.available() returns 0 after reading the 3010 bytes suggesting there is no more data to receive. I checked and 482278 is the size of the file.

Here-s the python script where i send the data:

with open(src_fname, 'rb') as f:
        f.seek(0, 2) # move the cursor to the end of the file
        size = f.tell()
        f.seek(0, 0)
        n_chunks = math.ceil(size / (512 * 1024))
        t_chunk = 0
    
        while t_chunk < n_chunks:
            # check for a previous upload
            send_cmd(miniterm, 'CHECK {}'.format(t_chunk))
            # Get the HASH
            output = get_output(miniterm)
            error = get_status(output)
            if error == 0:
                remote_hash = get_hash(output)
                print(
                    "LOCAL: Local Chunk Hash {} - Remote Chunk Hash {}".format(
                    all_hashs[t_chunk], remote_hash))

                # If the hash is correct, send next chunk
                if remote_hash == all_hashs[t_chunk]:
                    t_chunk += 1
                    continue
            # Check failed, just upload this chunk
            to_send = min(CHUNK_SIZE, size - (t_chunk * CHUNK_SIZE))
            send_cmd(miniterm, 'CHUNK {} {}'.format(t_chunk, to_send))
            # Send the chunk
            f.seek(t_chunk * CHUNK_SIZE, 0)
            miniterm.serial.write(f.read(CHUNK_SIZE))
            miniterm.serial.flush()
2

In general, the Arduino ignores buffer overflows. If your sketch can't read data as fast as your sending program can send data, some buffer will overflow and data will be lost.

2 Likes
3

Hello, thank you for the reply.

All the file is sent one time in a big chunk. Then the reader reads in pieces of 4K data (i tried making it 8K but it didn't help). The timeout only occurs in the last piece. It expects a certain amount < 4K but actually gets a couple of bytes less. The program thinks it has to read those extra bytes that it missed but there is nothing more in the buffer.

Do you think if it took less time to read each 4K piece it would not have this problem at the end?

If i send small binary files of 70K i don't have this problem, or it achieves to send it at a second try so this may support your theory. Could it be my serial connection? I am using a Mac M1 and Mini-Usb (for the board) USB cable. I have a USB-C adapter. Maybe it's hardware issue?

4

Yes. Alternatively, the sending side could take more time to write each 4K block.

5

Okay, the problem is that the sending side sends the whole file in one go as you can see in the last portion of the sending code:

miniterm.serial.write(f.read(CHUNK_SIZE))
 miniterm.serial.flush()

If i send a 70K file it reads okay. I will see if i can set CHUNK_SIZE to 64K. When i tried to change it it created a lot of random files. I'll try again.

6

So i put the buffer size that reads in the receiving side to 256Bytes instead of 4096. I sometimes get a timeout, but it works.

Thanks!

7

to_read does not specify how many bytes you want to read; it specifies how many bytes you can read before the buffer overflows.

To harden your communication, you wil need a handshake; using the USB ports on an Arduino, it's a bit tricky with binary data. If you can transfer your data base64-encoded instead of binary, you can implement the Xon/Xoff software handshake.

1 Like
8

Oh, that sounds like a good idea because i still get errors in uploads. Why is it trickier with binary data? I am using miniterm utility from pyserial, and it has an xonxoff option. Do i still need to encode?

9

Xon/Xoff uses two specific byte values (0x11 = Xoff and 0x13 = Xon) that can occur in your binary data; if your sender encounters Xoff, it will stop sending. That is not a problem if you transfer (ASCII) text as those values don't occur in regular text.

The alternative is a hardware handshake but none of the Arduino boards that I'm aware of has the hardware connection between processor and Serial-to-USB converter.

Text file transfer between PC and Arduino using Batch scripts contains a simple demo for the Arduino side of Xon/Xoff.

10

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.