Reading Data to Buffer - Missing Bytes

Hi All,

I’m trying to send data from my computer to my Arduino and vice versa over the MIDI protocol. Unfortunately MIDI SysEx messages are the only thing I can use for my particular application, although I don’t suspect it to be the problem in this situation. Anyways, I have a part of my code that looks something like this:

  buffer[0] = midiSerial.read();
  if (buffer[0] == 0xF0) {
        midiSerial.readBytesUntil(0xF7,(char*)buffer,buf);
        delay(200);
   }

In a completely random fashion, once the buffer gets to about the 50th-55th byte (always begins in this range), it begins to start missing bytes here and there for the next 20 or so bytes, and then the rest of the buffer is fine from about the 70th byte range. I’ve attached some screen shots of my serial monitor, in which I’ve simply sent the data from my computer as byte(x) = x, with the left side being the byte # and the right side being the received value (of which some go missing). While the actual location of the missing bytes is completely random, it’s always in the same ranges (e.g. starts at 50th - 55th, ends around the 70th).

I’m certainly pushing the 2k RAM limit, but other than this issue, the base of my program is completely fine. From that point I even tried shortening my buffers to give the RAM a couple hundred more bytes of space, but it doesn’t seem to make a difference. It should also be noted that I know readBytesUntil has it’s own issues, but I’ve tried the same program with an if function checking for the termination value and was having the same problem.

I’m going to keep fiddling with it and hope to come across something soon. If anyone has any guidance, it’d be much appreciated! Thanks in advance,

Pat

midiSerial.readBytesUntil(0xF7,(char*)buffer,buf);

At this point in the program how do you know that all the bytes expected have actually arrived and are available to be read ?

Good question, in my if (buffer == 0xF7) version I had a serial.available() > 0 statement, but for some reason I got the impression that readBytesUntil had its own method of checking if a byte was available. Regardless, is this likely a source to the problem since my issue is a lack of certain bytes and not duplicates of the same byte over multiple addresses? I feel like the latter would be an effect of not checking for availability, but if it's over checking for bytes it shouldn't be missing any, I'd think.

Regardless, I'll throw in a serial.available statement and see if that does the trick. Thanks for the help,

Pat

Regardless, I’ll throw in a serial.available statement and see if that does the trick

Where are you going to put it ?
The usual technique is to read each byte into a buffer as it becomes available and to use the buffer contents when the correct number of bytes, or better still an end of message marker is received.

At this point in the program how do you know that all the bytes expected have actually arrived and are available to be read ?

It doesn't matter. The readBytesUntil() method blocks waiting for data. It returns when the until byte arrives or the timeout happens. Assuming that the until byte has arrived is probably the source of the problem.

Of course, we can only see a tiny snippet of the code, so it's all guesswork. But, I guess that's OK.

The readBytesUntil() method blocks waiting for data. It returns when the until byte arrives or the timeout happens

Thanks for the clarification.

Thanks a lot for the help guys… The entire code starts from this little piece, it’s truly the only part of the code that handles the buffer reading. It runs a continuous loop waiting for that first byte to = 0xF0 and then executes some saving / reading from flash upon arrival ( but I’ve checked that the source isn’t the flash reading/writing).

I know for a fact that its not reading the termination byte early, its reading the entire 128 byte buffer being sent over. I actually just realized (a true “doh” moment) that I can set the Serial.Available() to be equal to my desired buffer length before executing the rest of the code. I’ve only used it to find if the availability is greater than zero. So I tried this:

for (;midiSerial.available() < 120;) { midiSerial.available(); }
        midiSerial.readBytesUntil(0xF7,(char*)buffer,buf);
        delay(200); //}

I figured checking the availability is putting the least amount of “strain” on the Arduino’s processing, so it surely shouldn’t miss a byte… but I still get the same issue in the same spot.

I have two hunches that I believe I can narrow it down to, from what I’ve tested and from what I know:

  1. It’s something Arduino related: for some reason my uno is incapable of reading the data consistently in the specified range because the clock slows down or somehow becomes out of sync with the frequency at which the midi is being sent (over a baud rate of 31250 bit/s).
  2. My second piece of hardware, the MIDI communicator between the computer and arduino, is unable to handle that much midi data. I truly do not think this is the case at all, as it’s a $300 board that ottb communicates in midi. However, in the nature of narrowing it down, I’ll try looking at it by checking the output of the MIDI (which is going to be a huge pain in the ass since I only have an oscilloscope and am starting at that 50th byte).

I’m still hoping to find someone who’s had a similar experience on #1, but thanks again for the replies and the help.

Pat

Ah yes, I've finally figured it out. Dumb mistake of the week:

On my desktop program (made with Processing, actually) I have three different types of arrays that are sent out, all of which are different sizes. I made my buffer size on the Arduino the max amount of bytes that should be sent to it (170 bytes).

Well it seems that array #1 was set to 256 bytes, even though I'm only using 170 bytes of useful information. Looks like it was sending an extra 86 bytes of zeros after the readBytesUntil had been terminated, which, while not entering the if statement, were getting stuck in the midi buffer or something and messing up the following array #2. Corrected that and everything works like a charm!

Kids, make sure you're sending the correct number of bytes the Arduino's expecting to make it nice and happy.

Thanks everyone for the help,

Pat