I'm getting serial Rx overruns but I thought I was doing it right!

Projects Programming
1

I have been working on a MEGA 2560 project for quite some time, and it has been going well so far. This time, adding some more software I found a problem that I thought I was fine with. Reading in serial data (MIDI, at 31-something Kbps), I was receiving Sys-X dumps from synthesizers and storing it to EEPROMs. (For older Korg DW-6000 and DW-8000) But I am now adding a new model, the Kawai K3m and it's a bit different. It doesn't have much more data, but it splits all the data into 4-bit nibbles so it sends a lot more bytes in one burst. The Korgs send send over 3000 bytes for a dump and that has worked reliably for hundreds of dumps.

The Korgs send data in 32 and 57 byte bursts, but the Kawai is sending a 78 byte burst, and after 63 bytes it quits consistently. Clearly the 64-byte Rx buffer is being overrun. I took out all the things I thought could be dragging the speed down (combining nibbles and storing them), and just did a quick get-and-save into an array now. But it still conks out at 63 bytes. So apparently I don't quite understand how to keep the buffer from overrunning.

Here's the stripped-down code from the Kawai that isn't working. I don't know how to get it much more efficient. I also present for comparison the two older routines that proved solid for thousands of dumps, where I do a lot of processing in the while loop before getting the next byte, I don't know why this routine isn't clearing out the Rx buffer in time. Of course the Kawai is a 35+ year old machine, so there's no option to get in and modify the code or the bit rate. It is what it is.

Please, please do not ask me to provide the whole program. It's 21908 lines of code. I'm not even sure I can post that much. I don't believe it is very relevant to this overrun.

The Kawai code snippet that is not working:

byte K3input[78];    //stores MIDI input before processing data.
byte MIDIinput = 0;
int count = 1;  //counts for 7+70+1=78 bytes received, then exits.  7 is header, 70 is data chunk (35x2), and 1 is EOX.

    while (count < 79) {
        if (Serial1.available()) {
            MIDIinput = Serial1.read();
            if (MIDIinput != 0xFE) {
                K3input[count] = MIDIinput;
                count++;
            }
        }
    }

And the same section in the DW6000 that works:

    timeout = millis() + 4000;    //4 second timeout
    while ( (count < 57) && (timeout > millis() ) ) {
        if (Serial1.available()) {
            MIDIinput = Serial1.read();
            if (EchoOut==true) { Serial3.write(MIDIinput); }
            count++;
        }
//MANY OTHER STATEMENTS HERE
    }

And the DW8000 code that works:

    timeout = millis() + 4000;    //4 second timeout
    while ( (count < 32) && (timeout>millis()) ) {
        if (Serial1.available()) {
            MIDIinput = Serial1.read();
            if (EchoOut==true) { Serial3.write(MIDIinput); }
            count++;
        }
//MANY OTHER STATEMENTS HERE
    }

These other two working examples that show "MANY OTHER STATEMENTS HERE" have pages of data moving and shuffling, masking and combining before re-looping in the while loop to read another byte of data, so I can't figure out what is so slow about the first, non-working example. Exactly how the bytes are processed in the examples is different, but the fundamental Rx of the data into the program was cut & pasted.

I read as much as I practically can in the forums, and found topics that touch on the matter, but not really addressing the overruns directly, they seem to show this tight loop will work. I also have a hard time believing a nearly 40 year old appliance would have a processor that can overload a (properly programmed) AT processor today. So yea, the problem has to be me and my code. Hopefully someone can spot where my obvious brain-fart has made it's mark.

2

first of all, with this lines you overrun your own array of K3input[78]
Definition of array with 78 elements means that array indexes has a range 0-77 and storing something in K3input[78] will give you out of bounds

3

Yes, I see that now. But that isn't the source of the problem. That was just me typing something in quick to try and test the root cause - in effect the 63-byte overrun caused the array typo!

As I originally had in the other examples, I didn't read the Rx bytes into an array. I only created that array to see if it would work faster. The array never gets to the point of overrun, because the serial read never delivers more than 63 bytes. But good eye!

4

the forum takes code as code-section up to 120.000 bytes long
what would be indeed interesting are the baudrates that you use

one option would be to increase the buffer to more than 78 bytes.
You are using a Mega 2560 which has 8 kB of SRAM

Another way might be to use a completely different concept for the receive-function

A really fast running = fast iterating function loop() in combination with a receive function like described in the Serial Input Basics - updated

for keeping loop() iterating really fast really all functions called by loop() have to work in a quickyl jump_in / jump_out manner

all repeating must be done by loop()
This ensures that as soon as a byte arrives in the receive-buffer that it will be taken out of the receive-buffer asap

All and every iterating through an array must be done by
jump_in
increment array-index by 1
process only that single array-index
and jump_out

and this will be repeated because loop() is iterating (looping) really fast

// function A() {
increment array-index by 1 
process  only that single array-index 
}

// function B() {
increment array-index by 1 
process  only that single array-index 
}

void loop() {
  A(); // quickly jump_in / jump_out
  B(); // quickly jump_in / jump_out
}

best regards Stefan

1 Like
5

The simplest solution for this would be to modify

#define SERIAL_RX_BUFFER_SIZE 64

from 'HardwareSerial.h' to something a bit bigger.
basically you can just add the line to the top of your sketch

#define SERIAL_RX_BUFFER_SIZE 80

since in HardwareSerial.h it says

#if !defined(SERIAL_RX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_RX_BUFFER_SIZE 16
#else
#define SERIAL_RX_BUFFER_SIZE 64
#endif
#endif
#if (SERIAL_TX_BUFFER_SIZE>256)

Then clearly the storing doesn't start when the first byte is received (or soon enough after that) at 31500bps transfer speed is not the issue at all, you must be more than half a packet behind when you start reading.

The whole program probably not, but as said, you are doing something wrong which prevents the start of the snippet that you've posted. More than likely increasing the buffer size will make your program work as expected, and you can do the nibble chunking within the reception routine again, but it is just a patch and at that baud-rate you should not really have that issue at all (the mega can handle speeds 10x that with some extra lines of code in the reception routine)

6

why do you test for "0xFE"? is this a termination character that is being waited on to process a complete string?

in many cases the input needs to be processed both when that termination character is received or when the buffer is full

see Serial.readBytesUntil()

why do you conditionally process input only for a limited amount of time? are you trying to process the input if no further input is received after a pause? readBytesUntil() does have a timeout

7

I was assuming this was the EOX character for the midi sysex, but actually that is '0xF7'

8

The 0xFE is the MIDI Active Sensing byte. The spec defines it as a realtime message that can be inserted anywhere in the MIDI stream, including in the middle of Sys-X messages. The Kawai K3m sends Active Sensing as soon as powered up, and can not be turned off. So because we get them randomly every 100mSec at any time during the message, they need to be filtered out or they screw up everything else. It is not a termination character. While observing the received data, I'm getting a whole lot of these and they are useless in this application. I think they are a leftover software feature from the keyboard version that can send note on events.

The end of Sys-x character 0xF7 arrives well after the overrun, and gets ignored like everything else.

The time-limit is to prevent the software from locking-up waiting for bytes that may not have been there. Often in a studio setting working on music, MIDI channels and cables are re-routed and switched, devices are disabled, not powered on, or in some other way prevented from getting or responding to a message. If the Arduino is caught in an endless loop waiting for a message, we effectively loose every bit of data and the current state of the machine while we power-cycle.

Anyways, this read the MIDI message is called soon after the request is sent to the synthesizer. It responds within a fraction of a second, and is done in the blink of an eye. After that, there really is no reason to expect any more data. I can see that in the MIDI activity LEDs that are part of my machine. These are really not random out-of-sync messages that appear at any time, they are responses to specific requests for data.

I think there is promise in trying this:

#define SERIAL_RX_BUFFER_SIZE 160

but it almost seems too simple, after reading about the other solutions to small buffer spaces and having to go in there and mess with the build files.

9

doesn't seem like this should be ignored. instead, it seems to indicate the start of a new message that needs to be captured separate from any message currently being read. once its termination character is received, either it should be processed or can be ignored and then the code can continue to capture any message previously being transmitted

not sure this code is doing what you expect. it says only read data if the buffer isn't too full or if less than some timeout. but it doesn't say what to do in the other case

part of the question is where is the code that processes the message that is presumably called when the termination character is received or when the receive buffer, K3input[] is full, that processing routine should be called

for example

    if (Serial1.available()) {
        msecLastChar == millis ();
        char c = Serial1.read();
        if (char c != 0xFE)
            readRtMessage ();

        else  {
            K3input [count=++] = c;
            if (TermChar == c || sizeof(K3input) <= count)
                process (K3input);
        }
    }
    else if (0 < count && millis () - msecLastChar > Timeout)
        process (K3input);
10

Well the RX buffer is 64 by default, and i was lookin8g for a method to set them individually for each UART, but that i did not find. Still i have increased the RX-buffer myself in certain situations and i know that it fixes what you experience. Still the issue is that the function that handles received midi messages, isn't called often enough. That is the actual cause of a rx-buffer overrun. You should be able to parse and store the info quickly enough for the buffer not to overflow. I suspect that you just don't check often enough to see if there is anything in there, or you start to late or something. Basically i think i want to see the call to the function that does the receiving.

11

The active sensing bit simply has to be ignored. It is a single byte message, with no termination. It is used to tell the receiving device to turn off all notes played if has not been received within about a second. As the Arduino doesn't make any sound, there's nothing to be turned off, and no termination to wait for before processing. It is a real time message that is unnecessary in this and many other applications. It was pretty much a legacy of old MIDI version 1 that is seldom used but was there when this old synth was designed.

The while loop does appear to work. The loop waits for all expected bytes to be received, and gives up after a few seconds. If nothing is received after the few seconds of time, the Arduino returns to it's normal loop waiting for user input and displaying the success or failure of the response from the synth. Anyway, that is the code for the DW6000 and DW8000 which works. The timeout has been removed only for the Kawai loop that doesn't work in a failed effort to reduce the time to empty the serial buffer. So the timeout isn't the source of the buffer overruns.

12

Deva_Rishi I think you are right. I can't wait to try this - I'm at work now and can't try it until tonight. You also just made me realize that by increasing the buffer space, I will be increasing it for all serial ports, so the size penalty is X4. Yikes - as I do not have a lot of memory to spare.

I believe what you say, the real solution is to look more closely at the code that calls this. It's a main loop with a lot of things going on, reading pots, checking user inputs, displaying data to a LCD screen, etc. So if I get what you are saying, the serial data is accumulated while the main loop is running, and my call to get the data is delayed such that by the time I start emptying the buffer it may be too late. From my understanding that makes a lot of sense.

So I'll try increasing the buffer space as you suggest, but if that works, the correct fix may be to not live with larger buffers but to call this Rx routine faster, or delay the data request sent out until I am ready to read it. If I remember correctly, the main loop lasts a few hundred milliseconds or more - enough time to have an overrun of data. I just never experienced it before because my buffer space of 64 bytes was always enough to hold an entire message from the other synths.

13

what are the bytes being sent that overrun the buffer in your one case?

and what processing is taking so long to prevent re-entering your while loop to read them?

14

Only for any ports used, and hence i went for 80, that is only 16 bytes.

well that is maybe where SerialEvent may come in handy.

For sure it is. at 31500 about 3 bytes per ms, buffer is full in 25.

If you are really that short on memory, you know where there is some up for grabs.

15

I think he is reading through the main loop. Normal i would wait with everything else after sending the sysex request, or at least wait for it specifically, but SerialEvent will resolve the issue rather elegantly

16

I think I can answer both of these questions. The bytes being received is a header (7 bytes) that identifies the synth manufacturer, the model, the reason of the data, etc. followed by 70 bytes of data that is the sound parameters that were requested. Having captured the data and displaying it, I was able to verify that the data exactly matched the sound parameters that were requested by calling that same data from the synths control panel - up until the overrun. Of course the first test showed a lot of 0xFE bytes - those were the active sensing bytes that I had to ignore. So this tells me the synth correctly received the request and was trying to respond.

I took a quick look at a copy of the code I have, and sure enough, Deva_Rishi seems to understand what is going on. My main loop has seven main functions.

  1. Update the LCD display based on the flags that were set to indicate what lines needed to be updated.
  2. Respond to any MIDI data dumps that were flagged as expected.
  3. Respond to any user button presses that were received (from serial).
  4. Respond to any encoder turns that were flagged from an interrupt routine that captured them.
  5. Respond to any pot turns (ADC channels that have voltage changes over a threshold).
  6. Handle any secondary MIDI port messages that need to be passed through.
  7. A short main loop delay (that slowed down the LCD refresh time. (at this point it is no longer needed!)

...what is of particular note here is the order. The MIDI data dump is requested by a user button press, and that is step 3. Meanwhile, step 2 is the place where a MIDI data dump is looked for and read from the buffer. It has to GO COMPLETELY AROUND THE MAIN LOOP - 4-5-6-1-2 before it can respond to the data! It was never a problem when I was receiving bursts of less than 64 bytes. Even a whole memory dump of 64 bursts was slow enough to capture and store. But with this Kawai synth sending >64 bytes, it make sense that the whole burst arrives before my main loop gets around to reading it.

Hopefully, the fix could be as simple as changing the order of the main loop steps. I can't wait to try it out tonight and see if that is the best fix. If so, I won't even bother to increase the Arduino's Rx buffer sizes.

17

I guessed it, it's a common error, but had you provided this info earlier, anyone could have pointed that out.
I mean seriously it is pretty much always that at those speeds. LCD displays can be slow in a blocking way,
The best method for Serial reception is actually writing the handling ISR yourself, but that means that you also have to set the other registers correctly without the use of the hardware Serial library. Mind you in this case it is UART specific. You can still use the other ports with their default settings and handlers. which are easy to use and comprehend. Defining the RX_ISR yourself means that store whatever you receive in the correct place straightaway, no extra rx-buffer at all. For a really good midi controller this is actually the way to go.

18

my question was why, in this case, are more bytes being sent and what are they and how many more are there?

why is the control panel sending more data?

how long do all you loop() processing sub-function take and based on that and the serial interface speed, what is the maximum # of byte that can be transmitted?

19

My brain was too far into the loop, seeing no good reason why it couldn't respond quickly enough, when I've been using this thing for at least a year with it working. I needed someone to tell me it was getting too full before I went to read it. Since the Arduino's speed usually is very fast, I just assumed it was ready for the slow MIDI response!

Here's the funny thing: I intentionally made the main loop that way. Knowing these old synths are sometimes very slow to respond, and MIDI itself is pretty slow, I thought rather than waste time waiting for the data, to go off and update the screen and do other stuff before getting around to the UART. (I'm generally intolerant of slow user interfaces, buttons or displays that take to long to respond.) Yes, the new stuff is faster, but not that fast. It takes time to do everything else, including handling TWO LCD controllers (I have a 40x4 LCD).

Also of note: I get quite frustrated trying to figure out the "secret" coding of Arduino. So much so with this project that I decided that from now on I will only do projects with Microchip in assembly. That's my real strong suit. When I take on a new Microchip PIC project, I get it done. When I take on an Arduino project, more often I put it aside after a while unfinished. Not saying it's wrong, but it's not for me.

This very project uses one PIC controller to read all the buttons, light all the LEDs, and capture encoder rotations, and communicate that to the Arduino with serial. I wrote that assembly code almost 10 years ago, and used it on many projects with not a single bug needing correction. I've used it on a big machine that uses 16 of the PICs, has over 200 buttons, 20 encoders, and 300 LEDs. It's only on Arduino software I get in trouble. In the PIC, I get to write all my ISRs, and everything else, so I know what is going on. With the Arduino I am constantly running into problems that could have been known with deep, detailed, thorough documentation - something I find hard to get online with the Arduino.

But I can't assume the loop changes will work - I need to try them first. I've been burned too many times to think the problem is solved before it was proven functional. Hopefully the loop issue explains all that is going on.

20

There source code is all there for you, and the rest is in the datasheet of the MCU.
I look stuff up in the core files at time, and i have my own example for the UART setup without the Serial class

Well writing your own rx-ISR should be easy enough in that case.

it does. Only other explanation can be a hardware issue.
Again time wise it is more efficient the have the loop in the order. Midi protocol is a compromise. They went for 31250 because it was slow enough to cover at least 5 meter in a none twisted pair cable, and works with an opto-coupler that was available at the time. slow enough so the synthesizers could keep up. So waiting for a long message is wasteful. The Serial object rx-ISR stores all incoming bytes in the buffer until they are read. available checks the amount of byte between the head and the tail etc.

Locate 'HardwareSerial.h' on your computer and open it in notepad++ check out the files in the vicinity etc.
You are always free to write assembler as well of course, but personally i am happy with this C++ variant these days, The freedom to modify anything to suit my needs does the trick.