I find myself in a need to modify the size of the SERIAL_RX_BUFFER_SIZE. Now, being difficult, I would like to do that without modification of the Arduino Platform files (i.e. "HardwareSerial.h")
It looks like the header file does support this in theory, with line like this:
Which in a normal C build environment would mean I just need to add -DSERIAL_RX_BUFFER_SIZE=128 to my build command and I am fine, but this is no normal C environment and for the life of me I cannot figure out how to add -D parameters, or even if that is possible
Any suggestions for the right way to accomplish this under Arduino? (I am using command line ArduinoIDE to build this)
But the question is why you need to change the size of the buffer. If your code reads that buffer often enough, there is no issue.
Yes, if you do not have much data and code does not do much else, it is not a problem. That is not my situation.
Since in my case it is a request/response - and in most cases my requests are well under 64 bytes, it is working fine, but when requests are larger, I start losing data. A little extra buffer space will go a long way, esp. given I have a max size for a possible packet.
Have a look at the examples in Serial Input Basics - simple reliable ways to receive data.
The maximum size that it can receive is determined by numChars. You can change it to anything you like. Note that if you need more than 255 chars you will need to change it from a byte to an int.
Robin2's example shows one way to receive data by constantly checking for new bytes. I suspect you have some other section of code that is "blocking" for some reason, for too long.
It is usually very easy to change that other section to check millis() instead of using delay. See also the sticky post at the top of this subforum: Using millis() for timing. A beginners guide. There are several good links in the General Design section of the Useful Links page.
If you are not using delay, what is taking so long? Printing too much can also cause this problem.
If you post your code, or the smallest version that shows the problem, we can offer more specific suggestions.
These examples are cute, but I am working on a bit different scale and pushing Arduino way beyond basic examples shown here. No, there are no delay's, and yes, there are tasks in scheduler that take a long time and occasionally blocking on other comms or require non-interrupted attention (for example driving steppers require exact, uninterrupted timing and the more often you stop/start the more steps you lose from your calibration) .
Since Arduino is not well suited to true multitasking, my core scheduler is cooperative in nature - so it is very hard to guarantee I can pick up characters arriving at sub-millisecond rate in time - especially when they arrive all at once as packet data tends to. Sure, I can set up an interrupt and a custom buffer and handler and handle it this way, but it would incur significant overhead - need to handle locking between interrupt handler and main thread - not to mention it would waste a timer. I could do that, or I could use existing timer and buffer that are already handling the port and just adjust the size of the buffer to my needs - which seems to require basic things that gcc (compiler Arduino uses under the hood) easily supports, but Arduino IDE wrapper on top if GCC seems to randomly have no support for - at least none that I could find
Yes, I can adjust the library's headers directly, but that becomes a pain to maintain as those libraries change and the project is compiled on multiple systems - so I am trying to avoid that.
So, if anyone knows a proper way to do pass in the macro definitions, it would be useful, but I am starting to suspect I may need to give up on using Ardiuno IDE for building altogether and go for something a but more capable. I tried a few makefiles in the past, but it seemed to be hard to get it in a compatible enough way without a lot of work
Since you are familiar with interrupts, I feel comfortable recommending NeoHWSerial. It is the original HardwareSerial, with one addition: attachInterrupt.
This uses the existing RX interrupt, so you would not need a timer to read chars "periodically". This also saves storing, calling available, and retrieving.
The simplest ISR you could provide would buffer some chars, effectively increasing the default rx buffer size.
A more capable ISR could do some parsing, avoiding the need to buffer anything.
The uart can only hold on to 2 chars before the rx ISR must be called (with any library). I assume you do not disable interrupts any longer than 2 char times (20/baud ms).
Since you are familiar with interrupts, I feel comfortable recommending NeoHWSerial. It is the original HardwareSerial, with one addition: attachInterrupt.
This uses the existing RX interrupt, so you would not need a timer to read chars "periodically". This also saves storing, calling available, and retrieving.
The simplest ISR you could provide would buffer some chars, effectively increasing the default rx buffer size.
A more capable ISR could do some parsing, avoiding the need to buffer anything.
The uart can only hold on to 2 chars before the rx ISR must be called (with any library). I assume you do not disable interrupts any longer than 2 char times (20/baud ms).
That looks promising. I can grumble and deal with making my ISR serviced buffer threadsafe - i kinda like that as I do have parsing buffer/code already, but even without using that lib, the idea of using a custom version of the HardwareSerial as a lib is brilliantly simple and doable and solves my immediate problem. I was thinking I would have to do the whole Arduino core lib, but that seems to not be the case...
I mostly did not want to have an interrupt collecting characters to a buffer and then ANOTHER interrupt collecting characters from that buffer to another buffer and then more code to move it from that second buffer to parsing - that seems crazy wasteful, and if my time with Arduino taught me anything, it is that every byte/pin/timer is precious
Thank you.
-HH
P.S. Thinking more about NeoHWSerial is rocking my world. This has potential to solve a number of problems for me... Thank you again
so it is very hard to guarantee I can pick up characters arriving at sub-millisecond rate in time -
Isn't that the reason for the 64 byte Serial input buffer? - you don't need to deal with each single byte, you just need to empty the buffer before it overflows.
Robin2:
Isn't that the reason for the 64 byte Serial input buffer? - you don't need to deal with each single byte, you just need to empty the buffer before it overflows.
What baud rate are you using?
...R
Running at 115200 - which fills that 64 byte buffer in what, about 5-6ms if my mental math is right (64/11520).. I am actually fine with the speed - I have a solid request/response protocol that will wait after a request until it is getting a response before sending next request - I am not worried about overrun from multiple requests. Its just that my max packet size is 146 or so bytes, and any time I go over 64, I risk losing part of it. Simply increasing the buffer to the right size fixes my problem, but the NewHWSerial may offload the buffer handling to me instead of keeping it in the stock hardware serial drive, which will work great (but will take some work to implement)
Robin2:
Can you not guarantee to empty the buffer every 4msecs, say? An Arduino can get through a lot of other stuff in 4msecs.
Nope. There are definitely some tasks that can take up to several seconds that should not be interrupted in the main thread Example is moving stepper motors, which require a precise timing and stopping and starting them causes pin drift. We still stop and start them, but try to minimize that to once every few seconds. So, not without writing a secondary interrupt handler, which as I pointed out is redundant especially since there is already an interrupt that does this, just has a buffer to that is too small. Plus writing a buffer for a buffer for a buffer for a buffer is not a particularly good programming technique.
I think I have several unexpected solution options, though it would still be really nice to be able to do basic -D's at build time.... (for our own stuff I ended up auto-generating a header file at build time)
hichhiker:
Nope. There are definitely some tasks that can take up to several seconds that should not be interrupted in the main thread Example is moving stepper motors,
Then it sounds like your only option is to modify the HardwareSerial code. You could make a complete copy of the Arduino IDE and modify the copy that you use for this project. That way you could use the standard version for your other projects.
I have a somewhat similar project insofar as I want my stepper movement to complete before I check the serial buffer. I just made sure that I never send more than 64 bytes. In fact my project needs a lot less than that.
Robin2:
Then it sounds like your only option is to modify the HardwareSerial code. You could make a complete copy of the Arduino IDE and modify the copy that you use for this project. That way you could use the standard version for your other projects.
Based on the above my options are to use NewHWSerial OR do a similar approach, but just make my own clone of just the HardwareSerial code and not the full Arduino libs. Both are significant efforts though and may require modification of 3rd party libs All because we cannot pass proper parameters to the compiler
Robin2:
Don't lose sight of the fact that the Arduino IDE has been designed to make things easy for newcomers to micro-processors.
I'm not sure how NeoHWSerial is any different from a "secondary interrupt handler".
It replaces HardwareSerial and uses the same interrupt HardwareSerial was using instead of using a second interrupt. It is essentially a modified version of HardwareSerial.
Both are significant efforts though and may require modification of 3rd party libs
You could put NeoHWSerial into your sketch directory (i.e., your source tree), instead of including it from the Libraries directory. Then it's under your change control. You wouldn't have to pay me very much.
You could even remove the RX buffer entirely, making it an ISR-only receiver.
Having it in your sketch directory would allow it to have config items in the header, in lieu of -D compile options. Or includee separate config files.
That doesn't seem like a lot of effort to me. What is your concern?
I also wanted to mention that Ne**o**HWSerial is schizo. If you attach and detach the RX ISR function at different times, it switches between dispatching immediately and simply buffering. That could help keep the ISR times short during critical operations.
-dev:
You could put NeoHWSerial into your sketch directory (i.e., your source tree), instead of including it from the Libraries directory. Then it's under your change control. You wouldn't have to pay me very much.
Erm, that reminds me to check out the licensing requirements for this... :o That said, I was not intending to modify that (assuming it works)
-dev:
You could even remove the RX buffer entirely, making it an ISR-only receiver.
Having it in your sketch directory would allow it to have config items in the header, in lieu of -D compile options. Or includee separate config files.
That doesn't seem like a lot of effort to me. What is your concern?
I also wanted to mention that Ne**o**HWSerial is schizo. If you attach and detach the RX ISR function at different times, it switches between dispatching immediately and simply buffering. That could help keep the ISR times short during critical operations.
Honestly just having slightly larger buffer is enough for my needs. But he efforts I am talking about is do to me using all 4 serial ports, each for a different purpose, using different protocols, and in one case, using a 3rd party library (XBee). As I understand that whether I use NeoHWSerial or my own hacked up version of real HardwareSerial - I will need to rename them, and even then I have to keep the real deal off the compilation - which means I will have to convert all 4 ports at once, including use by XBee lib, and then make sure all uses are functional. Then I would need to rewrite the code to use ISR driven buffers. Its all pretty doable, but will take some time. Though the "schizo" aspect seem like it may help as I can probably just use buffering for everything else except for once place I need ISR...
Lots to think about... But I am also exploring if I can simply modify the HardwareSerial.h and drop it somewhere before the SDK in gcc's include path, so that it is included in the build.. super hacky but it literally the easiest way to go.... Not sure if it would work or how portable it would be...
that reminds me to check out the licensing requirements for this... :o That said, I was not intending to modify that (assuming it works)
As I said, NeoHWSerial is the Arduino core HardwareSerial, with one modification I made. It's open source, NeoHWSerial is open source. You can modify it all you want. License files are there, if your lawyer wants to read them.
drop it somewhere before the SDK in gcc's include path....but it literally the easiest way to go
Umm...putting something in the tools' directories is the worst idea. Just put it in the sketch directory. What is so awkward about that?
If you have multiple sketches, put it in the libraries directory. However, if the configurations are different for each sketch, you have to put a copy in each sketch directory.
Both of those choices allow easy distribution and configuration control... unless you are also planning to distribute the entire tool chain.
You make this sound much harder than copying a few files.
-dev:
Umm...putting something in the tools' directories is the worst idea. Just put it in the sketch directory. What is so awkward about that?
If you have multiple sketches, put it in the libraries directory. However, if the configurations are different for each sketch, you have to put a copy in each sketch directory.
Both of those choices allow easy distribution and configuration control... unless you are also planning to distribute the entire tool chain.
You make this sound much harder than copying a few files.
I was not thinking tools, but I already have all the libraries and everything (but not SDK/IDE) in the source control already, i was actually thinking of treating it as a library, but I ASSUMED the main source directory will come after the SDK, which now I am starting to question based on your words, Is it that simple? - will my source folder take priority over SDK? Easy enough to check...
-HH
Update: Nope, does not pick it up from either the src or root directory of my project - although this may be a Sloeber issue, will try building with IDE
