Serial port "rx_buffer" gobbles up unnessary RAM

The serial port functions in HardwareSerial.cpp allocate 128 bytes of buffer for receiving data, this seems excessive. 16 bytes would be plenty at serial port data speeds.

At the moment, calling any serial port functions will use up 128 bytes of RAM.

Even just "Serial.begin()" will mean the RAM is used up.

Mostly I use the serial port for sending debug output, I never receive any data so this is a BIG overhead.

You are free to change it at your liking. the core file to edit is named HardwareSerial.cpp and is located in the arduino cores\arduino directory.

The relevant section is here:

// Define constants and variables for buffering incoming serial data.  We're
// using a ring buffer (I think), in which rx_buffer_head is the index of the
// location to which to write the next incoming character and rx_buffer_tail
// is the index of the location from which to read.
#if (RAMEND < 1000)
  #define RX_BUFFER_SIZE 32
#else
  #define RX_BUFFER_SIZE 128
#endif

Lefty

Yes, I saw that earlier (that's how I know it's 128 bytes...)

The problem is if I give my code to other people it won't work if they don't edit their copy of HardwareSerial.cpp.

I think it would be a good idea to do something like this in the library:

#ifndef RX_BUFFER_SIZE

#if (RAMEND < 1000)
  #define RX_BUFFER_SIZE 32
#else
  #define RX_BUFFER_SIZE 128
#endif

#endif

That way people can override the setting without any editing of the library.

Right now I think I have to copy HardwareSerial.cpp into my sketch with a different name and modify it there.

You can't overide the buffer size with an #ifndef since the size is set when the library is compiled, not in the context of your sketch.

In Arduino 1.0 there will also be a tx buffer that must be the same size as the rx buffer.

I have my own serial library that allows buffer sizes to be set at run time but, as you say, I can't release code that uses it since its ISR conflicts with programs that use Serial.

Would it be an idea to propose the buffersize as an optional parameter in

Serial.begin (long baudrate, int buffersize = 128);

// Yes I would like to set stop and startbits too ...

The problem is if I give my code to other people it won't work if they don't edit their copy of HardwareSerial.cpp.

You could add a readme.txt file in which the mods are described.

question: is your sketch out of RAM? do you need to recover those 128 bytes?

I like buffer size in begin(). In 1.0 begin() needs to specify RX and TX buffer sizes.

I implemented run-time buffer sizes in my rewrite of HardwareSerial. I now plan to remove output buffering so I will have begin(baudRate, bufSize).

I agree, it would be nice to set other options like stop bits and parity.

Better error handling would be nice. Detecting receive overruns would be useful.

I shouldn't complain since I am not actively trying to get these changes into the official distribution. I just don't have the temperament to do that. I soon become frustrated with the process.

Detecting receive overruns would be useful.

That would involve just setting a flag when data is received, not to difficult I think ... (take a deep breath and dive into some code)....

[removed raw thoughts]

See post below for working code

fat16lib:
I like buffer size in begin(). In 1.0 begin() needs to specify RX and TX buffer sizes.

That would need dynamic memory allocation. It seems much more efficient to use #define/#ifdef for this.

WRT overflow() I added the lost char counter into the ringbuffer

HardwareSerial.cpp

struct ring_buffer
{
  unsigned char buffer[RX_BUFFER_SIZE];
  int head;
  int tail;
  long lost;  // <<<< ADDED
};

and

inline void store_char(unsigned char c, ring_buffer *rx_buffer)
{
  int i = (unsigned int)(rx_buffer->head + 1) % RX_BUFFER_SIZE;

  // if we should be storing the received character into the location
  // just before the tail (meaning that the head would advance to the
  // current location of the tail), we're about to overflow the buffer
  // and so we don't write the character or advance the head.
  if (i != rx_buffer->tail) {
    rx_buffer->buffer[rx_buffer->head] = c;
    rx_buffer->head = i;
  } else rx_buffer->lost++;  // <<<<<<<<<<< ADDED
}

Added this function

// note: lost counter is not reset ...
uint32_t HardwareSerial::overflow(void)
{
 return  _rx_buffer->lost;
}

Added in hardwareSerial.h the prototype of course.

virtual uint32_t overflow(void);

Works like a charm!

test sketch, just keep sending chars :slight_smile:

void setup()
{
  Serial.begin(115200);
}

void loop()
{
  Serial.println(Serial.overflow());
  Serial.println(Serial.available());
  Serial.println("----------");
  delay(1000);
}

reported as issue 637, - http://code.google.com/p/arduino/issues/detail?id=637 -

It's dead simple to modify HardwareSerial, that's why I have rewritten it the way I like for my own use.

The problem is you can't ask users to replace HardwareSerial in order to use your software. HardwareSerial is in the Arduino core in a fundamental way since it uses the serial ISRs.

I don't see a clean way to use a #define in a sketch to override the size of the buffers in the HardwareSerial library.

Show me a working example of how to change the size of the ISR receive buffer in the library by using a #define in the user's sketch.

I would like to avoid dynamic memory but simple ideas using a #define don't work.

I don't see a clean way to use a #define in a sketch to override the size of the buffers in the HardwareSerial library.

I don't even see an ugly way :wink:
pity ...

I thought more and your right, my ugly way won't work.

The fact that HardwareSerial.h is included in almost every library via WProgram.h or Arduino.h in 1.0 killed my ugly idea.

fat16lib:
I don't see a clean way to use a #define in a sketch to override the size of the buffers in the HardwareSerial library.

Show me a working example of how to change the size of the ISR receive buffer in the library by using a #define in the user's sketch.

There isn't one, that's why this is in the "suggestions" area...!

Why suggest it if it can't be done with a #define.

How would you modify the library to make a #define work?

I have written dozens of Arduino libraries and don't know I would use a #define in a user sketch to set the size of an ISR receive buffer in a library.

fat16lib:
How would you modify the library to make a #define work?

I have written dozens of Arduino libraries and don't know I would use a #define in a user sketch to set the size of an ISR receive buffer in a library.

Like this:

#ifndef RX_BUFFER_SIZE
#define RX_BUFFER_SIZE 128
#endif

Then the user can do:

#define RX_BUFFER_SIZE 16
#include "serial.h"
...

The user #define will not be used when the library is compiled so this will not work.

fat16lib:
The user #define will not be used when the library is compiled so this will not work.

Ok, how about something like this in the header file?

#ifndef USER_RX_BUFFER
static byte rx_buffer[128];
#endif


class Serial {

  // User provides the rx buffer
  void begin(int baud, byte *rxBuffer, int rxBufferSize);

#ifndef USER_RX_BUFFER
  // Default - use the buffer defined in this header
  void begin(int baud) {
    begin(baud, rx_buffer, sizeof(rx_buffer));
  }
#endif

};

Then the user can define a macro "USER_RX_BUFFER" to exclude the built-in buffer and provide his/her own.

No there are libraries that call begin() to change baud rate for auto baud detection. These libraries also include the header so multiple buffers will be created and used.

My idea of dynamic RAM will not even work with existing code.

Serial is designed so you can call begin() and end() as often as you please in any place.

I give up on this issue.

Serial is designed so you can call begin() and end() as often as you please in any place.

So begin should

  • test if a buffer exists,
  • free it and
  • malloc a new one.

end should

  • test if a buffer exists,
  • free it

fat16lib:
No there are libraries that call begin() to change baud rate for auto baud detection. These libraries also include the header so multiple buffers will be created and used.

Ok...maybe making it completely transparent to people in all situations isn't going to easy.

But ... at least you make the buffer in the library tiny (eight bytes?) and give power users the option to expand it via a function call. Space is very tight on Arduino, there's no point in everybody paying "power user" prices for things they don't even use.