[SOLVED] Increase serial buffer size on Arduino Mega 2560

[Bavilo]

Hello everyone,

I have been trying to increase the buffer size of my Arduino Mega 2560 (Clone with CH340G chip).

I have tried the following:
in, \Arduino\hardware\arduino\avr\cores\arduino\HardwareSerial.h

Code: [Select]

#define SERIAL_TX_BUFFER_SIZE 16
#else
#define SERIAL_TX_BUFFER_SIZE 64
#endif
#endif
#if !defined(SERIAL_RX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_RX_BUFFER_SIZE 16
#else
#define SERIAL_RX_BUFFER_SIZE 64

to

Code: [Select]

#define SERIAL_TX_BUFFER_SIZE 16
#else
#define SERIAL_TX_BUFFER_SIZE 256
#endif
#endif
#if !defined(SERIAL_RX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_RX_BUFFER_SIZE 16
#else
#define SERIAL_RX_BUFFER_SIZE 256


In my sketch I am receiving strings from Serial1 using

Code: [Select]

raw_bluetooth_data = Serial1.readStringUntil('!');

From Android I am sending a String larger than 64 bytes. Sometimes if the sketch takes too long I can see the string being truncated at 64 bytes. I don`t see why this could happen since I have increase the buffer size to 256.

I have re-uploaded the code after making changes in the headerfile. Also restarting Arduino doesn`t help.

EDIT: Also changing the receiving method to this http://forum.arduino.cc/index.php?topic=396450 causes the String to truncate if the sketch takes too long.

The Sketch sometimes takes too long because I am polling one wire temperature sensors which cause blocking.

Any advice?

Thanks!

[sterretje]

Your Mega has 4 serial ports; guess you changed the wrong one.

If you redesign your software, there is probably no need to increase the buffer size. Have a look at serial input basics - updated. The trick is to regularly read the serial data.

[Bavilo]

Your Mega has 4 serial ports; guess you changed the wrong one.

If you redesign your software, there is probably no need to increase the buffer size. Have a look at serial input basics - updated. The trick is to regularly read the serial data.

Could you perhaps tell me where I can change the buffer size for Serial1? In HardwareSerial.h it doesn't mention any specific buffer sizes for the seperate ports...

[sterretje]

HardwareSerial1.cpp or .h? No system at hand to check.

[Bavilo]

HardwareSerial1.cpp or .h? No system at hand to check.

I changed HardwareSerial.h

Code: [Select]

/*
  HardwareSerial.h - Hardware serial library for Wiring
  Copyright (c) 2006 Nicholas Zambetti.  All right reserved.

  This library is free software; you can redistribute it and/or
  modify it under the terms of the GNU Lesser General Public
  License as published by the Free Software Foundation; either
  version 2.1 of the License, or (at your option) any later version.

  This library is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
  Lesser General Public License for more details.

  You should have received a copy of the GNU Lesser General Public
  License along with this library; if not, write to the Free Software
  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA

  Modified 28 September 2010 by Mark Sproul
  Modified 14 August 2012 by Alarus
  Modified 3 December 2013 by Matthijs Kooijman
*/

#ifndef HardwareSerial_h
#define HardwareSerial_h

#include <inttypes.h>

#include "Stream.h"

// Define constants and variables for buffering incoming serial data.  We're
// using a ring buffer (I think), in which head is the index of the location
// to which to write the next incoming character and tail is the index of the
// location from which to read.
// NOTE: a "power of 2" buffer size is reccomended to dramatically
//       optimize all the modulo operations for ring buffers.
// WARNING: When buffer sizes are increased to > 256, the buffer index
// variables are automatically increased in size, but the extra
// atomicity guards needed for that are not implemented. This will
// often work, but occasionally a race condition can occur that makes
// Serial behave erratically. See https://github.com/arduino/Arduino/issues/2405
#if !defined(SERIAL_TX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_TX_BUFFER_SIZE 16
#else
#define SERIAL_TX_BUFFER_SIZE 256
#endif
#endif
#if !defined(SERIAL_RX_BUFFER_SIZE)
#if ((RAMEND - RAMSTART) < 1023)
#define SERIAL_RX_BUFFER_SIZE 16
#else
#define SERIAL_RX_BUFFER_SIZE 256
#endif
#endif
#if (SERIAL_TX_BUFFER_SIZE>256)
typedef uint16_t tx_buffer_index_t;
#else
typedef uint8_t tx_buffer_index_t;
#endif
#if  (SERIAL_RX_BUFFER_SIZE>256)
typedef uint16_t rx_buffer_index_t;
#else
typedef uint8_t rx_buffer_index_t;
#endif

// Define config for Serial.begin(baud, config);
#define SERIAL_5N1 0x00
#define SERIAL_6N1 0x02
#define SERIAL_7N1 0x04
#define SERIAL_8N1 0x06
#define SERIAL_5N2 0x08
#define SERIAL_6N2 0x0A
#define SERIAL_7N2 0x0C
#define SERIAL_8N2 0x0E
#define SERIAL_5E1 0x20
#define SERIAL_6E1 0x22
#define SERIAL_7E1 0x24
#define SERIAL_8E1 0x26
#define SERIAL_5E2 0x28
#define SERIAL_6E2 0x2A
#define SERIAL_7E2 0x2C
#define SERIAL_8E2 0x2E
#define SERIAL_5O1 0x30
#define SERIAL_6O1 0x32
#define SERIAL_7O1 0x34
#define SERIAL_8O1 0x36
#define SERIAL_5O2 0x38
#define SERIAL_6O2 0x3A
#define SERIAL_7O2 0x3C
#define SERIAL_8O2 0x3E

class HardwareSerial : public Stream
{
  protected:
    volatile uint8_t * const _ubrrh;
    volatile uint8_t * const _ubrrl;
    volatile uint8_t * const _ucsra;
    volatile uint8_t * const _ucsrb;
    volatile uint8_t * const _ucsrc;
    volatile uint8_t * const _udr;
    // Has any byte been written to the UART since begin()
    bool _written;

    volatile rx_buffer_index_t _rx_buffer_head;
    volatile rx_buffer_index_t _rx_buffer_tail;
    volatile tx_buffer_index_t _tx_buffer_head;
    volatile tx_buffer_index_t _tx_buffer_tail;

    // Don't put any members after these buffers, since only the first
    // 32 bytes of this struct can be accessed quickly using the ldd
    // instruction.
    unsigned char _rx_buffer[SERIAL_RX_BUFFER_SIZE];
    unsigned char _tx_buffer[SERIAL_TX_BUFFER_SIZE];

  public:
    inline HardwareSerial(
      volatile uint8_t *ubrrh, volatile uint8_t *ubrrl,
      volatile uint8_t *ucsra, volatile uint8_t *ucsrb,
      volatile uint8_t *ucsrc, volatile uint8_t *udr);
    void begin(unsigned long baud) { begin(baud, SERIAL_8N1); }
    void begin(unsigned long, uint8_t);
    void end();
    virtual int available(void);
    virtual int peek(void);
    virtual int read(void);
    int availableForWrite(void);
    virtual void flush(void);
    virtual size_t write(uint8_t);
    inline size_t write(unsigned long n) { return write((uint8_t)n); }
    inline size_t write(long n) { return write((uint8_t)n); }
    inline size_t write(unsigned int n) { return write((uint8_t)n); }
    inline size_t write(int n) { return write((uint8_t)n); }
    using Print::write; // pull in write(str) and write(buf, size) from Print
    operator bool() { return true; }

    // Interrupt handlers - Not intended to be called externally
    inline void _rx_complete_irq(void);
    void _tx_udr_empty_irq(void);
};

#if defined(UBRRH) || defined(UBRR0H)
  extern HardwareSerial Serial;
  #define HAVE_HWSERIAL0
#endif
#if defined(UBRR1H)
  extern HardwareSerial Serial1;
  #define HAVE_HWSERIAL1
#endif
#if defined(UBRR2H)
  extern HardwareSerial Serial2;
  #define HAVE_HWSERIAL2
#endif
#if defined(UBRR3H)
  extern HardwareSerial Serial3;
  #define HAVE_HWSERIAL3
#endif

extern void serialEventRun(void) __attribute__((weak));

#endif



HardwareSerial1.cpp just contains this:

Code: [Select]

/*
  HardwareSerial1.cpp - Hardware serial library for Wiring
  Copyright (c) 2006 Nicholas Zambetti.  All right reserved.

  This library is free software; you can redistribute it and/or
  modify it under the terms of the GNU Lesser General Public
  License as published by the Free Software Foundation; either
  version 2.1 of the License, or (at your option) any later version.

  This library is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
  Lesser General Public License for more details.

  You should have received a copy of the GNU Lesser General Public
  License along with this library; if not, write to the Free Software
  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA

  Modified 23 November 2006 by David A. Mellis
  Modified 28 September 2010 by Mark Sproul
  Modified 14 August 2012 by Alarus
  Modified 3 December 2013 by Matthijs Kooijman
*/

#include "Arduino.h"
#include "HardwareSerial.h"
#include "HardwareSerial_private.h"

// Each HardwareSerial is defined in its own file, sine the linker pulls
// in the entire file when any element inside is used. --gc-sections can
// additionally cause unused symbols to be dropped, but ISRs have the
// "used" attribute so are never dropped and they keep the
// HardwareSerial instance in as well. Putting each instance in its own
// file prevents the linker from pulling in any unused instances in the
// first place.

#if defined(HAVE_HWSERIAL1)

#if defined(UART1_RX_vect)
ISR(UART1_RX_vect)
#elif defined(USART1_RX_vect)
ISR(USART1_RX_vect)
#else
#error "Don't know what the Data Register Empty vector is called for Serial1"
#endif
{
  Serial1._rx_complete_irq();
}

#if defined(UART1_UDRE_vect)
ISR(UART1_UDRE_vect)
#elif defined(USART1_UDRE_vect)
ISR(USART1_UDRE_vect)
#else
#error "Don't know what the Data Register Empty vector is called for Serial1"
#endif
{
  Serial1._tx_udr_empty_irq();
}

HardwareSerial Serial1(&UBRR1H, &UBRR1L, &UCSR1A, &UCSR1B, &UCSR1C, &UDR1);

// Function that can be weakly referenced by serialEventRun to prevent
// pulling in this file if it's not otherwise used.
bool Serial1_available() {
  return Serial1.available();
}

#endif // HAVE_HWSERIAL1


I assume HardwareSerial.h changes the buffer size for all ports. But yet...it still doesn't

[Bavilo]

Ok I have solved it...

the buffer size had to be set in a different directory (which I didn't even know of!)

AppData\Local\Arduino15\packages\arduino\hardware\avr\1.6.11\cores\arduino\HardwareSerial.h


no more truncating now.

[heinburgh]

Can anyone tell me if the file containing the setting for hardware serial buffer size has changed in the newer versions of IDE? I use 1.8.1 for mac os and cannot locate HardwareSerial.h anywhere on my harddrive.

[sterretje]

MACs have their own way of doing stuff as far as I know. If memory serves me right, use the command line and the 'find' command to locate the file.

[heinburgh]

I found it.

Code: [Select]

https://arduino.stackexchange.com/questions/33727/where-is-arduino-h-on-a-mac-can-i-download-a-copy-if-missing

Yes I'm relatively new to mac os and I would never have thought it is kept inside the app package.

NOW - after I changed the buffer size, how does one go about testing and confirming your buffer size? Is there a simple sketch that will disply your buffer sizes?

[sterretje]

Just add a long delay in setup; during this delay continuously send data to your Arduino. And next display what Serial.available returns. At 9600 baud, a byte takes about a millisecond to be transmitted so you can calculate how long you have to wait (roughly).

And the question is why you think you need to increase buffer size?

[heinburgh]

It's an EID counter. Each 16 byte EID number comes from an external reader. When you have two rfid's hanging around the antenna without moving past (sheep in my case), the reader has a habit of reading and sending the id's alternately as fast as it can (currently at 9600 baud.) When my mega receives each EID it has to check each EID number against a list of around 1,000 others. Using binary search I cut down the search time to between 7ms and 78ms per id, depending on where in the list it appears. When the EID is found there are five fields taken from a reference file and attached to that EID and written to a log file. All this for each and every EID, and this while the reader (sometimes) pumps out the numbers at pace.


With the buffer at 64 bytes it only takes four EID numbers to fill the buffer and make my mega go nuts. After I adjusted the buffer up to 256 bytes yesterday she eats up the numbers like a hungry lion. No matter how fast I throw the EID tags at it, it digests the data and patiently wait for more.

[Carlillosgld]

Ok I have solved it...

the buffer size had to be set in a different directory (which I didn't even know of!)

AppData\Local\Arduino15\packages\arduino\hardware\avr\1.6.11\cores\arduino\HardwareSerial.h


no more truncating now.

Excelente mil gracias...!!!
En mi caso use #define SERIAL_TX_BUFFER_SIZE 192 //bytes que necesitaba para no complicarme la existencia

ruta de archivo modificado en mi caso:

C:\Users\Acer\AppData\Local\Arduino15\packages\arduino\hardware\avr\1.6.20\cores\arduino\HardwareSerial.h
 
En Mega2560
Actual SRAM Free Memory: Bytes = 4828.00 of 8096 bytes
59.63 %

[late_id]

where I can found HardwareSerial.cpp if I'm using Arduino IDE?

[heinburgh]

I hate to open this topic again, but I recently upgraded my IDE to 1.8.9, macOS version.

The moment I upgraded, none of my sketches that uploads data to ubidots via sim800 works any more - the string that contains the data that is sent to through the modem is truncated to 64 characters. So realising that the hardwareserial.h file must have been overwritten I edited the new file, to increase the buffer size to 256 again.

But I cannot seem to force the update when I compile the sketch. I can't get it to say "New Build..." while uploading the sketch.

How do I force this?