Adjusting serial buffer size in IDE, mk2

i've tried posting the below on the Arduino Developers
mailing list (Redirecting to Google Groups), but my posting never even appeared. so will try posting it here in the hope that one of the developers still frequents this forum:

for some applications, it is necessary (or highly convenient) to have the serial buffer sizes different to the default sizes. to achieve this, the user has to modify the contents of the HardwareSerial.h file in ...\hardware\arduino\avr\cores\arduino. however, it is undesirable to have the (potentially inexperienced) user tinkering with these files.

i would like to propose the following solution be implemented, to allow the user to alter the serial buffer sizes:

  1. in HardwareSerial.h add in the following single (bolded) line:

#if (RAMEND < 1000)
#define SERIAL_TX_BUFFER_SIZE 16
#define SERIAL_RX_BUFFER_SIZE 16
#else
#define SERIAL_TX_BUFFER_SIZE 64
#define SERIAL_RX_BUFFER_SIZE 64
#endif
#include "OVERRIDE\serial_buffer_size.h" // ##### override default buffer size #####
#endif

  1. create a subdirectory ...\hardware\arduino\avr\cores\arduino*OVERRIDE*

  2. in that subdirectory create a zero-length file called serial_buffer_size.h

the above three steps provide a safe location where (a) a user can override default settings, which can (b) easily be restored to 'factory default' by truncating the files back to zero length, all without (c) altering existing functionality or compiler output provided step 3. holds. in principal the same method could be used to allow for overriding other settings on other core libraries.

i would then like to suggest that the following IDE changes be made to facilitate using the above provisions:

  1. to the Arduino IDE add an entry under the "Tools" menu called "OVERRIDES" where the user can select any settings that they wish to override, and select amongst safe override values. for instance, there may be Tools -> OVERRIDES -> Serial RX Buffer Size with options of 'default', '16', '64', '256', '1024' (and the same for Serial TX Buffer Size).

what do the developers think of this idea? steps 1 to 3 could initially be carried out with minimal effort (i have tested out steps 1 to 3 and verified they work as desired), with step 4 (or some variation thereof) implemented later on. overrides could be provided as global irrespective of the board selected, or made specific to only one board or type of processor.

i would much appreciate feedback.

cheers,
rob :slight_smile:

https://groups.google.com/a/arduino.cc/forum/#!topic/developers/4q9k_gvKpLk

yes, subsequent to starting the current thread, cmaglie was kind enough to sort out the problem i was having with the developers mailing list, so i was then able to post there. but there is no harm in a discussion here as well.

cheers,
rob :slight_smile:

Do this...

• Ensure the Arduino IDE is not running

• Locate boards.txt

• Make a backup copy

• Open boards.txt using your favourite Linux friendly text editor. I use Visual Studio or Delphi.

• Locate an entry of interest. I used the one for the Uno.

• Select the entire entry

• Copy the entire entry

• Navigate to the bottom of boards.txt

• Paste the entry

• Carefully change all of the tags in the new entry to something unique. I changed "uno." to "unox."

• Change the entry name. I changed my entry name to this...

unox.name=Arduino Uno - Custom Build

• Add this one option to the entry...

unox.build.extra_flags=-include "{build.path}/build_options.h"

• Save boards.txt

• Close the text editor

• Open the Arduino IDE

• Create a new sketch

• Add a tab named "build_options.h"

• Add whatever you want to "build_options.h". This is what I used...

#ifndef build_options_h
#define build_options_h

#define SERIAL_TX_BUFFER_SIZE 16
#define SERIAL_RX_BUFFER_SIZE 16

#endif

• Verify

• Enjoy

Note: When you upgrade the IDE boards.txt is removed. Make a backup before upgrading.


Complete example...

##############################################################

unox.name=Arduino Uno - Custom Build

unox.vid.0=0x2341
unox.pid.0=0x0043
unox.vid.1=0x2341
unox.pid.1=0x0001
unox.vid.2=0x2A03
unox.pid.2=0x0043

unox.upload.tool=avrdude
unox.upload.protocol=arduino
unox.upload.maximum_size=32256
unox.upload.maximum_data_size=2048
unox.upload.speed=1000000

unox.bootloader.tool=avrdude
unox.bootloader.low_fuses=0xFF
unox.bootloader.high_fuses=0xDE
unox.bootloader.extended_fuses=0x05
unox.bootloader.unlock_bits=0x3F
unox.bootloader.lock_bits=0x0F
unox.bootloader.file=optiboot/optiboot_atmega328.hex

unox.build.mcu=atmega328p
unox.build.f_cpu=16000000L
unox.build.board=AVR_UNO
unox.build.core=arduino
unox.build.variant=standard

unox.build.extra_flags=-include "{build.path}/build_options.h"

##############################################################

A big thank you to foraidt...

Coding Badly: excellent hint! :smiley:

after a bit of tinkering, found that the same effect can be achieved globally by changing the appropriate entry in the file platform.txt from:
build.extra_flags=
to:
build.extra_flags=-include "{build.path}/options.h"

unfortunately this means that the compile will fail if options.h does not exist in the sketch directory. is there any way to do an include that does not throw up an error message if the options.h file does not exist?

cheers,
rob :slight_smile:

Ugh. You're kind of a needy thing, aren't ya?

  1. Create and use a custom board entry. Which is one reason I suggested that route.

  2. Or, do this...

• Create an empty file named build_options.h in the core directory ({ArduinoBase}\hardware\arduino\avr\cores\arduino)

• Change the build.extra_flags option to this...

unox.build.extra_flags=-I "{build.path}" -include build_options.h

If you find limitations or problems please report back. If the technique works well please report back. At a minimum the Arduino folks are likely to add an empty options file to the core, which will make upgrading a bit less painful, but only if this technique is seen to be reliable.

Caveat: As far as I can tell, the options file (build_options.h) is not included in dependency checking. If you change that file you will very likely have to restart the IDE.

Coding Badly: you're a legend, that works absolutely brilliantly! :smiley:

the procedure:

(1) in the file platform.txt change the following line:
build.extra_flags=
to:
build.extra_flags=-I "{build.path}" -include options.h

(2) in the Arduino\hardware\arduino\avr\cores\arduino directory
create an empty file called options.h

now in any sketch directory you can create a file called options.h
that contains the things you want to change locally to that sketch.
in my case this is:
#define SERIAL_TX_BUFFER_SIZE 64
#define SERIAL_RX_BUFFER_SIZE 1024

(i suspect that both RX and TX sizes need to be specified as a pair)

when i open my project in the IDE, the options.h file is also opens. if i change the contents of options.h those changes are reflected in the next compile without needing to restart the IDE. it all seems to work quite seamlessly.

cheers,
rob :slight_smile:

"Legend"? Is that like Seinfeld's "breathtaking"?

Glad you have it working!

not quite - i guess it is a new zealand term (where i live). it means you've done something remarkable, and is a compliment. i just need to persuade the arduino developers to incorporate your method into the next release of the ide:
https://groups.google.com/a/arduino.cc/forum/#!topic/developers/4q9k_gvKpLk

cheers,
rob :slight_smile:

Oh, I get it. I just could not resist the opportunity to wind you up.

:smiley: i used to be a seinfeld fan way back when he was on tv over here.

i've tried to push things along a little more on the developers forum, so hopefully something may happen. in the meantime, i've been playing around with creating a tiny utility that can have an arduino .HEX file embedded in it and will act as a standalone firmware uploader - so you can send out a small standalone .EXE file that folks run to upload an updated (compiled) sketch to an arduino that is contained within a product.

to this end, i'm trying to figure how much of the STK500v1 protocol is actually implemented in the various arduino bootloaders.

cheers,
rob :slight_smile:

Start with ArduinoISP. That will tell you what avrdude needs to function correctly. I believe the two are exactly mated; ArduinoISP implements the minimum necessary for avrdude to program a target.

Any reason you don't just embed / include avrdude?

of the two existing methods of uploading code to an arduino: the Arduino IDE from source, and avrdude using a pre-compiled .HEX file, alas i've found neither are achievable by the typical user.

while the Arduino IDE is quite an accomplishment in providing accessibility to microcontroller programming for many users, it is an enormous download and beyond the ability of the average user to master for the single task of updating firmware.

and avrdude is itself still quite large (1mb), consists of four parts (avrdude.exe, avrdude.conf, libusb0.dll, the firmware .HEX to be uploaded), and requires a 'cryptic' command line incantation to make work. admittedly, there are wrappers (from batch files to small GUI applications), but most that i've looked at still leave open the opportunity for the end user to get horribly lost or make mistakes.

my solution consists of a single windows executable (or linux binary) that contains the .HEX file embedded within, which the user just needs to run. port selection can be done via a GUI or specified on the command line.

i started out with a trace output from avrdude:
avrdude -patmega328p -carduino -PCOM5 -b57600 -D -Uflash:w:ICSP_v1C.hex -v -v -v -v >trace.txt 2>&1

the 2>&1 at the end is important under windows, as avrdude seems to direct all of its output to stderr, which normally can not be redirected away from the console. this is perhaps a bit of a bug.

that output provided me with details of how avrdude talks to an arduino. i then got hold of microchip's "AVR061: STK500 Communication Protocol" (doc2525.pdf) and used that to derive meaning from avrdude's output. finally i looked over this page:
http://baldwisdom.com/bootloading/
which provided some hints about what things i didn't need to bother with when talking to an arduino bootloader.

i've now got working code, but i am still not 100% sure it will work with any arduino bootloader. fyi, below is my current code (written in C). the firmware has already been loaded into an array of unsigned char called ICSP (in ICSP_v1C.inc). outside of view is also a serial port unit and assignment of 3 to baud_rate.

i'd welcome comment!

cheers,
rob :slight_smile:

    if (baud_rate < 5)							// invoked by a baud rate of 1, 2, 3, or 4.
    {
        #include "ICSP_v1C.inc"
        int i, n;
        unsigned char buffer [140];					// 0x80 + 12d
        int bps[] = {0, 9600, 19200, 57600, 115200 };		// known arduino bootloader baud rates
 
        baud_rate = bps[baud_rate];
        if (serial_open (port, baud_rate, 100) < 0) {
            printf ("failed to configure port\n");
            serial_close();
            exit (-1);
        }
        printf("%i baud ", baud_rate);

        for (i = 0; i < 40; i++) {

//          sprintf ((char*) buffer, "%c%c", 0x30, 0x20);
            buffer[0] = 0x30;						// get synchronization
            buffer[1] = 0x20;

            serial_write (buffer, 2);
            printf (".");
            n = serial_read (buffer, 2);
            if ((n == 2) && (buffer[0] == 0x14) && (buffer[1] == 0x10)) i=100;
        }

        if (i < 100) { 
            printf (" arduino/STK500 not found\n");
            serial_close(); 
            exit (-1);
        }
        printf (" synchronized\n"); 

//      sprintf ((char*) buffer, "%c%c", 0x50, 0x20);
        buffer[0] = 0x50;						// enter program mode
        buffer[1] = 0x20;
        serial_write (buffer, 2);
        serial_read (buffer, 2);

        if ((n != 2) || (buffer[0] != 0x14) || (buffer[1] != 0x10)) { 
            printf ("failed to enter program mode\n");
            serial_close();
            exit (-1);
        }
        
//      sprintf ((char*) buffer, "%c%c", 0x75, 0x20);
        buffer[0] = 0x75;						// read signature bytes (3)
        buffer[1] = 0x20;
        serial_write (buffer, 2);
        n = serial_read (buffer, 5);

        if ((n != 5) || (buffer[0] != 0x14) || (buffer[4] != 0x10)) { 
            printf ("failed to get signature\n");
            serial_close();
            exit (-1);
        }
        unsigned ID = (buffer[1] << 16) + (buffer[2] << 8) + buffer[3];
        printf ("signature = %06x   device = %s\n", ID, ID == 0x1e950f ? "ATmega328P" : "(wrong uP)"); 
 
        for (i = 0; i < sizeof(ICSP) / 0x80; i++) printf (".");
        for (i = 0; i < sizeof(ICSP) / 0x80; i++) printf ("\b");

        for (i = 0; i < sizeof(ICSP); i += 0x80) {
            printf ("#");

//          sprintf ((char*) buffer, "%c%c%c%c", 0x55, (i >> 1) % 0x100, (i >> 1) / 0x100, 0x20);
            buffer[0] = 0x55;						// load address
            buffer[1] = (i >> 1) % 0x100;				// address low (word boundary)
            buffer[2] = (i >> 1) / 0x100;				// address high
            buffer[3] = 0x20;
            serial_write (buffer, 4);
            n = serial_read (buffer, 2);

            if ((n != 2) || (buffer[0] != 0x14) || (buffer[1] != 0x10)) { 
                printf ("\nfailed to load address %04x\n", i);
                serial_close();
                exit (-1);
            }

//          sprintf ((char*) buffer, "%c%c%c%c", 0x64, 0x00, 0x80, 0x46);
            buffer[0] = 0x64;						// program page
            buffer[1] = 0x00;						// length high (in bytes, NOT words)
            buffer[2] = 0x80;						// length low (order reverse to address)
            buffer[3] = 0x46;
            memcpy (&buffer[4], &ICSP[i], 0x80);		// data (128 bytes)
            buffer[4 + 0x80] = 0x20;
            serial_write (buffer, 4 + 0x80 + 1);
            n = serial_read (buffer, 2);

            if ((n != 2) || (buffer[0] != 0x14) || (buffer[1] != 0x10)) { 
                printf ("\nfailed to program page\n");
                serial_close();
                exit (-1);
            }
        }
        printf ("\n");

//      sprintf ((char*) buffer, "%c%c", 0x51, 0x20);
        buffer[0] = 0x51;						// leave program mode
        buffer[1] = 0x20;
        serial_write (buffer, 2);
        n = serial_read (buffer, 2);

        if ((n != 2) || (buffer[0] != 0x14) || (buffer[1] != 0x10)) { 
            printf ("failed to exit program mode\n");
            serial_close();
            exit (-1);
        }
        printf ("firmware uploaded to 'ascii ICSP' adapter OK\n"); 

        serial_close();
        exit (0);
    }
            buffer[0] = 0x30;						// get synchronization
            buffer[1] = 0x20;

Magic constants, like 0x30 and 0x20, are a maintenance nightmare. I suggest using the constants provided by Atmel...

#define CRC_EOP             0x20  // 'SPACE'
#define STK_GET_SYNC        0x30  // '0'

good point, i've now switched over to using the Atmel constants. must admit, it has made the code a tad easier to follow!

i'm particularly interested in determining how much of a subset of the STK500v1 protocol each of the arduino bootloaders requires to be present in communications from the host. at the moment my code implements the following:

  1. STK_GET_SYNC (loop until STK_INSYNC)
  2. STK_ENTER_PROGMODE
  3. STK_READ_SIGN (expect 0x1e950f for a 328p, warn if not)

do
4. STK_LOAD ADDRESS
5. STK_PROG_PAGE
loop until all data written

  1. STK_LEAVE_PROGMODE

so i've left out signon, all the fuse and lock stuff, and the setting of programming parameters. plus i don't bother with verifying. what i have seems sufficient to keep optiboot happy, but i am unsure if it is sufficient for the other Arduino bootloaders out there.

i guess WestfW may be the person to ask about this.

cheers,
rob :slight_smile:

The entire Optiboot command loop is only 160 lines. I suspect your eyeballs will work just fine to determine which STK500v1 commands are used / important...

yes, the optiboot commands are well defined, and i have already seen the list:

(supported)
STK_LOAD_ADDRESS
STK_PROG_PAGE
STK_READ_PAGE
STK_READ_SIGN
STK_LEAVE_PROGMODE
STK_GET_PARAMETER

(harmlessly ignored)
STK_UNIVERSAL
STK_SET_DEVICE
STK_SET_DEVICE_EXT
(from: HowOptibootWorks · Optiboot/optiboot Wiki · GitHub)

what i am interested in is hearing about what commands other arduino bootloaders require to be issued for correct operation. indeed, is there any definitive list of bootloaders that have commonly been loaded into shipped arduinos?

cheers,
rob :slight_smile:

Well, all the source code is under one of these directories...

{ArduinoRoot}\hardware\arduino\bootloaders\
{ArduinoRoot}\hardware\arduino\avr\bootloaders\

The atmega, optiboot, and stk500v2 bootloaders very likely cover the vast majority of the boards you could encounter. Those three are certainly a good place to start.

Given the fact that the constants start with "STK_" I suspect less than one minute with grep will answer your question.

were any 328p-based arduinos ever made that shipped with an STK500v2 bootloader? i was under the impression that only the 'mega' boards needed to use this protocol, due to the v1 protocol being limited to addressing no more than 128k of flash memory.

i've looked through a couple of the sets of source code, and see a common philosophy of implementing the minimum set of commands required to function. this leaves me pretty comfortable sticking to my 6-step approach above (although STK_ENTER_PROGMODE isn't needed). bearing in mind that i am only interested in supporting 328p-based arduinos.

cheers,
rob :slight_smile: