Go Down

Topic: Adjusting serial buffer size in IDE, mk2 (Read 9578 times) previous topic - next topic

robert rozee

i've tried posting the below on the Arduino Developers
mailing list (https://groups.google.com/a/arduino.cc/forum/#!forum/developers), but my posting never even appeared. so will try posting it here in the hope that one of the developers still frequents this forum:


for some applications, it is necessary (or highly convenient) to have the serial buffer sizes different to the default sizes. to achieve this, the user has to modify the contents of the HardwareSerial.h file in ...\hardware\arduino\avr\cores\arduino. however, it is undesirable to have the (potentially inexperienced) user tinkering with these files.

i would like to propose the following solution be implemented, to allow the user to alter the serial buffer sizes:
 
1. in HardwareSerial.h add in the following single (bolded) line:

#if (RAMEND < 1000)
#define SERIAL_TX_BUFFER_SIZE 16
#define SERIAL_RX_BUFFER_SIZE 16
#else
#define SERIAL_TX_BUFFER_SIZE 64
#define SERIAL_RX_BUFFER_SIZE 64
#endif
#include "OVERRIDE\serial_buffer_size.h"  // ##### override default buffer size #####
#endif

2. create a subdirectory ...\hardware\arduino\avr\cores\arduino\OVERRIDE

3. in that subdirectory create a zero-length file called serial_buffer_size.h

the above three steps provide a safe location where (a) a user can override default settings, which can (b) easily be restored to 'factory default' by truncating the files back to zero length, all without (c) altering existing functionality or compiler output provided step 3. holds. in principal the same method could be used to allow for overriding other settings on other core libraries.

i would then like to suggest that the following IDE changes be made to facilitate using the above provisions:

4. to the Arduino IDE add an entry under the "Tools" menu called "OVERRIDES" where the user can select any settings that they wish to override, and select amongst safe override values. for instance, there may be Tools -> OVERRIDES -> Serial RX Buffer Size with options of 'default', '16', '64', '256', '1024' (and the same for Serial TX Buffer Size).


what do the developers think of this idea? steps 1 to 3 could initially be carried out with minimal effort (i have tested out steps 1 to 3 and verified they work as desired), with step 4 (or some variation thereof) implemented later on. overrides could be provided as global irrespective of the board selected, or made specific to only one board or type of processor.

i would much appreciate feedback.


cheers,
rob   :-)

Coding Badly

i've tried posting the below on the Arduino Developers mailing list (https://groups.google.com/a/arduino.cc/forum/#!forum/developers), but my posting never even appeared.
https://groups.google.com/a/arduino.cc/forum/#!topic/developers/4q9k_gvKpLk


robert rozee

https://groups.google.com/a/arduino.cc/forum/#!topic/developers/4q9k_gvKpLk

yes, subsequent to starting the current thread, cmaglie was kind enough to sort out the problem i was having with the developers mailing list, so i was then able to post there. but there is no harm in a discussion here as well.

cheers,
rob   :-)

Coding Badly

#3
Jun 17, 2015, 07:07 am Last Edit: Jun 17, 2015, 07:10 am by Coding Badly

Do this...

• Ensure the Arduino IDE is not running

• Locate boards.txt

• Make a backup copy

• Open boards.txt using your favourite Linux friendly text editor.  I use Visual Studio or Delphi.

• Locate an entry of interest.  I used the one for the Uno.

• Select the entire entry

• Copy the entire entry

• Navigate to the bottom of boards.txt

• Paste the entry

• Carefully change all of the tags in the new entry to something unique.  I changed "uno." to "unox."

• Change the entry name.  I changed my entry name to this...

unox.name=Arduino Uno - Custom Build

• Add this one option to the entry...

unox.build.extra_flags=-include "{build.path}/build_options.h"

• Save boards.txt

• Close the text editor

• Open the Arduino IDE

• Create a new sketch

• Add a tab named "build_options.h"

• Add whatever you want to "build_options.h".  This is what I used...

Code: [Select]

#ifndef build_options_h
#define build_options_h

#define SERIAL_TX_BUFFER_SIZE 16
#define SERIAL_RX_BUFFER_SIZE 16

#endif


• Verify

• Enjoy

Note: When you upgrade the IDE boards.txt is removed.  Make a backup before upgrading.




Complete example...

Code: [Select]
##############################################################

unox.name=Arduino Uno - Custom Build

unox.vid.0=0x2341
unox.pid.0=0x0043
unox.vid.1=0x2341
unox.pid.1=0x0001
unox.vid.2=0x2A03
unox.pid.2=0x0043

unox.upload.tool=avrdude
unox.upload.protocol=arduino
unox.upload.maximum_size=32256
unox.upload.maximum_data_size=2048
unox.upload.speed=1000000

unox.bootloader.tool=avrdude
unox.bootloader.low_fuses=0xFF
unox.bootloader.high_fuses=0xDE
unox.bootloader.extended_fuses=0x05
unox.bootloader.unlock_bits=0x3F
unox.bootloader.lock_bits=0x0F
unox.bootloader.file=optiboot/optiboot_atmega328.hex

unox.build.mcu=atmega328p
unox.build.f_cpu=16000000L
unox.build.board=AVR_UNO
unox.build.core=arduino
unox.build.variant=standard

unox.build.extra_flags=-include "{build.path}/build_options.h"

##############################################################





A big thank you to foraidt...
http://stackoverflow.com/a/3387518

robert rozee

Coding Badly: excellent hint!    :D
 
after a bit of tinkering, found that the same effect can be achieved globally by changing the appropriate  entry in the file platform.txt from:
build.extra_flags=
to:
build.extra_flags=-include "{build.path}/options.h"

unfortunately this means that the compile will fail if options.h does not exist in the sketch directory. is there any way to do an include that does not throw up an error message if the options.h file does not exist?


cheers,
rob   :-)

Coding Badly


Ugh.  You're kind of a needy thing, aren't ya?

1. Create and use a custom board entry.  Which is one reason I suggested that route.

2. Or, do this...

• Create an empty file named build_options.h in the core directory ({ArduinoBase}\hardware\arduino\avr\cores\arduino\)

• Change the build.extra_flags option to this...

unox.build.extra_flags=-I "{build.path}" -include build_options.h


If you find limitations or problems please report back.  If the technique works well please report back.  At a minimum the Arduino folks are likely to add an empty options file to the core, which will make upgrading a bit less painful, but only if this technique is seen to be reliable.


Coding Badly


Caveat: As far as I can tell, the options file (build_options.h) is not included in dependency checking.  If you change that file you will very likely have to restart the IDE.


robert rozee

Coding Badly: you're a legend, that works absolutely brilliantly!    :D

the procedure:

(1) in the file platform.txt change the following line:
        build.extra_flags=
to:
        build.extra_flags=-I "{build.path}" -include options.h

(2) in the Arduino\hardware\arduino\avr\cores\arduino directory
create an empty file called options.h

now in any sketch directory you can create a file called options.h
that contains the things you want to change locally to that sketch.
in my case this is:
#define SERIAL_TX_BUFFER_SIZE 64
#define SERIAL_RX_BUFFER_SIZE 1024


(i suspect that both RX and TX sizes need to be specified as a pair)


when i open my project in the IDE, the options.h file is also opens. if i change the contents of options.h those changes are reflected in the next compile without needing to restart the IDE. it all seems to work quite seamlessly.


cheers,
rob   :-)



Coding Badly

Coding Badly: you're a legend, that works absolutely brilliantly!    :D
"Legend"?  Is that like Seinfeld's "breathtaking"?

Glad you have it working!


robert rozee

not quite - i guess it is a new zealand term (where i live). it means you've done something remarkable, and is a compliment. i just need to persuade the arduino developers to incorporate your method into the next release of the ide:
https://groups.google.com/a/arduino.cc/forum/#!topic/developers/4q9k_gvKpLk

cheers,
rob   :-)

Coding Badly


Oh, I get it.  I just could not resist the opportunity to wind you up.


robert rozee

 :D   i used to be a seinfeld fan way back when he was on tv over here.

i've tried to push things along a little more on the developers forum, so hopefully something may happen. in the meantime, i've been playing around with creating a tiny utility that can have an arduino .HEX file embedded in it and will act as a standalone firmware uploader - so you can send out a small standalone .EXE file that folks run to upload an updated (compiled) sketch to an arduino that is contained within a product.

to this end, i'm trying to figure how much of the STK500v1 protocol is actually implemented in the various arduino bootloaders.


cheers,
rob   :-)

Coding Badly


Start with ArduinoISP.  That will tell you what avrdude needs to function correctly.  I believe the two are exactly mated; ArduinoISP implements the minimum necessary for avrdude to program a target.

Any reason you don't just embed / include avrdude?


robert rozee

of the two existing methods of uploading code to an arduino: the Arduino IDE from source, and avrdude using a pre-compiled .HEX file, alas i've found neither are achievable by the typical user.

while the Arduino IDE is quite an accomplishment in providing accessibility to microcontroller programming for many users, it is an enormous download and beyond the ability of the average user to master for the single task of updating firmware.

and avrdude is itself still quite large (1mb), consists of four parts (avrdude.exe, avrdude.conf, libusb0.dll, the firmware .HEX to be uploaded), and requires a 'cryptic' command line incantation to make work. admittedly, there are wrappers (from batch files to small GUI applications), but most that i've looked at still leave open the opportunity for the end user to get horribly lost or make mistakes.

my solution consists of a single windows executable (or linux binary) that contains the .HEX file embedded within, which the user just needs to run. port selection can be done via a GUI or specified on the command line.


i started out with a trace output from avrdude:
avrdude -patmega328p -carduino -PCOM5 -b57600 -D -Uflash:w:ICSP_v1C.hex -v -v -v -v >trace.txt 2>&1

the 2>&1 at the end is important under windows, as avrdude seems to direct all of its output to stderr, which normally can not be redirected away from the console. this is perhaps a bit of a bug.

that output provided me with details of how avrdude talks to an arduino. i then got hold of microchip's "AVR061: STK500 Communication Protocol" (doc2525.pdf) and used that to derive meaning from avrdude's output. finally i looked over this page:
http://baldwisdom.com/bootloading/
which provided some hints about what things i didn't need to bother with when talking to an arduino bootloader.


i've now got working code, but i am still not 100% sure it will work with any arduino bootloader. fyi, below is my current code (written in C). the firmware has already been loaded into an array of unsigned char called ICSP (in ICSP_v1C.inc). outside of view is also a serial port unit and assignment of 3 to baud_rate.


i'd welcome comment!


cheers,
rob   :-)

Code: [Select]
    if (baud_rate < 5) // invoked by a baud rate of 1, 2, 3, or 4.
    {
        #include "ICSP_v1C.inc"
        int i, n;
        unsigned char buffer [140]; // 0x80 + 12d
        int bps[] = {0, 9600, 19200, 57600, 115200 }; // known arduino bootloader baud rates
 
        baud_rate = bps[baud_rate];
        if (serial_open (port, baud_rate, 100) < 0) {
            printf ("failed to configure port\n");
            serial_close();
            exit (-1);
        }
        printf("%i baud ", baud_rate);

        for (i = 0; i < 40; i++) {

//          sprintf ((char*) buffer, "%c%c", 0x30, 0x20);
            buffer[0] = 0x30; // get synchronization
            buffer[1] = 0x20;

            serial_write (buffer, 2);
            printf (".");
            n = serial_read (buffer, 2);
            if ((n == 2) && (buffer[0] == 0x14) && (buffer[1] == 0x10)) i=100;
        }

        if (i < 100) {
            printf (" arduino/STK500 not found\n");
            serial_close();
            exit (-1);
        }
        printf (" synchronized\n");

//      sprintf ((char*) buffer, "%c%c", 0x50, 0x20);
        buffer[0] = 0x50; // enter program mode
        buffer[1] = 0x20;
        serial_write (buffer, 2);
        serial_read (buffer, 2);

        if ((n != 2) || (buffer[0] != 0x14) || (buffer[1] != 0x10)) {
            printf ("failed to enter program mode\n");
            serial_close();
            exit (-1);
        }
       
//      sprintf ((char*) buffer, "%c%c", 0x75, 0x20);
        buffer[0] = 0x75; // read signature bytes (3)
        buffer[1] = 0x20;
        serial_write (buffer, 2);
        n = serial_read (buffer, 5);

        if ((n != 5) || (buffer[0] != 0x14) || (buffer[4] != 0x10)) {
            printf ("failed to get signature\n");
            serial_close();
            exit (-1);
        }
        unsigned ID = (buffer[1] << 16) + (buffer[2] << 8) + buffer[3];
        printf ("signature = %06x   device = %s\n", ID, ID == 0x1e950f ? "ATmega328P" : "(wrong uP)");
 
        for (i = 0; i < sizeof(ICSP) / 0x80; i++) printf (".");
        for (i = 0; i < sizeof(ICSP) / 0x80; i++) printf ("\b");

        for (i = 0; i < sizeof(ICSP); i += 0x80) {
            printf ("#");

//          sprintf ((char*) buffer, "%c%c%c%c", 0x55, (i >> 1) % 0x100, (i >> 1) / 0x100, 0x20);
            buffer[0] = 0x55; // load address
            buffer[1] = (i >> 1) % 0x100; // address low (word boundary)
            buffer[2] = (i >> 1) / 0x100; // address high
            buffer[3] = 0x20;
            serial_write (buffer, 4);
            n = serial_read (buffer, 2);

            if ((n != 2) || (buffer[0] != 0x14) || (buffer[1] != 0x10)) {
                printf ("\nfailed to load address %04x\n", i);
                serial_close();
                exit (-1);
            }

//          sprintf ((char*) buffer, "%c%c%c%c", 0x64, 0x00, 0x80, 0x46);
            buffer[0] = 0x64; // program page
            buffer[1] = 0x00; // length high (in bytes, NOT words)
            buffer[2] = 0x80; // length low (order reverse to address)
            buffer[3] = 0x46;
            memcpy (&buffer[4], &ICSP[i], 0x80); // data (128 bytes)
            buffer[4 + 0x80] = 0x20;
            serial_write (buffer, 4 + 0x80 + 1);
            n = serial_read (buffer, 2);

            if ((n != 2) || (buffer[0] != 0x14) || (buffer[1] != 0x10)) {
                printf ("\nfailed to program page\n");
                serial_close();
                exit (-1);
            }
        }
        printf ("\n");

//      sprintf ((char*) buffer, "%c%c", 0x51, 0x20);
        buffer[0] = 0x51; // leave program mode
        buffer[1] = 0x20;
        serial_write (buffer, 2);
        n = serial_read (buffer, 2);

        if ((n != 2) || (buffer[0] != 0x14) || (buffer[1] != 0x10)) {
            printf ("failed to exit program mode\n");
            serial_close();
            exit (-1);
        }
        printf ("firmware uploaded to 'ascii ICSP' adapter OK\n");

        serial_close();
        exit (0);
    }

Coding Badly

Code: [Select]
            buffer[0] = 0x30; // get synchronization
            buffer[1] = 0x20;


Magic constants, like 0x30 and 0x20, are a maintenance nightmare.  I suggest using the constants provided by Atmel...

Code: [Select]
#define CRC_EOP             0x20  // 'SPACE'
#define STK_GET_SYNC        0x30  // '0'



Go Up