Programming atmega chips via serial

Hello guys, i need help to glue all info i’ve found online in my head.

My goal: understand general mechanism of uploading sketch using default bootloader to Amega328 or Atmega2560

What i do understand (presumably): in the boot time, bootloader checks either it is a flashing (update) session, either its normal boot and he has to run existing sketch. Bootloader might be trigered using either ISCP pins (and protocol) or using rx/tx pins (genuine ones not soft. serial, that are used by usb port as i undestand). I do have serial open to processor via cp2012 adapter, but i guess this can allso work if i have bluetooth adapter wit open serial, ftdi chip or whatever else in the world for that matter. - If i did not get something - please correct me

What i dont get - how to understand routine how to trigger flashing of sketch via serial? What should i send to initiate, how should i send a sketch (i’ve read somwhere that hex has to be converted to binary?), how must i finalize it, on what baud rate should i work.

I know there are lot of gurus here, so please - put me on the right path.

Bootloader might be trigered using either ISCP pins...

No. The bootloader is started after MCU's reset if it is set. After start it waits for programming instructions on serial port or if the timeout occurs it start previously uploaded program.
ICSP is for programming without bootloader and of course the bootloader have to be uploaded via ICSP.

...but i guess this can allso work if i have bluetooth adapter wit open serial, ftdi chip or whatever else in the world for that matter.

Correct.

What i dont get - how to understand routine how to trigger flashing of sketch via serial? What should i send to initiate, how should i send a sketch (i've read somwhere that hex has to be converted to binary?), how must i finalize it, on what baud rate should i work.

The reset pulse starts the bootloader (as previously described), then it is about bidirectional communication - instructions and data. The datasheet for each MCU type contains details about serial programming. Data are in binary form. Arduino SW uses the avrdude tool for programming (on PC side) which is able to accept either HEX or binary data.
The baud rate is given by the bootloader, it's usually hard coded.

Thank you for useful references. Is there any good and simple source of code to look at and understand serial sketch upload using standard bootloader? I mean i get that i can reverse-engineer this from avrdude, but any general references where and what to look for (in terms of protocols, routine places or similar) would be highly appreciated :slight_smile:

And i reviewed datasheet. It talks mostly about burning bootloader and ISP upload, yet these are next steps in my journey :slight_smile:

There is no additional sketch needed for uploading. It is between avrdude and bootloader. Both are open source so code is available on the web. I think, you should start with OptiBoot bootloader which is standard for UNO. It uses stk500v1 protocol - the guide is available on Atmel/Microchip pages. Avrdude is way more complicated since it is capable to deal with many different programmers, protocols and it can be used for many MCUs.

See HowOptibootWorks · Optiboot/optiboot Wiki · GitHub (and the thinks therein.)
Not all bootloaders are optiboot, but the principles will be similar.
Remember that Optiboot is only 256 words of code. If you want to learn how the bootloader works, you'd probably be MUCH better off reverse-engineering the optiboot source code, rather than avrdude (which is huge!)

Oh look - I even made a flowchart!