How to flash bootloader section

Consider I have made a custom bootloader, my question is how to put it in the boot section ? What happens behind the scene while trying "burn the bootloader" option ?

You need an external "device programmer" connected to specific pins of the processor, which is then used with AVRDude (or similar) to program the chip with appropriate commands. (It turns out that talking to the bootloader in an arduino chip, and talking to a device programmer use similar techniques.)

In the case of most AVRs, the "device programmer" can be an Arduino running the "Arduino as ISP" sketch.

You can see the commands go by if your turn on the "verbose upload" option in the IDE preferences.
Last time I looked, this was actually a three-step process:

  • Erase chip and unprotect the booloader section
  • program the bootloader
  • re-protect the bootloader section from accidental overwriting.

Normally avrdude starts flashing from the starting address right but bootloader is stored in another part of the flash so how is this achieved ?

avrdude writes the data to whatever address in the flash the .hex file tells it to.

I have made a custom bootloader,

whatever address in the flash the .hex file tells it to.

The implication is that the link command for your bootloader (assuming it's written in C) has to include an option to set the start of the .text area at the proper address. Usually this will look like:

avr-gcc -g -Wall -Os -mmcu=atmega328p  [color=green][b]-Wl,--section-start=.text=0x7e00[/b][/color]  -Wl,--relax -nostartfiles -o optiboot_atmega328.elf optiboot.o