I would learn how to modify the bootloader

Using Arduino Microcontrollers
5

So an HEX file isn't a pure binary file.

It's not a binary file at all; it's an ascii representation of a binary file.
Each byte of the binary file is stored as two characters (0xFF becomes "FF"), and there is additional grouping, formatting, typing, error checking, and etc:

:10010000214601360121470136007EFE09D2190140
:SSAAAATTdddddddd...dddddddddCC
SS= size
AAAA= address
TT= type
dd...dd= data
CC= checksum

How file do I have to change, to modify the bootloader? bootloader.c?

optiboot/optiboot.c for optiboot, atmega/ATmegaBOOT_168.c for the old bootloader, etc.

Why is there a file named optiboot_atimega328.lst? What's for?

This is an informational "listing" file produced by compilation process, for reference (for one thing, so you can check that the code produced is small enough to fit in the "bootloader section" of the cpu.)