I am aware (thought I did not try it yet) that it is possible to program a standalone atmega (atmega328p-au) via ICSP by uploading the arduino booltloader and then uploading the sketch.
I was wondering if it is possible to upload only the sketch (hex file) and use the funcionalities of the Arduino libraries? So if i upload the hex file generated from compiling the arduino code via the icsp, will it work (particulary interested in serial comunication, Tx and Rx pin).
Yes, sketches do not need a bootloader to run correctly. The bootloader would only be used for serial uploading of the sketches. When the sketch is running the bootloader is not active or needed at all.
Many have used the standalone mega328P without any bootloader. As long as you have ICSP access then this will work well.
You want to use serial Tx/Rx communications to what?
If you plan to have serial Tx/Rx access to a PC-USB then you will need to us a FTDI adapter. If that is the case then you may still want to consider using a bootloader to work with that. That way you have the same interface for both uploading and serial communications.
Thank you very much for your prompt answer. I really appreciate your help.
I already have onboard TTL to RS232 converter and I will be programing the chip only through ICSP.
I just need serial input and output to send data over to the/and from computer.
If you elect to go bootloaderless, I would suggest using Tools:Burn Bootloader as a first step to ensure the fuses are programmed as needed, and then File:Uplaod Using Programmer to load your sketch & overwrite the bootloader code.
Using 16 MHz crystal? Use Uno as the board type. Board type is independent of how you end up using the serial interface (straight up, or with USB/Serial adapter).
Your TTL to RS232 - you can use a USB/RS232 adapter cable to connect to your PC, can find them online for ~$5
so I tried burning the bootloader to Arduino Nano (328p-au) using Arduino Uno as ISP and then also uploading a sketch to Nano with "Upload using a programmer" option (please see image below).
Everything works as expected, so no problems here.
When I try to do the same with my custom board (328p-au), I can burn bootloader just fine and also the sketch (no errors), but it is not working as expected. The only difference is that my board has 10k pullup on RESET and the Arduino Nano has only 1k. Otherwise, everything is connected ok (I have checked all the connections).
This shouldn't be a problem, right? I'm just trying to rule out this. I already think the problem is in my board design.