Dear all,
Just spend a few hours on a 328p-au chip to upload sketches via ISP and 'remembering' EEPROM.
One thing to keep in mind using ISP, it seems the fuses are only set while burning a bootloader, and not while uploading a sketch with a programmer. After changing the fuse settings in boards.txt (as explained by doughboy) and burning a bootloader it works.
Thanks you for the information to figure this one out!