The problem is that the sort of memory used on the SD card (NAND flash) is not always reliable, therefore a file system is a good way of identifying malfunctioning sectors and mapping them out. So while you could just write to a flat memory map, it might not prove reliable. Also a file system offers a form of ware levelling, so that the same sectors are not being written to all the time but writes are spread out over all sectors.