I wonder if you could arrange some kind of separate low-voltage cut-out between the battery and your buck/boost converter, such that the converter never sees an input voltage below that level. I don't have a specific solution in mind, but perhaps a relay which releases below 4V (say)?
Some "out of the box" thinking might be needed - I'm sure this is a very solvable problem, provided you can live with the system shutting down (or restarting) when the battery drops below (say) 4V.