I can’t speak to the utility of a tone detector if the Modem part of the project is already working. But, using a UART will decrease the difficulty of implementing the data link layer and above by an order of magnitude.
A UART immediately solves the problem of bit detection and demarcation. You go from trying to figure out where individual bits start and stop to having entire bytes handed to you in one step. Plus, you now have all the utility of the Arduino Stream class available and no shortage of tutorials and guides explaining serial data transfer ... such as Serial Input Basics