I found one of the silly mistakes I was talking about. The 3.3V regulator was taking its power from the Vin port. Therefore, It would not work if the board was powered by USB. I moved the connections so that the 3.3V regulator is powered by the 5V regulator.