is not only to indicate which wire connects to which, this is a necessary but not sufficient condition. If you are using a system which directly translates the schematic to a routing pattern, it may work, though even then there are certain "rules" that need to be followed.
A schematic diagram should also serve to explain the function of the circuit. In order to do this, it is necessary to follow certain general conventions. I believe you need to re-draft at least the left-hand side of your schematic in order not only for us to understand it and check for errors, but so that you can do so effectively as well, because this has obviously been the source of design problems.
A first convention is that the vertical axis suggests voltage. Wherever possible, all the grounds should be at the bottom of the diagram, preferably entering components from below. Similarly, supply feeds should be toward the top and enter from above (with the exception of the regulators where a feed from left to right makes sense). You have initially observed the convention of the power feed from the left-hand side, but things have become confused after then.
Power feeds - regulators and such - should run in sequence (from left to right) indicating flow. If as here, you have two alternate feeds (and you have correctly sourced both from the primary voltage rather than in cascade, to best utilise drop-out voltages), then make two parallel sections one above the other with the split to these on the left-hand side. Avoid round-about loops.
Wherever possible, buttons and their associated pull-ups should appear in the same vertical line from one power rail to the other, and similarly for (power) indicators.
The crystal loading capacitors (as with all bypass capacitors) should preferably be shown vertically, down to the ground. You need to organise the confusion of ground joins - almost all should go down to one rail along the bottom.
If you note the Arduino schematics, they frequently split parts of the I/O busses into completely separate sections of the overall diagram, simply to manage so many connections.