Quite trivial to do with a keypad attached to the Arduino that also attaches to the scoreboard.
Quite straightforward to do with a Bluetooth connection to the phone, as you can simply pair a phone with the scoreboard and then run a simple app on the phone that allows you to enter the score, sending it over the Bluetooth to your scoreboard.
Getting seriously harder (both for programming and actual use) if you want to put the Internet in the mix. You have to have an app on your phone, that somehow knows how to find your scoreboard. Probably easiest if you have your router assign a fixed IP on your local network to the scoreboard, and then use that IP to have your phone connect to it.
In any case there'd be two independent, identical units. Or three. Or four. Or however many players you want to entertain.